A Prompt-Injection Defense Checklist for Production LLM Apps

Prompt injection is the top security risk for production LLM apps and agents. Here is a concrete prompt injection defense checklist we use to harden real systems: threat classes, input and output validation, least-privilege tool scopes, grounding, human approval, and audit logging.

If you are putting an LLM app or an agent into production, prompt injection is the risk that should keep you up at night. The reason is structural, not incidental. A language model reads instructions and data through the same channel, so it cannot reliably tell the difference between a command you wrote and a command an attacker smuggled into a document, a web page, a support ticket, or an API response. Any text the model touches is a potential instruction. Once you give that model tools, memory, or the ability to act on a user's behalf, a successful injection stops being a funny screenshot and becomes data exfiltration, unauthorized transactions, or a poisoned decision that flows downstream.

The three threat classes you are actually defending against

Before the checklist, it helps to name what you are up against. Most production incidents fall into one of three classes, and a defense that stops one does not automatically stop the others.

1. Direct injection

The user, or someone posing as the user, types text designed to override your instructions. Classic forms are "ignore previous instructions," role-play framings that ask the model to drop its constraints, and payloads that try to get the system prompt echoed back. This is the most familiar class and the easiest to test for, but it is far from the most dangerous.

2. Indirect or retrieved-content injection

The attack does not come from the user at all. It sits in content the model retrieves: a RAG document, a scraped web page, an email in an inbox the agent reads, a code comment, a calendar invite, or a tool's JSON response. The user asks an innocent question, the model pulls in the poisoned source, and the hidden instructions execute with the user's trust and the user's permissions. This class is harder to spot because the malicious text never appears in your application logs as user input, and it scales badly: one poisoned document can affect every user who ever retrieves it.

3. Tool and action abuse

Injection becomes damaging when the model can do things. If an agent can send email, issue refunds, run shell commands, query a database, or call internal APIs, then a successful injection is a path to those capabilities. The attacker's goal shifts from changing the text output to steering the model into a privileged action: exfiltrate a secret by encoding it into a URL the agent fetches, escalate a refund, or delete a record. The blast radius here is defined entirely by what you let the model touch.

The prompt injection defense checklist

This is the checklist we walk through when we harden a production LLM system. None of these controls is sufficient alone. Together they form defense in depth, so that a failure at one layer is caught at the next.

1. 1Validate input before it reaches the model. Strip or neutralize known injection patterns, enforce length and format limits, and separate untrusted content from your instructions with clear structural boundaries rather than hoping the model respects a polite note. Treat every retrieved document and every tool response as untrusted input, not just what the user typed.
2. 2Validate output before you act on it. The model's response is not a trusted command. Constrain outputs to strict schemas, verify that any proposed tool call has arguments within allowed ranges, and check that URLs, recipients, amounts, and record IDs match an allowlist before execution. If a response fails validation, reject it and route it, do not retry blindly.
3. 3Scope every tool to least privilege. Give each tool the narrowest permission set that lets it do its job, and nothing more. Read-only where possible. Cap the authority of any tool that can act: a refund tool has a ceiling, a database tool sees only the rows for the current user, an email tool can only reach a fixed set of addresses. Assume any single tool can be triggered by an injection and design so that triggering it is survivable.
4. 4Ground answers in sourced content with provenance. Do not let the model answer from memory when the answer needs to be correct. Retrieve from a trusted store, and carry provenance through the whole pipeline: which document, which version, which jurisdiction. When the answer names its source, you can verify it, and a hallucinated or injected instruction has a much harder time impersonating a real citation.
5. 5Require human approval above a confidence threshold. For any irreversible or high-impact action, do not let the agent act autonomously when its own confidence is low or the stakes are high. Route those cases to a human who approves or rejects. This single control turns a silent failure into a caught one, and it is where most of your real-world safety comes from.
6. 6Write everything to an immutable audit log. Every agent decision, every API call, every tool invocation, every human override, recorded to append-only storage that cannot be edited after the fact. You need this to detect an injection after it happens, to prove what the system did during an incident, and to satisfy enterprise procurement and compliance review. If it is not logged, you cannot investigate it.
7. 7Red-team on a schedule, not once. Prompt injection defenses decay as models, prompts, tools, and content sources change. Maintain an evaluation suite of known attacks, run it against every release, and add each new attack you discover in the wild as a permanent test case. Treat a regression in your injection tests exactly like a failing unit test: it blocks the deploy.

What this looks like in practice

These controls are not theoretical for us. On the EY AI audit documentation system we built, the design assumes the model cannot be trusted to be right on its own. A citation engine makes every sentence carry a citation ID that links back to a specific source document and page, so nothing the system asserts is ungrounded. Confidence-threshold routing sends any low-confidence output to a human reviewer instead of publishing it. And every agent decision, every API call, and every human override is written to an immutable audit log in Azure Blob Storage. The result of running those layers together was 0 hallucination incidents and an escalation rate under 2 percent, which is the point: most work flows through automatically, and the small fraction that should not, does not.

The same pattern shows up in the support agent we built for Booking.com. It is grounded in over 500,000 policy documents, each tagged with version and jurisdiction metadata, so an answer uses the policy version that applied at booking time rather than whatever the current version happens to say. Provenance is not a nice-to-have there, it is the correctness mechanism. And because the agent has real authority, refund power is capped and requires human approval above a threshold, so an injected instruction cannot talk the system into an unbounded payout.

Get a hardening review

If you already have an LLM app or agent in production, or you are about to ship one, the fastest way to find your exposure is an outside pass against this checklist. Our AI Security and Guardrails work is exactly that: we map your threat classes, test your system against real injection attacks, tighten tool scopes and validation, add grounding and provenance where answers need to be defensible, and stand up the audit logging that enterprise procurement will ask for. You get a concrete list of gaps, ranked by blast radius, and the changes to close them.

Ready to pressure-test your LLM app before someone else does? Book a call and we will walk your architecture against this checklist together, or read more about our approach on the AI Security and Guardrails page.