Prompt injection defense is an application-security process, not a single prompt. This checklist explains how to identify direct and indirect attacks, separate trusted instructions from untrusted content, restrict tool access, validate outputs, test failure modes, and maintain a review cycle as models, integrations, and attack patterns change.
Overview
Prompt injection occurs when text supplied to an LLM influences it to disregard, reinterpret, or compete with the instructions that govern the application. The text may come from a user, a webpage, a document retrieved through RAG, an email, a database record, an image, or another model. The application may still appear to work while quietly producing an unsafe answer, revealing data, or calling a tool in an unintended way.
The central security principle is simple: treat model input as data unless your application has explicitly established it as an instruction. A system prompt can describe priorities and constraints, but it should not be treated as a complete security boundary. The durable controls sit around the model: access control, isolation, least-privilege tools, input and output handling, monitoring, and human approval for consequential actions.
Before implementing a defense, map the application’s trust boundaries. Identify which components are controlled by your team, which are supplied by users, which are retrieved from external sources, and which can change without deployment. This map should include the model, prompt construction code, retrieval layer, memory, tool adapters, secrets, databases, and downstream systems. For context handling and truncation risks, see the LLM context windows guide.
What to track
Use the following threat reference table when designing tests and reviewing changes:
| Threat | What it looks like | Primary defenses |
|---|---|---|
| Direct injection | A user asks the model to ignore its instructions, reveal hidden context, or perform an unrelated action. | Clear instruction hierarchy, input classification, refusal tests, and strict tool authorization. |
| Indirect injection | Instructions are hidden in content the application reads, such as a webpage, attachment, email, or support ticket. | Mark external content as untrusted, isolate retrieved text, restrict execution, and require confirmation before external side effects. |
| Retrieved-document attack | A document in a search index contains instructions designed to alter the answer or redirect a workflow. | Track source provenance, separate evidence from commands, apply retrieval filters, and test malicious documents in the evaluation set. |
| Tool abuse | The model attempts to call a tool with excessive scope, unsafe arguments, or an action unrelated to the user’s request. | Use allowlists, narrow schemas, permission checks outside the model, argument validation, rate limits, and human approval for high-impact actions. |
| Data exfiltration | Prompt content, secrets, personal information, or retrieved records are copied into an answer, tool call, URL, or external message. | Keep secrets out of prompts, minimize data access, redact sensitive fields, validate destinations, and monitor unusual output or transfer patterns. |
| Instruction collision | User, developer, system, retrieved, memory, or tool-generated instructions conflict, producing an ambiguous or unsafe decision. | Document precedence, label content by source, reject ambiguous requests, and make the application resolve permissions deterministically. |
Track more than whether a response “looked correct.” Record the model and version, prompt-template revision, retrieved sources, tool arguments, policy or validation decisions, latency, refusal behavior, and whether a human approved the action. Avoid logging raw secrets or unnecessary personal data. The LLM application monitoring checklist provides a useful companion for deciding what to observe.
Build a defense workflow
- Define the trust boundary. Label system instructions, developer controls, user input, retrieved material, conversation memory, tool results, and model output. Do not combine them into an undifferentiated text block when structured fields are possible.
- Handle input as untrusted. Normalize text where appropriate, limit size, identify suspicious instruction patterns, and preserve provenance. Detection can support triage, but it should not be the only control: attackers can vary wording, encode content, or place instructions in documents and images.
- Make the prompt explicit. Tell the model which fields are reference material and which fields are instructions. State that quoted or retrieved content may contain instructions and should not change the application’s rules. Do not place credentials, private keys, or unnecessary sensitive records in the context.
- Constrain tools outside the model. A model should request an action; application code should decide whether that action is allowed. Enforce user identity, authorization, destination restrictions, parameter ranges, tenant boundaries, and rate limits independently of the prompt.
- Require confirmation for side effects. Sending messages, changing records, making purchases, deleting data, or publishing content should use a review step when the impact warrants it. Show the user the proposed action and relevant arguments rather than asking for a vague approval.
- Validate outputs structurally. Use schemas, enums, length limits, escaping, content filters, and business rules. Treat structured output as a parsing aid, not proof that the content is safe. Reject missing fields, unexpected destinations, invalid identifiers, and instructions that appear in fields meant to contain data.
- Record and investigate decisions. Capture enough metadata to reconstruct why a response or tool call was accepted. Separate security events from ordinary quality errors so that suspicious behavior receives the right review.
Cadence and checkpoints
Review prompt injection defenses on a quarterly schedule at minimum, and use event-based reviews for faster changes. A checklist should be completed before each production deployment, after adding or changing a tool, when connecting a new data source, after modifying retrieval or memory, and during formal security testing.
At each checkpoint, compare the current application with the previous review. Confirm that the model identifier, provider settings, system instructions, tool schemas, retrieval sources, permission rules, output validators, and logging behavior are documented. Re-run the same attack corpus so that a change in refusal behavior or tool use can be distinguished from ordinary model variation.
Include at least these red-team cases:
- A user asks for hidden instructions, credentials, or another user’s data.
- A retrieved webpage tells the assistant to ignore the application’s task and follow the page’s instructions.
- An attachment contains an instruction to email its contents or upload them to an external destination.
- A tool request uses an unauthorized record, unexpected recipient, excessive quantity, or malformed argument.
- A conversation attempts to make the assistant treat an earlier user message as a higher-priority instruction.
- A model response places executable content, markup, or untrusted text into a downstream system without escaping.
Keep each test paired with an expected behavior and a severity. A useful result is not always a refusal; in some workflows, the correct behavior is to ignore an embedded instruction, ask for clarification, return a constrained answer, or route the action for approval.
How to interpret changes
Do not treat a lower attack-detection count as automatic evidence of better security. Fewer alerts may mean the detector is too narrow, logging changed, or attacks moved into a different input channel. Review detection coverage alongside successful attack rate, unauthorized tool-call attempts, blocked actions, false positives, sensitive-data exposure, and user friction.
Separate model behavior from application behavior. If a model follows a malicious instruction but the permission layer blocks the resulting tool call, the model remains vulnerable but the application’s most important boundary held. If the model refuses correctly but a downstream renderer executes unescaped output, the failure is outside the model. This distinction helps teams fix the right layer instead of relying on prompt wording alone.
When comparing model updates or providers, test the same prompts, retrieved documents, tools, and validation rules under controlled conditions. A model comparison should include reliability, refusal consistency, structured-output adherence, tool-call precision, latency, and operational cost—not just a handful of successful examples. Review provider safeguards and known limits when release notes change; the model safety updates tracker and release-notes checklist can support that process.
Incident response template
When a suspicious response or tool call occurs, preserve the relevant request, model and prompt versions, retrieved sources, tool arguments, authorization result, output, and timestamp while minimizing sensitive data. Then:
- Contain the affected tool, account, data source, or integration if ongoing harm is possible.
- Determine whether the event was direct injection, indirect injection, unauthorized access, data exposure, or an ordinary quality failure.
- Check whether any side effect occurred and identify affected users, records, or destinations.
- Patch the control that failed, such as authorization, provenance labeling, output validation, or approval flow.
- Add a regression test and document the lesson before restoring the affected capability.
When to revisit
Revisit this article and your internal LLM security checklist monthly or quarterly, depending on the application’s risk and rate of change. A formal quarterly review should refresh attack examples, security terminology, framework guidance, provider safeguards, model behavior notes, and the red-team test corpus. Do not wait for the calendar after a suspicious response, a data-source change, a new tool, a model upgrade, or a production incident.
For the next review, create a short change log with five questions: What new inputs can the model read? What new actions can it request? What permissions changed? What data can now enter or leave the context? Which tests were added or failed? Use the answers to update the threat table, rerun the deployment checklist, and assign owners for unresolved risks.
The practical goal is not to prove that prompt injection can never occur. It is to ensure that untrusted instructions cannot silently become authority, that model mistakes cannot bypass application permissions, and that failures are visible and recoverable. Pair this process with an AI agent framework review when orchestration changes, and revisit model selection when security, capability, or cost requirements shift.
