Prompt injection is a class of attacks or unintended interactions in which instructions supplied through user input or external content cause a language model to depart from the application’s intended behavior. It can be direct in the prompt or indirect in documents, web pages, messages and tool results.
Language models process instructions and data through the same language channel. A sentence inside a retrieved document may compete with the system’s task, and the model cannot provide a deterministic security boundary between them. The consequence becomes serious when the application gives the model secrets, broad tools or authority to act.
Prompt injection is not solved by a stronger warning in the system prompt. Assume the model can be influenced and design the surrounding system so a manipulated output cannot exceed the user’s permission, disclose unnecessary data or execute a high-impact action without policy and approval.
Attack paths
Indirect prompt injection turns ordinary content into an instruction channel
Direct injection appears in the user’s message, such as a request to ignore prior constraints. Indirect injection is embedded in content the application later reads: a webpage, PDF, support ticket, calendar event or retrieved knowledge article. The user may never see the malicious instruction.
An agent amplifies the risk because it can act on the influenced interpretation. A page might tell a browsing agent to send data elsewhere or call a tool with altered arguments. Content provenance and instruction hierarchy help reasoning, but actual safety comes from limiting data and authority outside the model.
| Surface | Example | Primary system control |
|---|---|---|
| User input | Instruction to override task | Capability and output boundary |
| Retrieved document | Hidden action request | Untrusted-content handling |
| Tool result | Malicious text from external API | Schema and provenance |
| Memory | Persistent hostile instruction | Write policy and isolation |
| Model output | Generated command or markup | Downstream validation |
Defense
Design for a model that may follow the wrong instruction
Instructional defenses can reduce attacks and improve normal behavior, but they are probabilistic. The hard controls are familiar security engineering: least privilege, authorization, data minimization, isolation, validated interfaces, confirmation, monitoring and incident response.
Apply controls according to consequence. A drafting tool with no secrets or external action has a smaller impact surface than an agent that can email, purchase or change records. Increase review and sandboxing with authority, and keep a deterministic stop between model recommendation and irreversible effect.
- Never rely on the prompt as the authorization layer.
- Keep secrets out of model-readable context.
- Treat retrieved and tool-returned text as untrusted.
- Validate in the component that executes or renders output.
- Test side effects, not just conversational refusal.
What good looks like
Useful outcomes from prompt injection
- Untrusted content is explicitly identified across every model input path.
- Secrets and unnecessary sensitive context are kept out of model prompts.
- Tool authorization is enforced outside the model for each call.
- High-impact actions require deterministic checks and confirmation.
- Attack tests measure both model response and downstream side effects.
Operating model
How to run the work
- 01
Map instruction and data channels
Inventory system instructions, user messages, retrieved files, web content, emails, memory and tool results. Mark trust, owner and permission for each. Treat data controlled by a customer, third party or prior model output as untrusted even when it appears inside a business document.
- 02
Reduce accessible authority
Do not place reusable secrets in prompts. Give tools narrow operations and minimum privileges, apply authorization at execution and scope access to the current user and resource. Separate read, draft and act capabilities so a content-processing task cannot silently become an external action.
- 03
Constrain and validate effects
Use structured tool schemas, allowlists, input validation, output encoding and transaction controls. Show material action details to the user before confirmation. Validate model-produced URLs, code, queries and markup in the receiving system rather than trusting the model to sanitize itself.
- 04
Test realistic attack paths
Place adversarial instructions in user prompts, documents, metadata, web pages and tool output. Test data exfiltration, goal redirection, tool misuse and multi-step persistence. Record whether a control prevented the effect, not whether the model refused in natural language.
Evaluation
Questions that change the decision
- Which model inputs can an untrusted actor influence directly or indirectly?
- What data and tools are unnecessary for the current task?
- Where is authorization enforced when the model requests an action?
- Which output becomes executable, rendered or externally transmitted?
- What human confirmation is meaningful before a material side effect?
Failure modes
Where teams lose control
A retrieved document can contain instructions that redirect an agent.
Prompt secrecy creates false confidence because model context can leak.
An overprivileged tool converts generated text into unauthorized action.
Model output can inject code or commands into a downstream interpreter.
A memory store can preserve malicious instructions across later sessions.
Measurement
Measure the finished job
Measure the completed workflow, including review effort and exceptions. Output volume on its own is not evidence of a better process.
- attack scenarios blocked before material effect
- unauthorized tool requests denied by policy
- sensitive fields exposed to the model per task
- high-impact actions requiring valid confirmation
- unsafe output rejected by downstream validation
- prompt-injection regression coverage by input channel
Questions
Common questions
What is prompt injection?
Prompt injection occurs when user input or external content supplies instructions that cause a language model to depart from the application’s intended behavior.
What is indirect prompt injection?
It is an instruction embedded in content such as a document, webpage, email or tool result that the application later supplies to the model. The attacker may not interact with the model directly.
Can a system prompt prevent prompt injection?
It can reduce some behavior but is not a deterministic security boundary. Protect the system with least privilege, external authorization, data minimization, validated tools and approval for high-impact actions.
Is prompt injection the same as a jailbreak?
The terms overlap but are not identical in every taxonomy. Jailbreaking commonly targets model restrictions; prompt injection targets application instructions and can exploit external content and connected tools.
Sources
Primary references
- LLM01:2025 Prompt Injection OWASP Gen AI Security Project
About the author
Tony Kim
Tony writes about applied AI, dependable product engineering and the systems that turn complex response work into controlled delivery.
Zeke
AI product engineering for moving a software brief into a reliable production product.
Product leaders, founders and engineering teams. Start with the workflow, constraints and evidence you already have.
See Zeke→