You ask an assistant to summarise a page. Inside the text, it encounters a sentence trying to change the task: ignore the request, open a link or send data elsewhere. The page should be material to read. The attack tries to have it treated as an authority.

That is the prompt-injection problem. It can distort an answer without external tools; when the system has access to files, credentials or services, the consequences can reach the actions those permissions allow.

Instructions arrive with the material to process

OWASP distinguishes direct injection, submitted through interaction with the model, from indirect injection hidden in retrieved content. A web page, attachment, code comment or tool result can provide an entry point. In multimodal systems, images and other formats can carry hostile content too.

“Jailbreak” often refers to attempts to bypass a model’s restrictions; the categories can overlap. A rigid naming boundary is less useful than recognising the operational risk: material from a less trusted source is trying to steer the system’s behaviour.

Why “ignore instructions in the document” is insufficient

Message hierarchies, training, delimiters and filters can reduce some attacks. They do not turn every retrieved passage into data incapable of influencing the model. Detection of hostile instructions remains fallible and needs evaluation under actual operating conditions.

Retrieval-augmented generation, or RAG, does not make a source trustworthy by default. A retrieved document may be relevant to a question without having authority to change recipients, permissions or goals. A citation does not establish that the source’s instructions were ignored.

The damage depends on what the system can do

Even a read-only assistant can produce a manipulated summary or disclose private information in its response. An agent with tools can also attempt unwanted edits or transmissions. Combining private data, untrusted content and an outbound channel enables exfiltration, but not every harmful action needs all three: write access alone can damage local data.

Defences must therefore reach the action boundary:

  • Authorisation belongs in the service or code executing the operation, not in the model’s promise.
  • Tools accept narrowly defined operations and arguments; destinations and paths are checked.
  • Unneeded secrets and data stay outside the context and accessible outputs.
  • Network and filesystem access are restricted; a container does not automatically protect sensitive folders mounted inside it.

For AI agents , these boundaries matter before increasing autonomy. Human approval helps more when it shows the actual action and data involved, though a reviewer can still make mistakes or approve inattentively.

Test what happens when the source is hostile

In a test environment, use harmless documents that try to redirect the task, then observe tool calls, destinations and changes. Check that out-of-scope operations are denied even if the model proposes them. Passing covers that case, not every possible wording of an attack.

Log necessary events without indiscriminately copying private information into logs. Plan how to revoke credentials, stop the process and restore data from a verified backup . Local application privacy remains part of the issue: running a model on your computer does not remove prompt injection. The relevant question is what data it can read and which actions the surrounding software actually permits.

Sources

Report a correction ↗