The idea
If a system concatenates a hidden system prompt with user text, a page it fetched, or an email it summarized, that untrusted text can try to override the original task. The model does not reliably distinguish “data” from “instructions.”
Attacks & defenses
Prompt Injection
Prompt injection is an attempt to make a model ignore its intended instructions by hiding new instructions in user content, documents, or tool output.
All AI security topics01 02 03 04
Why it matters
A successful injection can cause the model to reveal its prompt, misuse a connected tool, or produce misleading output that a user then trusts. The damage depends on what the application is allowed to do: a read-only summarizer is a smaller target than an assistant that can send mail or change access.
Defensive practices
Do not give a model tools that can change state unless a person confirms the action. Treat retrieved documents and web pages as untrusted. Separate system instructions from user content where the platform allows it. Constrain outputs to a schema when you only need structured fields. Test with adversarial strings before exposing a feature.
What this is not
Discussing prompt injection is not an invitation to attack systems you do not own. This note is for builders and reviewers of their own applications.