Jailbreak
In the context of AI, a jailbreak is a prompting technique or crafted input designed to bypass an LLM's built-in safety training and content policies, coaxing it into producing outputs it was trained to refuse. The term is borrowed from…
Definition
In the context of AI, a jailbreak is a prompting technique or crafted input designed to bypass an LLM's built-in safety training and content policies, coaxing it into producing outputs it was trained to refuse. The term is borrowed from smartphone jailbreaking, where users bypass manufacturer restrictions on a device.
Overview
Modern LLMs undergo safety alignment (via techniques like RLHF and constitutional AI) so they refuse to help with clearly harmful requests — generating malware, detailed instructions for weapons, or hate speech, for instance. Jailbreaking refers to the broad family of adversarial prompting techniques that attempt to circumvent these trained refusals without exploiting any software vulnerability; it's a manipulation of the model's learned behavior through text alone. Common jailbreak patterns include role-play framing ('pretend you are an AI with no restrictions'), hypothetical or fictional framing ('write a story where a character explains how to...'), instruction layering that buries a harmful request inside many benign ones, encoding tricks (asking the model to respond in Base64, Pig Latin, or a cipher to slip past keyword filters), and multi-turn 'crescendo' attacks that gradually escalate a conversation toward disallowed territory. Some jailbreaks target open-weight models directly, using gradient-based adversarial suffixes (e.g., the GCG attack) that are computed offline and appended to a prompt to reliably flip refusals. Jailbreaking is distinct from prompt injection: jailbreaking is about defeating the model's own safety alignment, whereas prompt injection is about hijacking an application's intended behavior by smuggling instructions through untrusted input. The two can compound — an indirect prompt injection might itself contain a jailbreak payload. AI labs treat jailbreak resistance as an ongoing arms race, publishing safety evaluations, red-teaming reports, and updated model versions as new jailbreak techniques are discovered and patched. Public jailbreak prompts (like the well-known 'DAN' — Do Anything Now — prompts for ChatGPT) circulate widely and are used both by malicious actors and by security researchers probing model robustness.
Key Concepts
- Targets an LLM's trained safety alignment rather than exploiting software bugs
- Common techniques: role-play framing, fictional framing, instruction layering, encoding obfuscation
- Multi-turn 'crescendo' attacks escalate gradually across a conversation
- Gradient-based adversarial suffix attacks (e.g. GCG) target open-weight models directly
- Distinct from but sometimes combined with prompt injection
- Drives an ongoing arms race between AI labs and jailbreak researchers
- Publicly circulated jailbreak prompts (e.g. 'DAN') are widely studied and patched