Rebuff

Definition

Rebuff is an open-source framework for detecting prompt injection attacks against large language model applications, combining heuristic filtering, a dedicated detection model, canary token insertion, and a database of known attack signatures to flag suspicious inputs before they reach an LLM. Because it is self-hostable, teams can run it entirely within their own infrastructure without sending prompts to a third-party security API.

Overview

Rebuff was created to give developers a self-hostable, layered defense against prompt injection, one of the more distinctive security risks introduced by LLM applications that accept free-form user text. Rather than relying on a single detection technique, it combines several approaches: heuristic pattern matching for common injection phrasings, a dedicated classifier trained to recognize injection attempts, and a vector database of previously seen attacks that can catch semantically similar new attempts. A notable technique Rebuff uses is canary token insertion, where a unique token is embedded into the prompt sent to the LLM; if that token later appears in the model's output in an unexpected way, it can indicate that the model followed injected instructions to reveal or manipulate hidden prompt content, giving a concrete signal of a successful injection rather than relying purely on input-side detection. Because it is open source, Rebuff can be self-hosted and integrated directly into an application's request pipeline, which appeals to teams that want more control over their security stack than a fully managed third-party service provides, or that have data residency requirements that make sending prompts to an external security API undesirable. Rebuff's layered approach reflects a broader reality in prompt injection defense: no single technique reliably catches all attack variants, so combining heuristics, model-based classification, and behavioral signals like canary tokens improves coverage without any one method needing to be perfect on its own. As an actively evolving open-source project, Rebuff's detection coverage depends on community contributions and how current its attack signature database is kept, and it is generally used as one component within a broader LLM application security posture rather than a complete standalone solution. Running Rebuff well requires ongoing attention rather than a one-time setup: the vector database of known attacks and the classifier model both benefit from periodic updates as new injection techniques surface, and a team that deploys Rebuff and never revisits its configuration will see its effectiveness erode over time relative to attackers who continue to adapt. The canary-token technique is particularly useful for catching injection attempts that other layers miss, since it detects an actual behavioral consequence of a successful attack rather than trying to predict maliciousness from the input text alone, though it only helps after the fact and cannot prevent the initial manipulation from occurring. Because it is self-hosted, Rebuff also shifts operational responsibility onto the adopting team, including keeping the underlying classifier and database current, monitoring its own false-positive rate against legitimate traffic, and integrating its output into whatever blocking or alerting logic the application needs.

Key Features

• Heuristic pattern matching for common prompt injection phrasings
• Dedicated classifier model trained to detect injection attempts
• Vector database of known attacks for semantic similarity matching
• Canary token insertion to detect successful injections behaviorally
• Self-hostable, open-source deployment for full control over the security stack
• Layered, multi-technique detection rather than a single filtering method
• Designed to integrate into existing LLM application request pipelines

Use Cases

Alternatives

Frequently Asked Questions

Related in AI Tools