Logging Prompts and Completions Responsibly — Agent Evaluation & Observability

Full prompt-and-completion capture is the single most useful debugging artifact you can have on an agent. When a user reports that the agent did something wrong, the trace of exactly what the model saw — the system prompt, the retrieved context, the tool calls it issued and their results, the final completion — is usually the only thing that lets you tell whether the model reasoned badly, the retrieval fed it the wrong context, or a tool returned bad data. Metrics and aggregate dashboards tell you *that* something is wrong; the raw payload is what tells you *why*.

That same payload is also the largest data-protection liability in your system. Users type real names, real email addresses, order numbers, sometimes card digits or health details, straight into a chat box, and an agent's tool calls routinely pull that same data out of a CRM or order system and echo it back into the completion. Unlike a structured database column that a DBA can point to and say "that's the sensitive one," a prompt is free text — sensitive data can appear anywhere in it, in any shape, with no schema to warn you it's there.

This lesson treats that as an honest tension rather than a problem with a single clean fix: you cannot get the debugging value of full capture and pretend the liability isn't there, and you cannot eliminate the liability by refusing to log anything without also losing the ability to debug the agent at all. The engineering job is to keep as much of the debugging value as you can while shrinking the liability — through redaction at write time, structured field policies, the right choice between reversible and irreversible transforms, bounded retention, sampling, and access control on whatever you do keep. None of this is a compliance checklist to satisfy a named regulation; specific legal obligations vary by jurisdiction and sector, and confirming them is a job for counsel, not for this lesson. What follows is the engineering discipline that makes a system defensible regardless of which rules end up applying to it.

Analogy🏏Cricket

🏏 Think of it like cricket: When the DRS system reviews a tight lbw shout at the Wankhede, the stump camera and stump mic don't record just the shot in question — they run continuously through the over, capturing everything: the exact seam position on Bumrah's release, the faint sound of bat on pad, and also whatever players say to each other between deliveries, a physio's aside about a niggle, a fielder's private word to the keeper. That full continuous feed is exactly why DRS overturns bad decisions that a single broadcast angle would miss — it is the single most valuable review artifact in the game. But none of that raw feed goes out over the air unfiltered. The broadcast director sits between the stump mic and the viewer, deciding in real time what airs and what stays in the production truck: the nick that overturns the decision goes to millions of fans, the physio's aside about a player's knee does not. The production team didn't get less useful by holding some of the feed back — they got more trustworthy, because players and coaches keep talking honestly near a live mic only because they know a human, not a live broadcast, sits between what's captured and what's aired. The insight: capturing everything is what makes the review system work, and filtering what leaves the capture point is what makes it survivable — you don't get to skip either half.