Building an Agent Observability Dashboard — Agent Evaluation & Observability

Lessons 18 through 22 built the raw material: spans wrapped around every model call and tool call, prompts and completions logged safely, token and cost fields attached to every run, and a loop guard that fires when an agent repeats itself. That is a lot of structured data landing somewhere every time the agent runs — and on its own, none of it tells anyone what happened yesterday. A team that has to grep through trace files to answer 'is the agent healthier this week than last week' will stop asking the question.

This lesson turns those traces into a dashboard backend a team actually opens: an ingestion path that takes a run record and writes it into a small SQLite store, a query layer that computes a short list of panels worth looking at, drill-down from any summary number to the individual runs behind it, and a rendered report. Everything here runs against a local file — `dashboard.db` — with no external metrics service, no API key, and no network call. That is a deliberate design choice, not a shortcut: a store you can seed, query, and inspect offline is one you can test in CI and reason about at 2am, and it scales to real production traffic before a team needs anything heavier.

The build is opinionated on purpose. Seven candidate panels get built — success rate over time, latency percentiles, cost per tenant, tool error rates, termination-reason breakdown, loop-guard fire rate, and drill-down — and each one is defended on what decision it drives, because a dashboard that shows every number that can be computed is worse than one that shows the five a team will actually act on.

Analogy🏏Cricket

🏏 Think of it like cricket: the giant screen at Eden Gardens doesn't show the raw ball-by-ball feed the official scorer is typing — deliveries, wides, byes, every umpire signal — it shows run rate, required rate, partnership, and the last-ten-overs summary, refreshed every ball. The scorer's ledger is complete and exhaustive; the big screen is opinionated. Someone decided that 60,000 fans in the stands need four or five numbers, not the full delivery log, and that those numbers have to update between balls, not at the end of the innings. When Virat Kohli is chasing 180 in a run-chase at the Chinnaswamy, the screen doesn't dump every dot ball from overs 3 through 7 — it shows the required run rate climbing, because that single number tells the crowd everything the raw feed would take an analyst ten minutes to derive. A telecast that just replayed the scorer's book on screen would be technically complete and practically useless — nobody could watch a match that way. The insight is that a dashboard's job is the same curation: take a complete, low-level log of everything that happened and surface the handful of numbers a person can act on in the fifteen seconds they're willing to glance at it.