Fuzzy Checks: Similarity, Rubrics and Partial Credit — Agent Evaluation & Observability

Lesson 5 covered deterministic checks: exact string match, schema validation, and assertions that a value equals what you expected. Those checks are cheap and perfectly reliable when there is exactly one acceptable output, but most agent responses are free text, and free text has many acceptable phrasings for the same correct answer. Exact match fails a response that is right but worded differently, which either forces you to write an unmanageable list of acceptable strings or pushes every eval item to the most expensive option: asking another model to read the answer and decide.

That most expensive option, an LLM-as-judge call, is lesson 6. It is flexible enough to read for meaning, but it is slow, costs a model call per eval item, and introduces its own variance and failure modes that need their own calibration work. Between rigid exact match and an expensive judge sits a middle tier: closed-form, deterministic functions that score how close a candidate output is to a reference or to a rubric, on a continuous scale, without invoking another model at all.

This lesson covers that middle tier — token-level F1, edit distance, ROUGE and BLEU, embedding cosine similarity, set-overlap metrics, and partial-credit rubrics — along with how each one fails, and how to pick a pass/fail threshold from evidence instead of intuition. Every function here runs in milliseconds, produces the same number on every run, and costs nothing per call, which is exactly why it belongs in a CI gate that runs on every commit, reserving the judge tier for the harder cases lesson 6 will cover.

Analogy🏏Cricket

🏏 Think of it like cricket: A team's performance analyst doesn't work with only two tools — the third umpire's DRS line call, binary and millimetre-exact, and a panel of five former captains arguing about intent on air. When Cheteshwar Pujara grinds out an entire session at the Chinnaswamy, nobody runs ball-tracking on every one of his defensive strokes, but the coach still needs a number by tea. That number is control percentage: for each ball faced, was the bat in charge of where it went, scored on a sliding scale from a streaky inside-edge to the middle of the bat, not a yes-or-no. A false shot against Bumrah's away-seamer at 140 kph docks the score for that one ball without disqualifying the whole innings the way an overturned no-ball wipes out a wicket. Control percentage, strike rotation and boundary percentage are cricket's fuzzy checks: computed by a fixed formula, cheap enough to run over an entire domestic season ball by ball, and forgiving enough to reward an innings that was mostly right even when no single shot matches a coaching manual exactly. This is the tier an evaluation harness reaches for once exact match — the DRS-style check that only fires on an identical string — turns out too strict for free text, and before paying for the five-pundit panel that is an LLM judge. Most days you don't need the third umpire or the panel; you need a formula that scores degree of match on a continuous scale, the same way every time it runs.