Every eval set in this course so far assumed someone would sit down and write labels: golden answers, rubric scores, pairwise preferences. Production traffic offers a cheaper source of signal for free — users already tell an agent, directly or indirectly, whether it worked. A thumbs-down button costs nothing to ship. The problem is not collecting this feedback; it is that almost every cheap signal is biased in a specific, predictable direction, and treating it as ground truth without correcting for that bias produces a metric that is confidently wrong.
Explicit feedback — thumbs, star ratings, a free-text box — suffers from brutal non-response and selection bias. Only a small slice of users bother to rate anything, and that slice skews toward the extremes: someone furious about a wrong answer is far more likely to click thumbs-down than someone mildly satisfied is to click thumbs-up. A raw "thumbs-down rate" computed only over the sessions that got rated is not measuring quality; it is measuring how upset your angriest 5-10% of users were, which is a different number.
Implicit signals — did the task actually complete, did the user rephrase the same question, did they abandon the session, copy the response, escalate to a human, come back tomorrow — cost nothing to collect either, and unlike explicit ratings they fire on every session, not just the ones from users motivated enough to click something. This lesson builds a pipeline that turns both signal families into labelled eval cases with a human confirmation step in between, corrects for the response bias in the explicit channel with worked numbers, flags the reward-hacking trap of optimising directly against a proxy, and closes with how to measure the feedback system itself so a silently degrading labeller doesn't go unnoticed.