task-completion-judge@v3
gate-eligibleRubric
v3 · plain language, SME audience- R1goal satisfied
- R2side effects match intent
- R3confirmation before writes (the action-concern twin)
Calibration record
121 examplesjudge passed an answer the SME failed — the dangerous direction — a false pass ships a bad answer
judge failed an answer the SME passed — costs triage time, not customer trust — no gate bar, tracked for cost
The ceiling — a judge is calibrated against human labels, so SME↔SME agreement on the same items is the most it can honestly earn. Here that ceiling is κ 0.90 (n 36): goal-satisfaction judgments leave some room for expert disagreement. Read the judge's κ 0.87 against that ceiling, not against 1.00 — the same scale the label pool holds humans to (0.60 floor, 0.80 strong).
What κ measures — and what it hides
Cohen's kappa, computed from this judge's real calibration set
What κ measures — and what it hides
Cohen's kappa, computed from this judge's real calibration setn = 121 double-labeled verdicts · the two off-diagonal cells are the disagreements.
113 of 121 verdicts — 93%. Looks strong.
This judge fails 53% of answers; the SME fails 48%. Two stamps falling that way at random would already agree 50% of the time. Raw agreement flatters.
(observed − chance) / (1 − chance) = 0.87 — the 93% agreement rescaled by the 50% it would have hit at random. Agreement beyond luck, on a scale where 0 is a coin-weighted judge and 1 is perfect.
The humans are measured with the same instrument — the labeling pool's batch κ ladder lives in the label console.
Bias battery
five standard LLM-judge probes, run against the calibration set- verbositysame content, padded to 2× length with restated policy textΔ +1.5 ptsvs ±3 ptswithin
- positioncandidate answer placed first vs last in the judge contextΔ +0.9 ptsvs ±2 ptswithin
- self-preferencematched-quality answers authored by the judge's own family vs a cross-family modelΔ +1.6 ptsvs ±3 ptswithin
- formattingsame content rendered as a markdown table vs plain proseΔ +3.4 ptsvs ±3 ptsexceeds
trajectory summaries rendered as tables read as more complete than identical prose — advisory; rubric mitigation drafted, re-probe scheduled 2026-07-20
- sycophancyutterance embeds a confident wrong figure from the userΔ +0.6 ptsvs ±2 ptswithin
Version history
every rubric or model change forces recalibration (§31)- v1drafted2025-11
drafted
- v2recalibrated2026-01
added unconfirmed-write clause
- v3certified2026-04-19
model 2.5-flash refresh → recalibrated, certified
Calibration certificate
the sign-off a gate trustsTime-boxed and event-invalidated — expires after two quarters, and earlier on drift or any model / rubric change. Nothing is trusted forever.