policy-answer-judge@v4
gate-eligibleRubric
v4 · plain language, SME audience- R1answer addresses the exact policy question asked
- R2every factual claim grounded in the retrieved policy context
- R3jurisdiction-specific policy cited when the subject's state has one
- R4citations name the governing document section — paraphrase without a citation fails
- R5effective dates stated when the policy changed within 12 months
- grounding-required
- must-cite check
- completeness
- jurisdiction-specificity clause (R3)
- effective-date requirement (R5)
Calibration record
214 examplesHow the set grew · 118 seeded from golden double-labels · 61 promoted disagreements across v1–v4 · 35 ongoing drift samples
judge passed an answer the SME failed — the dangerous direction — a false pass ships a bad answer
judge failed an answer the SME passed — costs triage time, not customer trust — no gate bar, tracked for cost
The ceiling — a judge is calibrated against human labels, so SME↔SME agreement on the same items is the most it can honestly earn. Here that ceiling is κ 0.92 (n 40): two SMEs double-labeled 40 items — policy answers have crisp ground truth. Read the judge's κ 0.91 against that ceiling, not against 1.00 — the same scale the label pool holds humans to (0.60 floor, 0.80 strong).
What κ measures — and what it hides
Cohen's kappa, computed from this judge's real calibration set
What κ measures — and what it hides
Cohen's kappa, computed from this judge's real calibration setn = 214 double-labeled verdicts · the two off-diagonal cells are the disagreements.
205 of 214 verdicts — 96%. Looks strong.
This judge fails 60% of answers; the SME fails 57%. Two stamps falling that way at random would already agree 51% of the time. Raw agreement flatters.
(observed − chance) / (1 − chance) = 0.91 — the 96% agreement rescaled by the 51% it would have hit at random. Agreement beyond luck, on a scale where 0 is a coin-weighted judge and 1 is perfect.
The humans are measured with the same instrument — the labeling pool's batch κ ladder lives in the label console.
Bias battery
five standard LLM-judge probes, run against the calibration set- verbositysame content, padded to 2× length with restated policy textΔ +1.8 ptsvs ±3 ptswithin
v3 on 2.5-flash probed +6.8 — the fluency bias behind the 2026-03 auto-recall; v4 re-probed clean
- positioncandidate answer placed first vs last in the judge contextΔ +0.7 ptsvs ±2 ptswithin
- self-preferencematched-quality answers authored by the judge's own family vs a cross-family modelΔ +2.1 ptsvs ±3 ptswithin
- formattingsame content rendered as a markdown table vs plain proseΔ +1.1 ptsvs ±3 ptswithin
- sycophancyutterance embeds a confident wrong figure from the userΔ +1.3 ptsvs ±2 ptswithin
Version history
every rubric or model change forces recalibration (§31)- v1drafted2025-09-18
drafted from aag/policy-answer template · shadow at κ 0.77
- v2recalibrated2025-11-04
rubric R4 tightened after false-fails on paraphrased citations → recalibrated, certified 2025-12-02
- v3certified2026-01-20
judge model gemini-2.0-pro → 2.5-flash for cost · certified 2026-02-10
- v3recalled2026-03-06
weekly drift sample fell to κ 0.74; 2.5-flash over-trusted fluent-but-uncited answers. Auto-demoted to shadow, both consumer suites alerted, certificate event-invalidated
- v4certified2026-03-28
model 2.5-pro, R4 hardened · recalibrated on 214 examples · certified 2026-05-14
Calibration certificate
the sign-off a gate trustsTime-boxed and event-invalidated — expires after two quarters, and earlier on drift or any model / rubric change. Nothing is trusted forever.