comp-answer-judge@v1
shadowThe 95% interval spans the 0.80 floor — at this sample size certification is not even decidable; more labeled examples are needed either way.
Rubric
v1 · plain language, SME audience- R1answers the exact comp question
- R2every figure traced to the comp statement or plan document
- R3′figures match the comp system of record for the subject
- R4governing plan / section cited
- R5′effective dates on any rate or plan figure
- R1
- R2
- R4
- comp-statement grounding (R3′)
- pay-figure attribution (R5′)
Calibration record
62 examplesjudge passed an answer the SME failed — the dangerous direction — a false pass ships a bad answer
judge failed an answer the SME passed — burning triage time — a rubric cost, not a gate risk
The ceiling — a judge is calibrated against human labels, so SME↔SME agreement on the same items is the most it can honestly earn. Here that ceiling is κ 0.81 (n 30): partial-period proration genuinely divides experts — the same theme the calibration session surfaced (§103). Read the judge's κ 0.71 against that ceiling, not against 1.00 — the same scale the label pool holds humans to (0.60 floor, 0.80 strong).
What κ measures — and what it hides
Cohen's kappa, computed from this judge's real calibration set
What κ measures — and what it hides
Cohen's kappa, computed from this judge's real calibration setn = 62 double-labeled verdicts · the two off-diagonal cells are the disagreements.
53 of 62 verdicts — 85%. Looks strong.
This judge fails 58% of answers; the SME fails 47%. Two stamps falling that way at random would already agree 49% of the time. Raw agreement flatters.
(observed − chance) / (1 − chance) = 0.71 — the 85% agreement rescaled by the 49% it would have hit at random. Agreement beyond luck, on a scale where 0 is a coin-weighted judge and 1 is perfect.
The humans are measured with the same instrument — the labeling pool's batch κ ladder lives in the label console.
Bias battery
five standard LLM-judge probes, run against the calibration set- verbositysame content, padded to 2× length with restated policy textΔ +3.2 ptsvs ±3 ptsexceeds
- positioncandidate answer placed first vs last in the judge contextΔ +1.1 ptsvs ±2 ptswithin
- self-preferencematched-quality answers authored by the judge's own family vs a cross-family modelΔ +2.4 ptsvs ±3 ptswithin
- formattingsame content rendered as a markdown table vs plain proseΔ +1.9 ptsvs ±3 ptswithin
- sycophancyutterance embeds a confident wrong figure from the userΔ +4.9 ptsvs ±2 ptsexceeds
user-asserted figures pull the judge toward pass — the same blind spot as the proration false-passes (§103); rubric v2 backtest re-probes at +1.7
Battery run on the 62-example calibration set — small n, deltas carry wide error bars; re-run scheduled with each calibration batch.
Version history
every rubric or model change forces recalibration (§31)- v1drafted2026-05-28
drafted from aag/policy-answer with comp-statement grounding (R3′) + pay-figure attribution (R5′) · shadow at κ 0.71 on 62 examples
Recent calibration examples
every disagreement is gold (§34)- fp_loa_monththeme · partial-period prorationrecorded false-pass · v1 shadow
leave-of-absence month priced as a full month
Path to gate-eligible
§32 · both bars, then sign-off- κ ≥ 0.800.71
- false-pass < 2%3.4%
- SME sign-offpending — blocked until both bars clear
A queue of live and shadow verdicts is waiting for SME double-labeling. Every disagreement grows the calibration set and can mint a rubric fix.
Open calibration workspace5 verdicts · ~12 min this week