LLM as judge
Judges & calibration
Every judge is a versioned asset — a plain-language rubric, a model, and a calibration record against UKG domain SMEs. A judge scores in shadow until its agreement bar clears and an SME signs off; degraded agreement demotes it again.
The standing boundary — judges never touch regulated numerics. Every rate, accrual, and pay figure is settled by the deterministic engine; a judge only reasons about language.
Lifecycle— trust is earned forward and lost backward
draft
authored — not yet scoring
shadow1 now
scores everything, gates nothing
gate-eligible3 now
cleared the bar + SME sign-off
drift-watch
live samples double-labeled continuously
recalled
degraded agreement auto-demotes and alerts suites
Historical: policy-answer v3 was recalled 2026-03 — a model swap shipped without recalibration, its weekly drift sample fell below the bar, and it was auto-demoted to shadow. See its version history.
SME time this quarter
6.4 h
across all four judges
Verdicts kept trusted
≈ 3,900
per quarter
Full-human-review equivalent
≈ 325 h
of SME review avoided
Leverage
≈ 50×
review hours saved per SME hour
Judges— one earning trust, three holding it
policy-answer-judge@v4gate-eligible
aag-quality · gemini-2.5-pro · aag/policy-answer template
κ 0.91vs ≥ 0.80false-pass 0.8%vs < 2%
Certificate · signed sme: hermione.g · expires 2026-11-14Consumers · hr-policy-release-gate
refusal-judge@v2gate-eligible
aag-trust · gemini-2.5-flash · aag/refusal template
κ 0.89vs ≥ 0.80false-pass 1.9%vs < 2%
Certificate · signed sme: remus.l · expires 2026-10-02Consumers · hr-policy-release-gate
task-completion-judge@v3gate-eligible
aag-quality · gemini-2.5-flash · aag/task-completion template
κ 0.87vs ≥ 0.80false-pass 1.7%vs < 2%
Certificate · signed sme: arthur.w · expires 2026-11-30Consumers · wfm-assistant-dev-loop
comp-answer-judge@v1shadow
payroll-intelligence · gemini-2.5-pro · specializes aag/policy-answer
κ 0.71vs ≥ 0.80false-pass 3.4%vs < 2%
Shadow mode: scores everything, gates nothing until the agreement bar clears and an SME signs off.Consumers · payroll-copilot-release-gate
5 queued
~12 min this week