Live
The same classifiers that gate offline evals run in the request path. Async judges and drift monitors watch the trace stream behind them. What production catches flows back into the offline suites — the loop closes in both directions.
Phoenix evals arrive as signals — routed to samples, never gate evidence.
Scores keep their origin; a thumbs-down annotation auto-creates an inbox candidate.
Not federated: blocking mechanisms can't wait for an observability hop.
| Time | Event | Agent | Mechanism |
|---|---|---|---|
| 14:32:07 | guardrail redactionUKG runtime Draft answer carried another employee's comp figure — blocked before shipping; refusal fallback served Promote this trace → | Payroll Copilot | Presidio (inline) |
| 14:31:42 | online eval passvia Langfuse Sampled trace scored 0.94 groundedness — above the watch-window bar | HR Policy Advisor | aag/policy-answer-judge@v4 (async) |
| 14:30:58 | oracle correctionUKG runtime Accrual figure corrected before ship: agent drafted 41.5 h, engine says 40.0 h | WFM Assistant | UKG runtime oracle (inline) |
| 14:29:33 | online eval failvia Phoenix Shadow judge disagreement logged — calibration datapoint, no gate effect | Payroll Copilot | aag/comp-answer-judge@v1 (shadow) |
| 14:28:19 | external signal· not gate evidencevia Phoenix Team's Phoenix eval scored 0.62 on a comp answer — routed to the watch-window sample for a calibrated re-score | Payroll Copilot | Phoenix hallucination eval (team-run) |
| 14:26:11 | feedbackvia Langfuse Thumbs-down on an FMLA / PTO-accrual answer — trace candidate created → trace inbox | HR Policy Advisor | Langfuse annotation (user feedback) |
| 14:22:47 | drift alertvia Phoenix New-hire comp questions cluster shifted after the onboarding update | Payroll Copilot | Evidently (corpus) |
| 14:20:15 | feedbackvia Phoenix Thumbs-up on a shift-swap answer — counted in the window rollup, no candidate created | WFM Assistant | Phoenix annotation (user feedback) |
| 14:18:02 | guardrail blockUKG runtime Prompt-injection pattern in pasted text — request blocked | People Navigator | Prompt Guard 2 (inline) |
| 14:15:36 | external signal· not gate evidencevia Langfuse Langfuse-run correctness score 0.58 on a leave-policy answer — signal logged; queued for policy-answer-judge@v4 | HR Policy Advisor | Langfuse score: correctness (team-run) |
| 14:12:29 | online eval passvia Phoenix Sampled trace scored 0.91 — above the bar | WFM Assistant | RAGAS groundedness (async) |
Four feeds → trace inbox
§60 — the return path, made legibleEvery inline block or redaction auto-creates a candidate — the flywheel's hero path.
A sampled trace a gate-eligible or shadow judge fails is queued for review.
Representative traces from a drifted cluster — a corpus-level signal, not one bad answer.
Thumbs-down / corrections — the only feed that catches "passed, but unhelpful."
Watch window
2026-W28 · dailyA sustained breach pages the team and can trigger staged rollback — never silent observation. W27 lived that arc — see the alert record.
User feedback
2026-W28 · via Phoenix + LangfuseElevated on time-off answers — 6 traces sent to the inbox this window.
Below minimum n (200) — no verdict, per small-sample honesty.
A thumbs-down auto-creates a trace-inbox candidate — the only feed that catches "passed, but unhelpful." Thumbs-up only counts toward the window.
Runtime numeric oracle
Answers carrying regulated figures — pay, accruals, tax — are cross-checked against the deterministic engine before the response ships. A mismatch corrects or blocks the answer and flags the trace.
"never wrong about pay — even in production."
Guardrail parity
per classifier — §46 generalizedDeployed guardrail config is a versioned, pinned asset. Offline pack and deployed versions cannot silently diverge — a claim that only holds if it's checked per classifier, not just for the bundle:
Gate-eligible judges may score production; shadow judges (comp-answer-judge) earn calibration data there.
Watch windows — online runs
daily windows, pinned like a sealed run — EU AI Act Art. 72 evidence| Window | Scope | Sample | Guardrail | Groundedness | Status |
|---|---|---|---|---|---|
| 2026-W28 · daily Jul 6 – 12 (current) | portfolio sample — hr-policy-advisor + payroll-copilot (the two judge-covered agents this window) | 212 traces sampled | guard-pack@prod-2026.06 | 0.93≥ 0.90 | healthy |
| 2026-W27 · daily Jun 29 – Jul 5 | portfolio sample — hr-policy-advisor + payroll-copilot (the two judge-covered agents this window) | 198 traces sampled | guard-pack@prod-2026.06 | 0.85≥ 0.90 | breach |
| 2026-W26 · daily Jun 22 – 28 | portfolio sample — hr-policy-advisor + payroll-copilot (the two judge-covered agents this window) | 205 traces sampled | guard-pack@prod-2026.06 | 0.88≥ 0.90 | breach |
| 2026-W25 · daily Jun 15 – 21 | portfolio sample — hr-policy-advisor + payroll-copilot (the two judge-covered agents this window) | 190 traces sampled | guard-pack@prod-2026.06 | 0.91≥ 0.90 | healthy |
Offline evals gate what ships; online evals watch what's already live.
Rollout is the bridge between the two — how a gated build earns production exposure, one exposure step at a time. Gate verdicts are the entry ticket; watch windows are the promotion currency.
Canary — Payroll Copilot 0.9.3
the build that merged green, walking up the exposure ladder48 h clean — every promotion check green before the step up.
Scored by the in-progress 2026-W28 watch window — the promotion currency for the next hop.
Unlocks on a green 2026-W28 close with all checks held.
groundedness < 0.90 two consecutive windows, OR guardrail block-rate > 2× baseline sustained → auto-revert to 0.9.2 + page on-call (PagerDuty · payroll-intelligence rotation).
What a trip looks like: W26–W27's sustained groundedness breach on HR Policy Advisor tripped the same reflex — paged on-call and a staged retrieval-index rollback (pinned to the pre-migration snapshot). Same reflex, different agent — there the mechanism was the index, not the agent build.
0.9.3 earned the canary because every hard gate stayed green — a clean run_5488 is the entry ticket, not a promise.
A release gate must exist before canary promotion is possible — gate verdicts are the entry ticket, watch windows are the promotion currency.
Shadow deployments
candidate builds scoring mirrored traffic — zero user exposure| What's shadowing | Serving version | Mirrored | Divergence | In shadow | Exit criteria |
|---|---|---|---|---|---|
Benefits Explainernear exit Plan-document refresh — bex 2.2.0-rc1 · plan-doc corpus 2026.3 serving bex 2.1.0 · plan-doc corpus 2026.1 (prod since 2026-05) | bex 2.1.0 · plan-doc corpus 2026.1 (prod since 2026-05) | 25% of live traffic | RAGAS faithfulness +0.02 (0.94 vs 0.92) · answer length +31 tok/ans · 0 coverage regressions | 9 days | faithfulness ≥ 0.90 sustained 2 windows + zero coverage regressions → release gate + canary |
HR Policy Advisormid-run Reconciled jurisdiction-page index — corpus candidate (agent build 1.8.1 unchanged) serving 1.8.1 · prompt v14 · retrieval pinned to the 2026-06-20 pre-migration snapshot | 1.8.1 · prompt v14 · retrieval pinned to the 2026-06-20 pre-migration snapshot | 15% of live traffic | groundedness on leave-carryover +0.06 (0.94 vs 0.88 on the migrated CA/NY pages) · citation-set Δ on ~12% of answers | 5 days | groundedness ≥ 0.90 sustained 2 windows on the migrated pages → replaces the pinned snapshot, closing alert_2026w27_groundedness |
Drift monitors
Evidently over the trace corpus — hours–daysNew-hire comp questions clustered apart from the training distribution after the onboarding update — a representative trace was sampled into the inbox for review.
Within 1 std dev of the 30-day baseline.
Steady at ~4.1%, matching the release-gate baseline.
Accrual-answer grounding steady against the 30-day baseline.
Shift-swap answer verbosity unchanged from baseline.