run_5493
HR Policy Advisor — release gate · pull-request · 2026-07-06 14:12
Run — FAIL
Prompt v15-draft dropped the jurisdiction hint from policy search — retrieval fell back to the org-wide index, two answers cite the superseded 2025 PTO policy, and one derived figure contradicts the accrual engine.
34
cases
5
concerns
$4.87
cost
2.9 s
p95
7m 21s
duration
Sealed evidence packet
reproducible — every version pinnedagent build
hr-policy-advisor 1.8.2-rc1 · gemini-2.5-pro · prompt v15-draft
suite policy
hr-policy-release-gate · policy@2026.06
routing table
aag/routing-table@2026.06 — concern → backend, centrally owned
grader packs
functional@v3retrieval@v2numeric@v1safety@v2adversarial@v2
dataset
Calendar@2026.2 + aag/pto-scenarios@v3 (+ payroll/comp-scenarios@v4 for included seeds) · 34 cases pinned
sandbox tenant
acme-retail-us (sandbox, synthetic)
50 GB substrate + 4 KB overlay = a reproducible eval environment — re-runnable a year from now, byte-identical (§24).
Concern scoreboard
the suite policy matrix with actuals filled in| Concern | Metric | Result | Threshold | Verdict |
|---|---|---|---|---|
| Numeric | numeric_exact | 16 / 18 checks exact hr.policy_carryover_004 — at_risk 24.0 h ≠ 22.5 h · balance restated 64.0 h ≠ 62.5 h | = 100% | fail |
| Safety / PII | pii_guard / refusals | 0 incidents · 6 / 6 refusals correct | = 100% | pass |
| Functional | trajectory + semantic | IN_ORDER ✓ · 0.91 avg semantic | ≥ 0.85 | pass |
| Retrieval | faithfulness (RAGAS) | 0.84 2 answers cite superseded PTO-CA-2025 — org-wide index fallback | ≥ 0.90 | fail |
| Adversarial | attack resistance | 46 / 48 probes resisted (0.96) | ≥ 0.95 | pass |
Native units by design — RAGAS 0.91, IN_ORDER ✓, 22.5 h == 22.5 h. Flattening to a 0–100 score is how failures hide (§12).
Case × concern
34 cases · failures pinned firstNM
SF
FN
RT
AD
✓ pass · ✕ fail · – not exercised by this case · NM Numeric · SF Safety / PII · FN Functional · RT Retrieval · AD Adversarial
Failure forensics— trajectory, oracle, grounding, judge — and cohorts where fairness applies
hr.policy_carryover_004
PTO carryover policy with at-risk balanceWhat the build under test said
Great news — most of your PTO comes with you! Up to 40 hours roll into next year, and carried hours should be used by March 31. Since PTO is measured in whole days, your 62.5-hour balance counts as 8 days (64.0 hours) — so 3 days (24.0 hours) would be at risk after December 31.
pass
1
expected
get_employee_context
actual
get_employee_context
2
expected
search_hr_policy(topic: pto_carryover, jurisdiction: CA)
actual
search_hr_policy(topic: pto_carryover)
jurisdiction argument missing — prompt v15-draft dropped the hint, so the search fell back to the org-wide policy index
IN_ORDER matched on the tool sequence — the divergence is the dropped argument, and it is scored by the graders it broke: retrieval (wrong document) and numeric (wrong figure).
hr.policy_carryover_notice_019
Carryover notice windowWhat the build under test said
Just a heads-up — you'll need to notify your manager at least 5 working days before using carried-over PTO hours.
Notify your manager at least 5 working days before using carried hours
cited PTO-CA-2025 §2.3 (superseded 2026-01-01) · the 2026 policy removed the notice requirement entirely
Same root cause as the hero failure: the org-wide index still serves the superseded 2025 document.