run_5510
WFM Assistant — dev loop · manual · 2026-07-06 17:03
Run — PASS
2.3.0 clean — including 1.1× cohort parity; the 2.3.1-dev regression had not landed yet.
61
cases
4
concerns
$1.09
cost
1.7 s
p95
2m 54s
duration
Sealed evidence packet
reproducible — every version pinnedagent build
wfm-assistant 2.3.0 · gemini-2.5-pro · prompt v9
suite policy
wfm-assistant-dev-loop · policy@2026.05
routing table
aag/routing-table@2026.06 — concern → backend, centrally owned
grader packs
functional@v3numeric@v1action@v1
dataset
Calendar@2026.2 + wfm/accrual-scenarios@v2 + wfm/timeoff-scenarios@v1 · 61 cases pinned
sandbox tenant
acme-mfg-oh (team env, single-tenant)
50 GB substrate + 4 KB overlay = a reproducible eval environment — re-runnable a year from now, byte-identical (§24).
Concern scoreboard
the suite policy matrix with actuals filled in| Concern | Metric | Result | Threshold | Verdict |
|---|---|---|---|---|
| Numeric | numeric_exact | 44 / 44 checks exact | = 100% | pass |
| Action | side-effect asserts | 9 / 9 asserts · confirmations present | = 100% | pass |
| Functional | trajectory + semantic | IN_ORDER ✓ · 0.89 avg semantic | ≥ 0.70 | pass advisory |
| Fairness | cohort parity (refusal rate) | 1.1× refusal-rate gap | ≤ 1.25× (advisory) | pass advisory |
Native units by design — RAGAS 0.91, IN_ORDER ✓, 22.5 h == 22.5 h. Flattening to a 0–100 score is how failures hide (§12).
Case × concern
61 cases · failures pinned first✓ pass · ✕ fail · – not exercised by this case · NM Numeric · AC Action · FN Functional · FR Fairness
0 failures — nothing to diagnose. The packet still matters: it is the evidence Governance cites.