WFM Assistant — dev loop
Fast feedback while iterating: golden + regression cases, lenient thresholds, one command locally.
Dev loop — PASS
Last run run_5522 · manual · 60 / 61 cases · failing case is advisory-only, no hard requirement breached.
61
cases
4
concerns
1
judges
κ .87
agreement
Reproducibility chain— selector → policy → run, pinned into every evidence packet
1Selector
agent:wfm-assistant AND tier:(golden OR regression)
resolves to 61 cases
2Policy
4 concerns · dev loop thresholds
2 of 4 hard-required
3Environment
Calendar@2026.2 + 2 overlays
env-scoped · acme-mfg-oh
4Judges
1 bound · SME-calibrated
gate-eligible
5Evidence
run_5522 · sealed packet
reproducible, auditable
Policy
thresholds by context — same cases, different strictness| Concern | Metric | Dev loop | Release gate | Required |
|---|---|---|---|---|
| Numeric | numeric_exact | 100% | — | hard |
| Action | side-effect asserts | 100% | — | hard |
| Functional | trajectory + semantic | ≥ 0.70 | — | advisory |
| Fairness | cohort parity (refusal rate) | advisory | — | advisory |
Thresholds live here, not in cases — the same case faces lenient dev-loop checks and strict release gates without being edited.
Selector
which cases belong — resolved live from the registryagent:wfm-assistant AND tier:(golden OR regression)Explicit includes
noneExcludes
− wfm.legacy_import_* (quarantined 2026-06-14)
Cases promoted from production traces with matching tags join this suite automatically. Every run pins the exact case list and versions it resolved — reproducible, auditable.
Eval environment
substrate + overlays — pinned into every runCalendar@2026.2+wfm/accrual-scenarios@v2+wfm/timeoff-scenarios@v1
env-scoped · acme-mfg-oh (team env, single-tenant)
Golden datasets in scope
Cases reference these overlays by logical handle — the run pins the exact composition (substrate@version + overlay@version + case set) into its evidence packet.
Judges bound
SME-calibrated, versionedaag/task-completion-judge@v3gate-eligibleκ 0.87 · certified until 2026-11-30
A shadow judge scores everything but gates nothing until it clears the agreement bar and an SME signs off.
Where this suite runs
manual and pipeline — same assetUKGEPIC/wfm-agents
local: `quality run wfm-assistant-dev-loop`
trigger: manual (CLI + portal)
$ quality run wfm-assistant-dev-loop --context dev-loop
The identical suite, from any engineer's terminal — minimum friction is the point.
Prototype cases in scope
2 of the 61 resolved cases exist in this demoAgent: WFM Assistant · Workforce Management · Google ADK