wfm.request_time_off_012
validv4goldenfrom traceRequest time off — transactional write — WFM Assistant (Google ADK)
UKGEPIC/wfm-agents · evals/cases/wfm.request_time_off_012.yaml@ 88e01b2PR #231 · reviewedpromoted from trace tr_2d91c by sme: arthur.w · scrubbed: presidio v2.4
One asset → 3 backends— routed centrally by AAG · each grader runs on exactly one engine, so nothing is scored twice
portable-v1 · authored once
wfm.request_time_off_012
“Request next Friday off”
3 concerns · 1 YAML asset · UKGEPIC/wfm-agents
Functional
→ ADK eval
functional@v3 · in-sync
Numeric
→ UKG Deterministic Engine
numeric@v1 · oracle of record · in-sync
Action
→ UKG Action Harness
action@v1 · in-sync
The assertion
side effect: PTO request row created in sandbox tenant, status=pending, 8h on 2026-07-03 · requires explicit confirmation
Golden context
expected_tools · in_order
get_employee_context → get_accrual_balances → check_schedule_conflicts → submit_time_off_request
Regulated figures · oracle-checked
balance_after40.0 h
Checked against ukg-accrual-engine at tolerance 0 — never an LLM judge.
Compiled → 3 targetswfm.request_time_off_012.functional.configwfm.request_time_off_012.numeric.payloadwfm.request_time_off_012.action.payload all in-sync
Portable eval case
portable-v1input
“Request next Friday off”
entities
overlay handles — never raw substrate IDs
employees.standard_accruals_oh→ E-30188 · via wfm/accrual-scenarios@v2 on Calendar@2026.2schedules.clear_fri_jul3→ no conflicts for 2026-07-03 · via wfm/timeoff-scenarios@v1params
{"today":"2026-06-29"}tenant
acme-mfg-oh (sandbox, synthetic)
expected_tools
order: in_order
- 1
get_employee_context - 2
get_accrual_balances - 3
check_schedule_conflicts{"date":"2026-07-03"} - 4
submit_time_off_request{"date":"2026-07-03","hours":"8"}
numeric
regulated — never an LLM judge
| balance_after | 40.0 h | oracle: ukg-accrual-engine | tolerance 0 |
action
transactional — sandboxed side-effects
must call
submit_time_off_requestrequires explicit confirmationside effect: PTO request row created in sandbox tenant, status=pending, 8h on 2026-07-03
teardown: cancel_time_off_request on teardown
metadata
pillar:wfmtier:goldentransactionalregulated:accrualssensitivity: tenant-realisticpersona: employee
datasets
pinned compositions this case belongs to
kind
drives grader packs
FunctionalTool trajectory, response match, and semantic regression bulk.NumericExact checks against the UKG calculator of record. Never an LLM judge.ActionSide-effect assertions for agents that write to systems of record.
Graders
inherited from AAG packs via kind: — each routedFunctionalfunctional@v3inherited→ ADK eval
tool_trajectoryin-order tool match
response_semanticLLM judge
response_lexicalROUGE-1
Numericnumeric@v1inherited→ UKG Deterministic Engine
numeric_exactoracle comparison
Actionaction@v1inherited→ UKG Action Harness
side_effect_assertsandbox state diff
confirmation_gatetranscript assertion
Routed, not replicated. Each grader goes to exactly one backend. Nothing is forced onto a backend that cannot express it, and the same case is never scored twice for the same thing.
Routing
concern → backend · centrally owned by AAG| Concern | Backend | Why this backend |
|---|---|---|
| Functional | → ADK eval | Native to the agent runtime — owns the regression bulk (trajectory, ROUGE, semantic judge). |
| Numeric | → UKG Deterministic Engine | Regulated figures are checked against the calculator of record — never an LLM judge. |
| Action | → UKG Action Harness | Side-effect assertions need a sandboxed UKG tenant; no OSS framework can express them. |
Compiled output
3 routed targetsArtifacts generated per routed backend (full compiled view shown on the hero case, hr.policy_carryover_004):
FNwfm.request_time_off_012.functional.config→ ADK evalin-sync
NMwfm.request_time_off_012.numeric.payload→ UKG Deterministic Enginein-sync
ACwfm.request_time_off_012.action.payload→ UKG Action Harnessin-sync