Portable Evals/cases

wfm.accrual_balance_007

validv3goldenauthored

Check accrual balances — WFM Assistant (Google ADK)

UKGEPIC/wfm-agents · evals/cases/wfm.accrual_balance_007.yaml@ 88e01b2PR #219 · reviewed

One asset → 2 backends— routed centrally by AAG · each grader runs on exactly one engine, so nothing is scored twice

portable-v1 · authored once

wfm.accrual_balance_007

“Check my accruals”

2 concerns · 1 YAML asset · UKGEPIC/wfm-agents

Functional

→ ADK eval

functional@v3 · in-sync

Numeric

→ UKG Deterministic Engine

numeric@v1 · oracle of record · in-sync

The assertion

You have 48.0 hours of PTO, 16.0 hours of sick time, and 8.0 hours of personal time available as of June 29.

Golden context

expected_tools · in_order

get_employee_context → get_accrual_balances

Regulated figures · oracle-checked

pto_balance48.0 h

sick_balance16.0 h

personal_balance8.0 h

Checked against ukg-accrual-engine at tolerance 0 — never an LLM judge.

Compiled → 2 targetswfm.accrual_balance_007.functional.configwfm.accrual_balance_007.numeric.payload all in-sync

Portable eval case

portable-v1

data + graders · thresholds live in suite policy

input

“Check my accruals”

entities

overlay handles — never raw substrate IDs

employees.standard_accruals_oh→ E-30188 · via wfm/accrual-scenarios@v2 on Calendar@2026.2

params

{"as_of":"2026-06-29"}

tenant

acme-mfg-oh (sandbox, synthetic)

expected_tools

order: in_order

1get_employee_context
2get_accrual_balances

expected_output

You have 48.0 hours of PTO, 16.0 hours of sick time, and 8.0 hours of personal time available as of June 29.

numeric

regulated — never an LLM judge

pto_balance	48.0 h	oracle: ukg-accrual-engine	tolerance 0
sick_balance	16.0 h	oracle: ukg-accrual-engine	tolerance 0
personal_balance	8.0 h	oracle: ukg-accrual-engine	tolerance 0

metadata

pillar:wfmtier:goldenregulated:accrualssensitivity: syntheticpersona: employee

datasets

pinned compositions this case belongs to

wfm-accruals-golden

kind

drives grader packs

FunctionalTool trajectory, response match, and semantic regression bulk.NumericExact checks against the UKG calculator of record. Never an LLM judge.

Graders

inherited from AAG packs via kind: — each routed

Functionalfunctional@v3inherited→ ADK eval

tool_trajectoryin-order tool matchExpected tools called, in order, with matching args.

response_semanticLLM judgeProse semantically matches the golden answer.

response_lexicalROUGE-1Lexical overlap with the golden answer.

Numericnumeric@v1inherited→ UKG Deterministic Engine

numeric_exactoracle comparisonExtracted figures equal the oracle's output. Tolerance 0.

Routed, not replicated. Each grader goes to exactly one backend. Nothing is forced onto a backend that cannot express it, and the same case is never scored twice for the same thing.

Routing

concern → backend · centrally owned by AAG

2 targets

Concern	Backend	Why this backend
Functional	→ ADK eval	Native to the agent runtime — owns the regression bulk (trajectory, ROUGE, semantic judge).
Numeric	→ UKG Deterministic Engine	Regulated figures are checked against the calculator of record — never an LLM judge.

Compiled output

2 routed targets

in-sync

Artifacts generated per routed backend (full compiled view shown on the hero case, hr.policy_carryover_004):

FNwfm.accrual_balance_007.functional.config→ ADK evalin-sync

NMwfm.accrual_balance_007.numeric.payload→ UKG Deterministic Enginein-sync