Eval authoring studio

New eval

Declare on the left, and the eval.yaml (center) and the per-backend compiled artifacts (right) regenerate live — one causal chain, always visible. No steps: every pane is editable in any order.

Demo beats····guidance, not steps — nothing gates

Case identity

the scenario is fixed — the declarations are yours

wfm.timeoff_over_cap_016

Year-end time-off over the carryover cap (CA, part-time)

WFM Assistant·golden · employee

“Take Dec 29–31 off for me — will the rest of my PTO carry over?”

Concern declarations

declaring a concern is the only wiring step

Declaring a concern is the only wiring step — each one pulls in its default pack and routes to its backend. The compiler slices the case; nothing is scored twice. Routed now: functional@v3 · retrieval@v2 · numeric@v1 · action@v1 → 4 backends.

Golden data

what a correct answer looks like — typed, not prose

Expected tools

1.get_employee_context
2.get_accrual_balances
3.project_carryover
4.check_schedule_conflicts
5.submit_time_off_request

Must cite

POL-0233 §4 — California PTO carryover

Expected output

Your Dec 29–31 request (15.0 h) is submitted for approval. That leaves 27.0 h of PTO — all of it carries over. Without this time off, 4.5 h would have exceeded California's carryover cap.

Pinned by your declarations

Action

Sandbox asserts exactly one pending time-off request exists after the run; teardown resets only the overlay's scope (§25).

Golden truth

derived by the engine, or declared by an SME (§26)

carryover_hours = 27.0 horacle: ukg-accrual-engine

Fixture / dataset binding

bound by logical handle — never a raw substrate ID (§23)

unbound

substrate Calendar@2026.2

as_of 2026-12-28

tenant acme-mfg-oh (sandbox, synthetic)

Nothing bound yet — pick a library handle or mint a new declaration. Until then the compiled artifacts can’t run (see the readiness rail).

Evaluator bindings

inherited from declared concerns; click the catalog to add extras

Inherited — from kind: [functional, retrieval, numeric, action]

functional@v3tool_trajectory · response_semantic→ ADK evaldeterministic· trusted by construction

why: core — five tools must run, in order

retrieval@v2groundedness · context_precision · context_recall→ RAGASdeterministic· trusted by construction

why: must_cite present

numeric@v1numeric_exact→ UKG enginedeterministic· trusted by construction

why: regulated figure — engine of record (§9)

action@v1side_effect_assert · confirmation_gate→ UKG Action Harnessdeterministic· trusted by construction

why: the request writes to the schedule (§10)

Evaluator catalog

the list you know from Langfuse — the difference is after you click

Metricsmath — run immediately · deterministic

LLM judgestemplates — instantiate in shadow (§32) · non-deterministic

Readiness

what must be true before this can run

1 blocking

No dataset configured — bind a fixture before this can run.

Advisory — will happen, doesn’t block

Derived truth: the engine computes 27.0 h and auto-pins it; a substrate refresh revalidates it automatically (§26).
Drift is corpus-level — Evidently watches this case's traces in Live (§55); never gated per case.

Resolve the blocking prerequisite to open a PR.

eval.yaml

generated live — change a declaration and this regenerates

13 lines

This is everything the team writes — even with every concern declared it stays ~15–20 lines. Graders, routing, and thresholds are inherited.

evals/cases/wfm.timeoff_over_cap_016.yaml — authored

case: wfm.timeoff_over_cap_016
agent: wfm-assistant
kind: [functional, retrieval, numeric, action]
tier: golden
persona: employee
input: "Take Dec 29–31 off for me — will the rest of my PTO carry over?"
subject: « bind a fixture — no dataset configured »
as_of: 2026-12-28
golden:
  tools: [get_employee_context, get_accrual_balances, project_carryover, check_schedule_conflicts, submit_time_off_request]
  must_cite: [POL-0233 §4]
  numeric: { carryover: 27.0h, oracle: ukg-accrual-engine, truth: derived }
  action: { writes: time_off_request, confirm_before_write: true }

Compiled output

per backend · always in sync

4 blocked

deterministic— trusted by construction

FNwfm.timeoff_over_cap_016.evalset.json→ ADK eval

blocked: no dataset configured — required before this can run

RTeval_timeoff_ragas.py→ RAGAS

blocked: no dataset configured — required before this can run

NMcarryover_oracle_check.json→ UKG engine

blocked: no dataset configured — required before this can run

ACside_effect_assert.json→ UKG Action Harness

blocked: no dataset configured — required before this can run

non-deterministic— trusted via calibration

No LLM judge bound — the inherited packs are all deterministic, trusted by construction. Pick a judge from the catalog to add a calibrated (or shadow) evaluator here.

Compile log0/4 written · 4 blocked

✕ blocked · ADK eval · wfm.timeoff_over_cap_016.evalset.json

✕ blocked · RAGAS · eval_timeoff_ragas.py

✕ blocked · UKG engine · carryover_oracle_check.json

✕ blocked · UKG Action Harness · side_effect_assert.json

The other way cases are born: promotion from a production trace — trace inbox. Both origins end as PRs; the case detail records which one it was (§7).