SME/label/t-8801

Shared label pool

cross-team

142

items remaining in the pool

9 assigned to you

Batch completion204/300

204 of 300 items have 2 or more labels

Up next

5 in your queue

1
PTO carryover, wrong jurisdiction
HR Policy agent·2 of 3
2
Overtime calc dispute
Pay agent·1 of 3
3
FMLA eligibility window
HCM agent·0 of 3
4
Shift differential
Frontline agent·0 of 3
5
Garnishment order
Pay agent·0 of 3

Labelers online

7 now

PNMTSFDCANJI+1

Drawn from 4 product teams. Each item is routed to labelers who have not seen it, blind to prior labels. Labeling outside your home team is expected in the shared pool.

Backing label stores

central rails

Argilla
golden datasets · case authoring
Label Studio
consensus, inter-annotator agreement
Confident AI
annotation queues for red-team

Item #t-8801

HR Policy agent

Bryte 2.0 · ADK2.4.1-rcOpen full trace

PII redactedtenant-isolated

redacted at the OTel collector (Presidio) before egress · tenant acme-retail-us (sandbox, synthetic)

trace

7c2e4a..b913

captured

2026-06-27 11:18 UTC

tenant

acme-retail-us (sandbox)

span

gen_ai.invoke_agent

User inputwhat the worker asked

“If I do not use all my PTO this year, how much carries into next year? I am based in California.”

Tool callsORDER REGRESSION

Expected order: get_employee_context then search_hr_policy. Observed order is reversed, so retrieval ran without the worker's state.

observed tool order

// 1. called first, no employee context yet, jurisdiction defaulted
search_hr_policy({ topic: "pto_carryover", jurisdiction: "federal" })
  → retrieved FLSA-FED sec 7   // wrong corpus for a CA worker
// 2. arrived too late to steer retrieval
get_employee_context({}) → { work_state: "CA" }

Deterministic engineCA POLICYpto_carryover_calc

on CA policy: balance 62.5h, cap 40h → carryover 40h, forfeit 22.5h on Dec 31. This is the ground truth for the number. The agent never reached it because it retrieved federal policy.

Agent responseCONTRADICTS ENGINE

“Good news: there is no federal cap on PTO carryover, so all of your unused time rolls into next year. With 62.5 hours unused, you keep the full 62.5 hours and nothing is forfeited. You can use it whenever you like.”

Your label

blind to other labelers

Rubric checklist

Correct jurisdiction applied
worker is CA, agent answered from federal FLSA, not PTO-CA-2026
Carryover cap and deadlines stated
omitted the 40-hour cap, the Dec 31 forfeit, and the Mar 31 deadline
Number matches deterministic engine
agent claimed all 62.5h carry over; engine says 40h carry, 22.5h forfeited
Grounded in a cited CA policy chunk
no citation to PTO-CA-2026 sec 4.2 or 4.3
PII handling
no SSN, bank, or address surfaced; collector redaction held on the output

Note

Labeler: padma.p · Payroll SME · cross-team reviewer · 41 labeled today

Agreement · this item

awaiting yours

2 of 3 labels in · 1 more needed to close

SR
Sofia R.
HCM
submitted
PN
Priya N.
Pay
submitted
PP
You
reviewer
pending

Blind labeling. Peer verdicts stay hidden until you submit. The distribution is revealed only after your label lands, so no one is anchored by another labeler's call.

Batch agreement

human to human

0.74

κ · Cohen's kappa · pairwise mean, human to human

floor0.60

current0.74

strong0.80

Above the 0.60 floor. Below the 0.80 strong gate, so this batch ships labels but is flagged for rubric tightening. Krippendorff's alpha tracks alongside for items with more than two labels.

Labeled204 of 300 items

Full 3 of 3168 items

Tied 1-1-111 items → adjudication

Adjudication

11 queued

Dana W.

Accountable owner · HR Policy

on call

hr-policy-quality@ukg

Tied items route to the accountable owner for the domain. Their call becomes the gold label.

Carryover deadline edge case

t-8772·tied 1-1-1

Accrual cap vs carryover cap

t-8790·tied 1-1-1

Open adjudication queue

Closes the loop. Adjudicated gold labels feed back as candidate goldens for SME sign-off, closing the loop from production to the durable asset.