Order status replies include another customer's email in 3% of responses
Critical · 4 traces Signal
refund_order is called without confirmation on cancellation requests
Critical · 3 traces Signal
Task Completion eval fails when lookup_order returns an empty result
High · 1 trace Signal
Agent loops on search_kb when the query includes an order number
High · 5 traces Signal
Escalation handoff drops ticket context before reaching a human
High · 2 traces Signal
Agent states the wrong return window for international orders
High · 3 traces Signal
Answer Grounding eval fails on 20% of retriever spans
Medium · 2 traces Signal
Tone eval flags curt replies on repeat contacts
Medium · 6 traces Signal
get_shipping_status is retried 3 or more times per session
Medium · 4 traces Signal
Billing and refund tickets get the wrong intent label
Medium · 3 traces Signal
Final answer ignores tool output when knowledge base results are stale
Medium · 2 traces Signal
search_kb spans show recurring latency spikes above 2 s
Low · 2 traces Signal
Long system prompt inflates token cost on simple greetings
Low · 8 traces Signal
Replies repeat the same apology across turns
Low · 3 traces Signal
refund_order is called without confirmation on cancellation requests
Critical · Signal · support-agent · 3 traces · Nov 12 to Nov 13
Create Evaluator
Add to Dataset
GitHub
OVERVIEWWhen a customer asks to cancel, the agent calls refund_order before asking the customer to confirm. Policy requires an explicit yes before any refund.EVIDENCE
3 of 41 cancellation sessions issued a refund with no confirmation turn
Refund amounts in these traces ranged from $18.00 to $64.00