SIGNAL ISSUES
Critical2
High4
Medium5
Low3
support-agent 14
Order status replies include another customer's email in 3% of responses Critical · 4 traces Signal
refund_order is called without confirmation on cancellation requests Critical · 3 traces Signal
Task Completion eval fails when lookup_order returns an empty result High · 1 trace Signal
Agent loops on search_kb when the query includes an order number High · 5 traces Signal
Escalation handoff drops ticket context before reaching a human High · 2 traces Signal
Agent states the wrong return window for international orders High · 3 traces Signal
Answer Grounding eval fails on 20% of retriever spans Medium · 2 traces Signal
Tone eval flags curt replies on repeat contacts Medium · 6 traces Signal
get_shipping_status is retried 3 or more times per session Medium · 4 traces Signal
Billing and refund tickets get the wrong intent label Medium · 3 traces Signal
Final answer ignores tool output when knowledge base results are stale Medium · 2 traces Signal
search_kb spans show recurring latency spikes above 2 s Low · 2 traces Signal
Long system prompt inflates token cost on simple greetings Low · 8 traces Signal
Replies repeat the same apology across turns Low · 3 traces Signal

refund_order is called without confirmation on cancellation requests

Critical · Signal · support-agent · 3 traces · Nov 12 to Nov 13
Create Evaluator
Add to Dataset
GitHub
OVERVIEW When a customer asks to cancel, the agent calls refund_order before asking the customer to confirm. Policy requires an explicit yes before any refund. EVIDENCE
  • 3 of 41 cancellation sessions issued a refund with no confirmation turn
  • Refund amounts in these traces ranged from $18.00 to $64.00