A test set is a collection of test cases used to evaluate behavior. A test case is one specific example: an input plus the criteria that decide whether the result was acceptable. For an agent, a case usually also carries initial state, the tools that were available, constraints on the expected trajectory, and success criteria for the run as a whole.
Test cases should be small enough to debug and realistic enough to matter. Toy examples earn their place as smoke tests. Production reliability comes from cases that resemble what users actually do, which in practice means cases promoted from traffic you already served.
The set is where coverage lives. The case is where precision lives. Most suites that produce numbers nobody trusts are failing at the case level: vague criteria, one case testing three things at once, or a failure that tells you nothing about which part broke.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- A case is one behavior with a decidable outcome. If a failure cannot tell you what to fix, the case is too broad.
- A useful case carries more than input and expected output: a stable ID, fixtures for initial state, available tools, assertions, metadata, and a pointer to the trace it came from.
- For agents the expected trajectory is part of the case. Scoring only the final answer cannot tell a correct result reached properly from one reached by luck.
- Express trajectory expectations as constraints rather than one exact path. Required tools, forbidden tools, ordering rules, and a step budget survive prompt changes; a nine-step script does not.
- Mix deterministic assertions with judged criteria. Schema validity, forbidden phrases, and tool argument checks are cheap, unambiguous, and catch a surprising share of regressions.
What one test case contains
A case is a record with named fields rather than a row with two columns.
- Case ID. Stable and never reused. It is how a failure this week is compared to the same case last week.
- Input. The user request, and for multi-turn behavior the conversation prefix leading to the turn under test.
- Initial state and fixtures. Account status, permissions, locale, cart contents, whatever the system reads that is not in the message. A case that passes only when a seeded account happens to exist will fail later for the wrong reason.
- Available tools and context. Which tools were registered and which documents the retriever could see. Adding one tool changes selection behavior on unrelated tasks, so the inventory is part of the test conditions.
- Expected result or criteria. A reference answer where one genuinely exists, otherwise the criteria a judge or a rule applies.
- Assertions. The specific checks that decide pass or fail, listed explicitly.
- Metadata. Intent, segment, language, difficulty, failure category, owner.
- Provenance. The
trace_idorspan_idthis case was promoted from.
Provenance is skipped most often and saves the most time. A case built from a real request arrives with the retrieved context and the tool calls that actually happened.
Agent cases carry the trajectory, not just the answer
Scoring the final response is enough for a single model call, not for anything that takes steps. Two runs can produce the same correct sentence after one clean tool call or after four failed calls, a retry loop, and a lucky guess, and only one of those belongs in production. Cases that catch this are looking for the failures ordinary software tests never see, where the output is fine and the path is not.
So an agent case specifies path expectations as constraints:
- Tools that must be called, such as a balance lookup before any refund statement.
- Tools that must not be called, such as a write action on a read-only request.
- Ordering rules where order carries meaning, like verifying identity before disclosing account details.
- A step or token budget, since a run that resolves correctly in thirty steps is a cost failure.
- Required recovery behavior when a tool returns an error: retry, fall back, or escalate.
Write these as assertions over the trace rather than as constraints, not a literal expected sequence. Record tool responses as fixtures so the case measures the agent, not downstream APIs.
Assertions: what the case actually checks
Deterministic checks first, because they are cheap and unambiguous. Valid JSON against a schema. A required field present. A currency value matching the ledger. No forbidden phrase in the output. A tool called with arguments that parse. These catch the regressions that break integrations, and they never disagree with themselves.
Judged criteria handle open-ended quality and need calibration before they gate a release.
Designed retrieval cases, in the style of RAG evaluation with LLM evals and benchmarking, are worth writing when traffic cannot expose a capability directly.
Organizing the set
Most teams end up with three tiers: a small smoke set on every commit, a regression library holding one case per past incident, and a representative set that mirrors production traffic. Running them as repeatable tasks over a versioned set is what makes results comparable, and it is why teams wire LLM evaluations into CI/CD pipelines rather than leaving them as ad hoc scripts. Two habits keep a suite trustworthy: quarantine flaky cases instead of tolerating them, since a case that fails randomly trains everyone to ignore red, and review the cases that always pass, because a case nobody has failed in six months is no longer providing information.
FAQ
How do I write a test case for an agent?
Start from a real trace. Take a session that mattered, keep the user request and initial state, record what each tool returned so the run is reproducible, then write down what the agent should have done as constraints: required tools, forbidden tools, the step budget, and what the final answer must contain. Score path and outcome separately so a failure tells you which one broke.
How many test cases does a suite need?
There is no target number. The requirement is that every behavior you would be upset to break has a case, and every slice you report on has enough cases for its number to be stable between runs. Suites grow from incidents, so a mature one is mostly a record of what has gone wrong before.
How do you handle nondeterminism in test cases?
Reduce it where you can and measure it where you cannot. Replay recorded tool responses, pin model and prompt versions, and lower sampling temperature for cases meant to be exact. For cases that stay variable, run them several times and treat the pass rate as the result rather than demanding a single green run. A case that passes eight times out of ten is information, not noise.
Can I unit test an LLM application the way I unit test code?
Partly. Deterministic assertions behave like unit tests and belong in CI. Judged criteria behave like a measurement instrument: they have variance, they need calibration, and gating on a small score difference produces false alarms. Most working suites run both, with the hard gates on the deterministic layer.
