Evaluation-driven development is the practice of writing the evals for an LLM or agent feature before or alongside the feature itself, then letting those eval results decide what you change next. The name borrows from test-driven development on purpose: the measurement comes first, and the implementation gets judged against it rather than against whoever last read the output and thought it looked fine.
This is emerging vocabulary, not settled industry terminology. Teams have worked this way for about as long as they have shipped LLM features, usually without agreeing on a name, and you will see the same practice called eval-first development or simply keeping evals in the loop. The practice is real and specific; the label is still being argued over, so expect to define it when you use it in a design doc.
The reason to adopt it is narrow and practical. Prompt and agent development has almost no natural feedback signal. You change a system prompt, read three outputs, decide it is better, and ship. Two weeks later a different reviewer reads three different outputs and decides it got worse. Neither of you measured anything. An eval set turns that argument into a number that at least moves for a reason.
Key takeaways
- Write the cases and the scorer before or while you build, so the definition of “working” exists before you start tuning against your own taste.
- The analogy to test-driven development holds for ordering and breaks for determinism. You cannot write a failing eval and then make it pass reliably, and the eval itself may be the thing that is wrong.
- Failures found in production become permanent cases. That growth is what separates a living eval set from a one-time benchmark.
- Cost and runtime shape the cadence. Cheap deterministic checks run constantly, judge-based scoring runs on a sampled or scheduled basis.
- The output of the loop is a decision, not a dashboard. If a score moves and nothing ships or blocks, you are collecting metrics rather than practicing this.
How the loop actually runs
Collect cases before writing the prompt. Pull real inputs if you have them: support transcripts, search logs, the questions people already ask a human. If the feature is new, write 20 to 30 cases by hand from what you expect users to do, and mark which ones look hard. A set assembled from imagination is weaker than one assembled from traffic and much better than nothing.
Define the scorer next, in writing. For each case, state what a good response looks like specifically enough that two engineers would grade it the same way. That sentence is usually the first draft of a judge prompt or a deterministic check. This step carries most of the value, because it forces the team to notice they disagree about the requirement before anyone writes code.
Build, run the set, read the failures. Not the aggregate. The individual failing cases, with the full trace of what the system did. Aggregates tell you the direction. The failures tell you the fix.
Feed production back in. Every real failure a user finds becomes a case in the set, which means the set converges on the shape of your actual traffic over time. This is the part that makes the practice compound, and it is the mechanic behind moving an agent from a working pilot to something you can operate in production. It only works if a failing session is easy to find and easy to turn into a case, which in practice means traces and eval scores stored against the same records rather than in two systems nobody joins.
Where the analogy to TDD breaks
Be honest about this, because teams that expect a red-green-refactor rhythm get frustrated and abandon the practice.
You cannot write a failing eval and make it pass deterministically. A unit test that goes green stays green. An eval score is a distribution. The same prompt scored twice returns different numbers, so “it passes” becomes “it scores 0.87 plus or minus something you have to measure.” Thresholds replace assertions, and a one-point move on a 50-case set is usually noise.
The eval can be wrong. In TDD the test is the specification. Here the scorer is software you wrote quickly, often a judge prompt, and it has its own failure modes: it rewards verbosity, it misses a subtlety the rubric never mentioned, it marks a correct answer wrong because the phrasing differs from the reference. When the eval and your judgment disagree, the eval is a suspect, not an authority. Sometimes the right move is to fix the scorer.
You cannot enumerate the input space up front. Test-driven development works partly because a function has a knowable contract. A text input has no boundary. You will always be writing cases in response to failures you did not anticipate, which is why the set grows rather than converging on complete.
The loop is slow and it costs money. A judge-scored suite takes minutes and real spend per run, so running the whole thing on every save is not an option. Teams tier it: deterministic checks such as schema validity and forbidden-content assertions on every commit, the core judged set per pull request on a sample, the full set before a release. Choosing those tiers sensibly means knowing where evaluation spend actually accumulates, because judge calls are only one line of the bill.
What it is not
It is not the same as continuous evaluation, which is what runs against live traffic after release. Evaluation-driven development is the development discipline that decides what gets built and what ships, and it uses both offline sets and production signal. It is also not a release gate. Gating is one possible action at the end of the loop, and you can practice this without any automated gate.
FAQ
Is evaluation-driven development just TDD for AI?
It copies the ordering and not the mechanics. Writing the check before the implementation is the shared idea. Everything downstream differs: outputs are graded on a scale by an imperfect scorer instead of asserted exactly, results vary between identical runs, the input space cannot be enumerated, and the check itself is under active development alongside the feature.
How many eval cases should I start with?
Enough to cover the paths you know about and few enough that you will actually read the failures, which for most features means 20 to 50. Small and curated beats large and unexamined. Grow the set from real failures rather than by generating variations of cases you already pass.
What do I do when the eval says pass and users complain?
Treat it as a gap in the eval set and go find the case it is missing. That complaint is now your highest-value case. A suite that stays green while the product degrades is the normal failure mode of this practice, and the fix is always more representative inputs rather than a stricter threshold on the inputs you already have.
Can I practice this without an LLM judge?
Yes, and it is worth starting there. Schema validation, required-field checks, forbidden strings, tool-argument validity, and exact match on cases with a known answer are cheap, deterministic, and catch a real share of regressions. Add judges when you need to grade something a rule cannot express, such as groundedness or tone.
