A scoring function is the logic that turns an input, output, context, trace, or trajectory into a score or label. It can be deterministic code, an LLM judge, an embedding similarity calculation, a rules engine, or a human annotation workflow.
It is the executable half of an evaluation. The criteria describe what good means in words; the scoring function is the thing that actually runs, consumes an example, and emits a value your pipeline can aggregate, alert on, and compare across releases. If the criteria are the specification, the scoring function is the implementation, and it should be treated with the same care you would give any other piece of production code.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- A scoring function has a contract: it takes defined inputs and returns a defined output type, usually a label, a float, or a pass/fail plus an explanation.
- Code-based scorers are cheap, fast, and deterministic. Model-based scorers handle judgment calls that code cannot express, at the cost of latency, spend, and their own error rate.
- Use code wherever code can do the job. Format validity, schema conformance, and citation resolution never need a model.
- Scorers must be versioned. If the scorer changes, historical scores are no longer comparable, and a quality “improvement” may just be a scoring change.
- A model-based scorer needs its own accuracy number, measured against human labels, before its output means anything.
The contract
Every scoring function should have an explicit signature. Inputs typically include the model output, the original input, and whatever context the criteria reference: retrieved documents, tool call results, or the full span tree for an agent. Outputs are one of a few shapes.
- Binary label.
hallucinatedorfactual,passorfail. Easy to aggregate, easy to alert on, and the right default for most checks. - Categorical label. A small set of named levels matching a graded rubric. Keep the set short enough that every value is actionable.
- Continuous score. A float, usually 0 to 1. Useful for similarity and ranking, risky when produced by a language model asked to invent a number.
- Pairwise preference. Which of two outputs is better. Often more reliable than absolute scoring for subjective quality, and the natural fit when comparing two prompt versions.
Whatever the shape, return an explanation alongside it. A label with no reasoning cannot be audited, and the first thing anyone asks when a score looks wrong is why.
Code-based and model-based scorers
Code-based scorers are ordinary functions. Does the output parse as JSON, does it match the required schema, is every cited document ID one that was actually retrieved, does it contain a pattern your policy forbids. They are effectively free, they run in milliseconds, and they return the same answer every time. Anything a code scorer can check should be checked by a code scorer.
Model-based scorers cover the rest: groundedness, relevance, tone, whether an agent chose a sensible tool. These are the checks where the property is real but no formula expresses it, and deciding when a judge model is the right instrument comes down to whether the property is genuinely subjective or whether someone has not written the deterministic check yet.
Embedding-based scorers sit in between. Cosine similarity against a reference answer is cheap and captures rough semantic overlap, but it cannot distinguish an answer that contradicts the reference from one that agrees with it. Both are about the same topic and land close in vector space.
For high-stakes or ambiguous scoring, one judge can be replaced by several. Aggregating the verdicts of multiple models acting as a jury reduces the influence of any single model’s idiosyncrasies, at proportionally higher cost per example.
Version scorers like code
Scoring functions should be versioned and evaluated like code, because a scorer change silently rewrites the meaning of your metric. If the judge prompt gains a clarifying sentence on Tuesday, Wednesday’s scores measure something slightly different from Monday’s. The chart still connects the points, and it now shows a trend that never happened.
A version identifier should cover everything that can change the output: the prompt template, the model name and version, decoding parameters, the parsing logic, and any thresholds. Store that identifier with every score you write. When a metric moves, the first question is whether the system changed or the scorer did, and that should take seconds to answer.
Model-based scorers have a second source of movement that code scorers do not: the provider can update the underlying model without you doing anything. Re-running a fixed sample through the scorer on a schedule catches that before it becomes a mystery.
Where scoring functions break
Silent defaults on parse failure. A judge returns malformed output, the wrapper catches the error and records a 0, and your quality metric drops for reasons that have nothing to do with quality. Record parse failures as their own status, never as a score.
Nondeterminism. Two runs of the same judge on the same example can disagree. Sample a few examples repeatedly to measure how much your scorer varies with itself, and treat that variance as the noise floor beneath any change you are trying to detect.
Uncalibrated confidence. A model asked for a score from 0 to 1 produces numbers that look precise and are not. If you need graded output, give the scorer named levels from a rubric rather than a continuous range.
Cost that scales with traffic. A model-based scorer running on every span of every request is a second inference workload sitting next to your application, and the hidden costs of evaluation at scale are what push most teams toward sampling, cheaper scorer models, and tiering where a code check runs first and only its flags escalate to a judge.
FAQ
What is the difference between a scoring function and a metric?
The metric is the quantity: faithfulness, tool-call accuracy, toxicity. The scoring function is the specific implementation that produces it for one example. Two teams can track the same metric with different scoring functions and get different numbers, which is why comparing metric values across organizations rarely means much.
Should a scoring function use the same model as the application?
It can, but check it first. A model evaluating output generated by the same model can prefer its own phrasing and structure, which quietly inflates scores. Comparing labels from a different judge model, or from human reviewers, is how you find out whether that bias is affecting you.
How do I test a scoring function?
Like any other function. Build a small set of examples with known correct labels, including the tricky boundary cases, and assert that the scorer produces them. For model-based scorers, add an agreement measurement against human labels and re-run it whenever the prompt or model changes.
Can a human review workflow be a scoring function?
Yes. A queue where reviewers apply a rubric and record a label is a scoring function with a person in the middle. It is slow and expensive per example, which is why it is normally reserved for calibration samples, ambiguous cases, and the ground truth used to validate automated scorers.
Should scores be stored with the trace?
Yes, attached at the span level rather than aggregated away. A score on a span tells you which step failed. A score on a request tells you only that something did.
