Safety evaluation measures whether an AI system avoids harmful, unsafe, unauthorized, or policy-violating behavior. It is a category of tests rather than a single metric, and it usually spans toxicity, self-harm content, dangerous instructions, data leakage, jailbreak and prompt injection resistance, unsafe tool calls, and whatever domain policy your product has to hold to.
For agents, safety evaluation has to include actions, not just text. The harmful behavior may never appear in a response. It may be a tool call, a permission escalation, a file write, a transaction, or a read of data the user was not entitled to see. Grading the final message of a session where the agent quietly modified a record measures the wrong artifact.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Safety evaluation is a set of categories, not one score. Each harm type needs its own cases, scorer, and threshold.
- Safety sets are adversarial by construction. Sampling representative traffic will not surface behavior that only appears under pressure.
- For agents the unit under test is the action, so tool calls, permissions, and data access belong in the suite alongside the text.
- Red teaming finds new failure classes. The suite keeps the ones you already found from coming back.
- Passing a safety suite means the system did not fail the attacks you thought of. That is evidence, not a guarantee.
What a safety evaluation covers
There is no canonical list, and the useful one is specific to your product. Most include:
- Harmful content. Toxicity, harassment, self-harm, and anything your policy prohibits regardless of who asked.
- Dangerous assistance. Output that would help someone cause physical, financial, or security harm.
- Data exposure. Secrets from the system prompt, another user’s records, or personal data that retrieval pulled into context.
- Adversarial resistance. Whether the system holds against jailbreak attempts from users, and against prompt injection carried inside content it ingests.
- Unsafe actions. For anything with tools: writes that should have been reads, calls outside the user’s permissions, irreversible operations taken without confirmation.
- Domain policy. The rules that only apply to you. A medical product, a lending product, and a game share almost nothing here.
Write the list before the tests. The list is your scope claim, and anything not on it is not being measured.
Building the set
Safety sets are built differently from quality sets. A quality set tries to look like production traffic. A safety set deliberately does not, because the behavior you are hunting is rare, deliberate, and provoked.
Three sources feed one. Real incidents come first: every escape, near miss, and support complaint becomes a permanent case, which is how you learn a fix stayed fixed. Published adversarial benchmarks buy breadth cheaply. Generated variants extend both, with a person confirming each label.
Keep a benign control set beside it. A model that refuses everything scores perfectly on attack resistance and is useless as a product, so measuring refusal rate on legitimate requests in the same run is what stops a safety fix from shipping as a regression. Evaluation design tends to fail before the evaluation runs, usually because nobody defined the failure precisely enough to label it consistently, and safety sets are where a vague label does the most damage.
Red teaming as a category
Red teaming is structured adversarial probing, by people or by systems acting as attackers, aimed at finding behavior nobody anticipated. It is not the same activity as running a safety suite. Expert manual probing is slow and finds the deepest problems, because a domain expert knows what a plausible-sounding wrong answer costs in their field. Broad internal or crowdsourced exercises trade depth for coverage. Automated adversarial generation scales, and it explores close to what it has already seen.
All three produce the same deliverable: new cases for the permanent suite. A red team exercise whose findings never become regression tests is a report, and reports do not stop the next release from reintroducing the bug.
Before and after release
Pre-release, safety evaluation is a gate. The suite runs on every change to the model, prompt, retrieval configuration, or tool set, because any of those moves behavior. Zero tolerance is the normal setting: one confirmed harmful output blocks, rather than averaging into a pass rate.
Post-release it splits in two. Guardrails run inline under a latency budget. Offline evaluation runs over sampled traffic and answers what guardrails cannot: whether attempts are climbing, which phrasings get through, and whether one segment is being over-blocked.
Both depend on the behavior being recorded. Reconstructing what an agent did from its final answer is guesswork, while tracing the tool calls and intermediate steps gives the evaluation something concrete to score. The controls around the model matter as much as the weights: what the agent may call, with which credentials, and where a human sits in the loop are properties of the harness rather than the model, and they are the cheapest place to bound a failure you cannot fully prevent.
What passing does not tell you
A safety suite measures the attacks you thought of, on the cases you wrote, with the scorers you chose, against the version you tested. It is a lower bound on your failure rate.
Three limits belong in any honest report. The set ages, because adversarial technique moves and last quarter’s set tests last quarter’s attacks. The scorer is fallible, because a judge deciding whether a response was harmful is itself a model with blind spots. And absence of evidence is not evidence of safety, particularly for rare high-severity behavior, where a few hundred cases cannot detect a failure rate that still matters at production volume.
That is not an argument for skipping the work. It is an argument for reporting it plainly: this suite, these categories, this date, this pass rate, these known gaps.
FAQ
What is the difference between safety evaluation and quality evaluation?
Quality evaluation asks whether the system did the job well: correct, grounded, relevant, complete. Safety evaluation asks whether it did something it should never do. The two get different thresholds. A quality metric can slip a few points and still ship. A confirmed safety failure is a block, not an average.
How do I build an AI safety eval set?
Start from your own incidents and complaints rather than a generic benchmark, because those are the failures your product actually produces. Define each harm category precisely enough that two people label the same output the same way. Add published adversarial cases for breadth, generate variants of anything that worked, and keep a benign control set so you can see over-refusal. Then keep every case permanently.
Is red teaming the same as safety evaluation?
No. Red teaming is exploratory and finds failures nobody anticipated. Safety evaluation is repeatable and confirms that known failures stay fixed. They feed each other: red team findings become suite cases, and gaps in the suite tell the red team where to look next.
How often should safety evals run?
The full suite runs on every change that can move behavior: model version, system prompt, retrieval configuration, tool definitions. A faster subset can run per commit. Separately from releases, run against sampled production traffic on a cadence, because what users try changes even when your system does not.
Can a safety evaluation prove my system is safe?
No. It can show that a defined set of behaviors did not occur under a defined set of tests. Coverage is partial, adversaries adapt, and the scorer has its own error rate. Report what was tested and when, and assume unfound failures are present rather than absent.
