Toxicity measures whether an output contains abusive, hateful, harassing, or otherwise harmful language. It is scored with classifiers, LLM judges, policy models, human review, or some combination, and it is usually applied both to what a user sends in and to what the model sends back.
Treat toxicity as a safety signal, not a quality score. A perfectly polite answer can be wrong, ungrounded, or in violation of your policy, and toxicity will not notice any of that. The metric answers one narrow question about the language of a response. Its value comes from being narrow, and most of the trouble teams have with it comes from expecting it to be broader.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Toxicity scores the language of an output, not its accuracy, its policy compliance, or its fairness. Those are separate evaluations.
- The threshold is a product decision, not a technical default. It sets the trade between letting harmful text through and blocking legitimate text.
- Toxicity classifiers inherit the distribution they were trained on and degrade on domains, dialects, and languages that distribution did not include.
- Reclaimed slurs, quoted abuse, clinical language, and counter-speech are the cases where automated scoring is least reliable and human review matters most.
- Over-blocking has a real cost that rarely appears on a dashboard, because blocked-but-harmless requests leave as silent user failures.
What toxicity evaluation actually measures
Most toxicity scoring returns a probability, sometimes broken into subcategories such as identity attack, insult, threat, or sexual content. A classifier produces that score from the text alone. An LLM judge produces it by reading the text against a rubric you wrote.
The two differ in ways worth knowing before you pick one. A dedicated classifier is fast and cheap enough to run inline on every request, and it is consistent: the same input gets the same score. A judge is slower and costlier, but it can read context, follow a policy definition specific to your product, and explain itself, which matters when someone appeals a block.
Neither reads intent reliably, and intent is what separates abuse from its lookalikes. The same sentence can be an attack, a quotation of an attack in a moderation report, a line of fiction, or a user describing what was said to them. Text-only scoring sees one string in all four cases.
Why the threshold is the hard part
A toxicity score is continuous and the decision is binary, so somebody has to pick a cut point. That choice is the actual policy, and it does not have a correct answer.
Set the threshold low and you block more harmful content and also more legitimate content: medical questions using anatomical terms, a support ticket quoting the abuse a user received, a community using reclaimed language among themselves. These false positives are expensive and mostly invisible. A blocked user does not file a ticket saying “your classifier misjudged me.” They leave.
Set the threshold high and harmful output reaches users, which is the failure everyone measures because it is the one that gets screenshotted.
Two things make this tractable. First, choose the threshold from your own score distribution, on your own traffic, rather than accepting a vendor default that was tuned on a different population. Second, use bands instead of one line: auto-allow below a floor, auto-block above a ceiling, route the middle to human review. Most of the disagreement in any toxicity dataset lives in that middle band, and sending it to people is more honest than pretending a fixed cut resolves it.
Cultural context and classifier drift
Toxicity is not a property of a string. It depends on who is speaking, to whom, and where. Annotators from different backgrounds disagree on the same text, and that disagreement is not noise to be averaged away. It is the actual signal that the category is contested. When a model is trained on labels from one annotator population, it encodes that population’s judgments, including their blind spots.
This is why a classifier that scores well on its benchmark can misfire on your traffic. A model trained largely on public forum comments will over-flag dialects underrepresented in its training labels, and clinical or legal text where explicit terminology is required. It will under-flag coded harassment its training data never contained, and it will be weakest in languages that made up a small share of the corpus. None of this shows up until you check the score distribution by segment on your own data.
Adversarial pressure makes it worse. Users who want to get harmful content out of a model learn what the filter catches and rephrase around it, which is the same dynamic that makes jailbreaking and guardrails an ongoing effort rather than a fixed configuration. A toxicity filter that has not been re-checked against recent traffic is being evaluated on last quarter’s attacks.
Because disagreement concentrates in the borderline cases, running several judges as a jury is a useful diagnostic. Where the judges split, your policy definition is ambiguous, and that is a rubric to rewrite rather than a model to swap. It is also worth deciding deliberately whether a judge belongs in this loop at all, since a cheap classifier inline plus a judge on the ambiguous middle usually beats a judge on every request.
FAQ
What is the difference between toxicity and bias?
Toxicity is about harmful language in a specific output: is this text abusive, hateful, or harassing. Bias is about systematic differences in how a system treats groups, which only becomes visible across many outputs compared against each other. A model can produce no toxic language while still giving noticeably different quality of service to different groups, and only the second is a bias problem.
Is toxicity the same as a safety evaluation?
No. Toxicity is one signal inside safety evaluation. A full safety picture also covers prompt injection, data leakage, dangerous instructions, self-harm content, and policy violations that involve no offensive language at all. Reporting a low toxicity rate as if it means the system is safe overstates what was measured.
Should I run toxicity as a guardrail or as an offline eval?
Both, for different reasons. Inline it is a guardrail that blocks or rewrites before a user sees the output, so it has to be fast and it will use a classifier. Offline it is an eval over sampled traffic that tells you whether the rate is moving, which cases cluster near the threshold, and whether the guardrail is over-blocking a particular segment.
How do I know if my toxicity classifier is wrong for my domain?
Label a sample of your own traffic by hand, with more than one annotator, and compare against the classifier’s scores. Look at false positives and false negatives separately, and break both down by language, dialect, and topic. Aggregate agreement can look fine while one segment is being flagged at several times the rate of the rest.
Can toxicity scores be compared across providers?
Not directly. Each scorer has its own category definitions, its own labeled data, and its own calibration, so a 0.7 from one is not a 0.7 from another. Comparisons are only meaningful on a shared labeled set that you scored with both.
