Weak supervision uses noisy, indirect, or programmatically generated labels to create training or evaluation signal at scale. Instead of a person reading each example and recording a judgment, you write rules, reuse an existing classifier, mine metadata, or prompt a model, and accept that some of the resulting labels are wrong.
The reason to do it is arithmetic. Hand labeling produces high-quality labels at a rate set by how many people you can pay and train, and that rate is usually far below the volume you have. Weak supervision inverts the constraint: you can label everything, and the open question becomes how wrong the labels are and whether the error is random or systematic. Random error mostly averages out with volume. Systematic error does not, and that is where weak supervision goes wrong.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Weak supervision trades label accuracy for volume, generating labels from heuristics, metadata, existing models, or an LLM instead of hand annotation.
- Individual sources are called labeling functions. Each covers part of the data, abstains on the rest, and has its own error rate.
- Correlated errors across labeling functions are the main failure mode. When two functions are wrong for the same reason, their agreement looks like confirmation and is not.
- Any weak label set needs a human-labeled reference set that no labeling function touched, or you cannot measure how wrong it is.
- Weak labels suit triage, coverage, and prioritizing what to hand-label. On their own they are a poor basis for release gates or safety claims.
Where weak labels come from
- Heuristics and rules. Keyword and pattern matching, length and format checks, regular expressions over the output. Cheap, fast, high precision on the narrow slice they cover.
- Existing models. A classifier trained for a related task, or a small model repurposed as a labeler. Its errors come along with its predictions.
- Structure and metadata. A support ticket’s resolution code, whether a generated query parsed and executed, whether a retrieved document came from the section the question was filed under.
- Behavioral signals. A thumbs-down, an immediate rephrase, an abandoned session, a retry, a handoff to a human agent. Indirect, and often the most honest signal you have, since it comes from users rather than from your assumptions. It also has to be recorded to exist, which is one of the quieter arguments for tracing and evaluating agent sessions rather than logging final responses alone. For guidance on testing agent behavior, see agent evaluation.
- Model-generated labels. Prompt a capable model to judge the example. This is the LLM as a judge pattern, and it is weak supervision whenever the output is treated as a label rather than as a measurement you have validated. Pair those labels with prompt learning from production feedback when the goal is to improve the system rather than only score it.
How labeling functions combine
The unit of weak supervision is the labeling function: code or a prompt that looks at an example and either emits a label or abstains. Two properties describe each one. Coverage is the fraction of examples it labels at all. Accuracy is how often it is right when it does.
You then have to resolve the votes into one label per example. Majority vote is simple and treats every function as equally trustworthy, which is rarely true. The better approach estimates each function’s accuracy from how often the functions agree with each other and weights the votes accordingly. The output is usually a probabilistic label rather than a hard class. Keep that confidence: downstream you can train only on the confident portion or route uncertain examples to a person.
Report coverage, overlap, and conflict rate for the whole set. A function covering 2% of the data cannot carry a dataset however precise it is, and two functions that conflict constantly encode a disagreement nobody has resolved.
Correlated errors are the main failure mode
The vote-weighting logic assumes the functions fail independently. They usually do not. Three regular expressions written by the same engineer in the same afternoon share that engineer’s model of the problem. Two functions built on the same keyword list are close to one function counted twice. A rule and an LLM prompt that both look for a refusal phrase will both miss a polite non-answer that never uses it.
When errors correlate, agreement stops being evidence. The aggregator sees three functions concurring and raises confidence, when one blind spot voted three times. The dataset ends up confidently wrong on a structured subset of examples, which is worse than being noisily wrong everywhere, because a model trained on it learns the blind spot as the rule.
The version that bites hardest in LLM work is circularity. If you generate labels with a model, then evaluate that same model against those labels, the evaluation cannot see any failure the labeler shares. The score looks fine and it is measuring agreement with itself.
Validating weak labels
Keep a small, human-labeled, adjudicated reference set that no labeling function and no model in the pipeline helped produce. Everything you can honestly claim about a weak label set comes from comparing against it:
- Per-function precision and coverage. Which functions pull their weight and which contribute noise. Drop the ones near chance rather than letting them dilute the vote.
- Aggregate accuracy, overall and by slice. A label set that is accurate on English and much worse on everything else is a fixable problem that one number hides.
- Whether the confidence means anything. Group examples by the aggregator’s confidence and check that the high-confidence group really is more accurate. If not, the confidence is decoration.
Then decide by consequence. Weak labels suit finding candidate failures, expanding coverage of a rare class, deciding which examples a person reads next, and training a first-pass filter. They are not a sound basis for a release gate, a safety claim, or a published accuracy figure, because there the error you cannot characterize is the whole risk.
FAQ
What is the difference between weak supervision and semi-supervised learning?
Weak supervision gives you labels for a lot of data and those labels are noisy. Semi-supervised learning starts from a small set of trusted labels plus a large unlabeled set, and uses structure in the unlabeled data to extend what the labeled examples teach. The difference is what you are short of: label quality in the first case, label quantity in the second. They are often combined.
Do LLM-generated labels count as weak supervision?
Yes, whenever you use them without measuring them. A model asked to score outputs is a labeling function with unknown accuracy, systematic preferences, and its own blind spots. Measure its agreement with human labels on a reference set and you know where it holds up, which is the point of validation.
How much human-labeled data do I need alongside weak labels?
Enough to estimate accuracy for each slice you care about, normally a few hundred adjudicated examples rather than thousands. Reporting one overall accuracy figure needs less than defending accuracy separately for five customer segments.
How do I detect correlated labeling functions?
Compute pairwise agreement and look for pairs that agree far more than their individual accuracies would predict. Then read the code: functions that key off the same field, share a word list, or derive from the same model are correlated by construction whatever the statistics say. The most direct test is to sample examples where every function agrees and label those by hand.
Does weak supervision replace human labeling?
No, it changes where people spend attention. The productive arrangement is programmatic labels for breadth, human labels for the reference set and the hard cases, and a measured agreement number between the two so you know what the breadth is worth.
