LGTM@K is a joke metric name from a Vespa.ai talk on evaluating information retrieval systems alongside LangChain. It stands for “looks good to me @ k,” a play on serious reporting conventions such as Precision@K. Speakers listed LGTM@10 on a slide the way you might list nDCG or MRR. A few people in the audience nodded before realizing it was not a real formula. The name stuck because it captures a habit teams still fall into: eyeballing the top few search results, declaring them fine, and shipping.
LGTM@K is not a metric you should report. It is a warning label for subjective retrieval review. When someone says the retriever “looks good” on ten queries, you still do not know whether relevant documents rank high, whether bad results crowd out good ones, or whether performance holds on the long tail of production traffic. The joke lands because the gap between confident human spot checks and measurable ranking quality is wide.
If you own RAG or search quality, treat LGTM@K as the anti-pattern and reach for ranked retrieval metrics with explicit definitions instead.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- LGTM@K originated as humor in an IR evaluation talk: “looks good to me @ k,” not a mathematical measure.
- Subjective review of a handful of top results hides ranking errors that hurt RAG and search in production.
- Precision@K counts how many of the top k results are relevant; it is simple and strict about the cutoff.
- nDCG rewards putting highly relevant documents higher in the list, not just including them somewhere in the top k.
- MRR focuses on how soon the first relevant result appears, which matters when users or models stop after one good hit.
Why the joke resonates
Labeling query-document pairs takes time. It is tempting to run ten familiar questions, glance at the top three chunks, and move on. That optimizes for your memory of what should match, not for what users actually ask. IR talks introduce Precision@K, nDCG, and MRR so teams write down what “good” means and compare runs with numbers another reviewer can repeat. LGTM@K names the shortcut everyone recognizes.
RAG pipelines combine retrieval with generation. A model can sound authoritative while citing the wrong passage. If evaluation stops at “looks good to me,” you will tune prompts before you fix chunking, embeddings, or ranking that lost the right document.
What to measure instead of LGTM@K
Precision@K answers a blunt question: of the top k results, what fraction are relevant? If k is 5 and three results are relevant, Precision@5 is 0.6. It ignores order within the cutoff. Two rankings with the same relevant set in the top k score identically. That makes it easy to explain and cheap to label if relevance is binary.
nDCG (normalized discounted cumulative gain) adds graded relevance and cares about rank. Highly relevant documents earn more credit when they appear near the top. A relevant item at position five counts less than the same item at position one. nDCG is the better default when some answers are partially useful or when rank order inside the top k changes what the LLM reads first.
MRR (mean reciprocal rank) emphasizes the first hit. For each query, take the reciprocal of the rank at which the first relevant document appears. If the first relevant result is at rank 3, the score is 1/3. Average across queries for MRR. This fits workflows where the model or user mostly consumes the top result, or where finding one good document is enough.
None of these replace human judgment entirely. They constrain it. You still choose relevance labels and k to match your pipeline. You just stop pretending that ten informal checks equal an evaluation set.
How this shows up in RAG workflows
RAG quality splits into retrieval and generation. LGTM@K fails at retrieval because it has no protocol. Two engineers can disagree whether the third of five chunks is relevant. Metrics force a label schema and a cutoff at the k your pipeline actually passes to the model.
Build a labeled query set from real traffic, not only demo questions. Score retriever changes when you swap embeddings, chunk sizes, or hybrid weights. Iteration on retrieval labels and ranking moved recall sharply in a focused project described in how Arize Skills improved RAG recall. Measure retrieval before you declare victory on the full answer.
Run evals repeatedly, not only once before launch. See LLM and agent evaluation platforms for platform context. For where ranking fits relative to chunking, see introduction to retrieval augmented generation.
Failure modes that LGTM@K hides
Manual review overweights the first result. Demo queries overfit your index. Reviewing three hits while the model reads eight misstates quality. High MRR with low Precision@5 can mean one good hit plus noisy context. Pick metrics that match how your pipeline consumes results, and use nDCG when partial relevance matters.
FAQ
Should I ever use LGTM@K in a report?
No. Use it in conversation to call out informal review. In reports, use Precision@K, nDCG, MRR, or recall at the k your system uses, with labels defined upfront.
Which metric should I pick first for RAG?
If your pipeline passes a small k to the model and relevance is mostly binary, start with Precision@k at that k. If rank order inside the list strongly affects answers, add nDCG. If only the first good document matters, track MRR as well.
How many labeled queries do I need?
Enough to cover major intent clusters and failure modes you have seen in logs. There is no magic number. A few hundred diverse labeled queries beats thousands of duplicate phrasing. Refresh labels when products and policies change.
Can LLM-as-judge replace retrieval metrics?
Not entirely. Judges can score final answers and sometimes estimate relevance, but ranked retrieval metrics stay cheaper, more stable, and closer to the retriever component. Use both layers when RAG fails in production.
Why did LGTM@K spread beyond the original talk?
Because every retrieval team recognizes the behavior it mocks. Naming the anti-pattern makes it easier to push for labeled eval sets, regression tests on ranking changes, and traces that show which documents actually entered the prompt.
