You are comparing an incumbent search system with a new semantic retriever. You pool candidates from both systems, judge the previously unseen results, update the evaluation set, and calculate the metrics. The new retriever now leads by 3%. How much confidence should you place in that result?
Would the conclusion survive another query sample, a different candidate ordering, another credible judge, or a third retriever contributing previously unseen evidence to the pool?
LLM judges make this kind of evaluation easier to run at scale. They can generate relevance labels, expand incomplete judgment sets, and make offline evaluation cheaper to repeat. But the LLM judge is only one component of the evaluation pipeline; the rest of the pipeline gives the judge context and turns its relevance labels into a score.
The question is whether the winner reflects a real retrieval improvement or choices made elsewhere in the evaluation setup.
For diagnosing where a search failure began, The Search Evaluation Ladder separates indexing, candidate retrieval, ranking, and evaluation failures.
An offline relevance score such as nDCG@10 is produced by an evaluation setup. When an LLM supplies the relevance judgments, the judge is one component of that setup. The queries selected for evaluation, the candidates exposed for judgment, the relevance criteria, the evidence shown to the model, and the way judgments are aggregated all contribute to the final system decision.[1][4]
A simplified view of the pipeline looks like this:
query sample
↓
retrieval runs
↓
candidate pool
↓
document / passage presentation
↓
LLM judge ← relevance criteria / rubric
↓
relevance judgments
↓
metric / aggregation
↓
system decisionA few working terms make the rest of the article easier to read:
- Evaluation setup: the full measurement system, including the query sample, retrieval runs, candidate pool, document presentation, rubric, LLM judge, metric, aggregation rule, and treatment of unjudged documents.
- Candidate pool: the evaluation candidate set: the documents or passages selected for judgment. A judge cannot label useful documents that never enter this pool.
- Relevance reference: the judgments used to score the systems: which query-document pairs have been judged and how relevant they were judged to be. This is often called qrels or a relevance dataset.
- System decision: the conclusion the evaluation supports, such as "retriever A is better than retriever B" or "the migration candidate is good enough to replace the current system."
- Defensible change: a change another competent team could have chosen before seeing which system won, such as a different query sample, another capable judge, or a broader candidate pool.
Even when the judge produces useful relevance judgments, the final system decision can still depend on choices elsewhere in the evaluation setup.
Relevance Reference Drift
Can your benchmark favor the incumbent?
Consider a controlled comparison in which the query set, corpus, and retrieval runs are held fixed while the relevance reference changes:
same queries
same corpus
same retrieval results
different relevance reference
↓
does the system decision change?Relevance Reference Drift describes a change in the apparent quality or ordering of retrieval systems under this intervention. Retrieval has not changed; only the reference used to score it has. If the system decision changes, the conclusion depends on which relevance reference was used.
This matters when the reference was built around a narrower retrieval regime. A semantic retriever may surface relevant documents that lexical search rarely exposed. Refreshing the judged set can therefore change the comparison, depending on which documents enter the pool and which reference is treated as authoritative.
This is a longstanding issue in retrieval evaluation: incomplete judgments and candidate sets shaped by the systems that produced them can influence the resulting comparison.[5][6][7] Newer retrieval methods make the issue more visible because they can surface relevant documents that older systems rarely retrieved.[8] LLMs make it cheaper to add missing judgments, but they do not remove the dependency between the relevance reference and the candidate set used to construct it.
Elastic describes the visible symptom: apparently poor results can turn out to contain useful documents that were never judged.[9]
Migration makes the distinction concrete. Years of judgments from an Elasticsearch or OpenSearch system will tend to reflect documents that system surfaced for assessment. A different retrieval architecture may surface different relevant evidence that the old reference covers poorly. Matching the old reference and serving users well are related goals, but they are not the same goal.
The robustness question is therefore: would the same system decision hold under another defensible relevance reference?
If the answer changes, the original conclusion is narrower. It may still be useful, but it is conditional on that particular reference.
What Happens to Unjudged Documents?
Incomplete judgments raise a practical scoring question: what does the evaluation do when a retrieved document has no relevance judgment?
If unjudged documents are treated as non-relevant, a system can be penalized for retrieving useful material that simply falls outside the relevance reference. Classical work on evaluation with incomplete relevance information showed that standard effectiveness measures can become unreliable as the judged set becomes substantially incomplete.[6]
Different evaluation procedures handle missing judgments differently. Condensed-list evaluation removes unjudged documents before computing the metric, while measures such as Judged@k report how much of the returned ranking has actually been assessed.[10]
Two separate questions are therefore involved:
- Relevance reference: Which documents have been judged, and what did the judgments say?
- Evaluation rule: What does the metric do with documents that have not been judged?
If the winner changes depending on how unjudged documents are scored, the result needs more scrutiny.
Judge–Pool Coupling
What evidence can the judge actually see?
A relevance reference is often built by judging a candidate pool assembled from several retrieval systems:
BM25 \
dense +--> candidate pool --> LLM judge
hybrid /
↓
relevance referenceThe judge can assess only the documents that enter the pool. A document missed by every participating retriever is invisible to the evaluation.
Including the challenger in the pool is therefore necessary, but it does not eliminate pool dependence. It changes the pool whose coverage must now be trusted.
Judge–Pool Coupling describes this dependency: the relevance reference depends on the candidate pool, while the candidate pool depends on the retrieval systems used to build it. The judge may be independent of those systems, but the systems still determine which evidence the judge gets to see.
This is familiar from classical pooling: the systems used to construct a candidate pool can influence the resulting relevance reference and the reliability of later evaluation.[7][8] Vespa makes a similar point for LLM-judging workflows: judgments collected from one retrieval technique should not be reused blindly to evaluate a substantially different technique, because the new technique may retrieve relevant query-document pairs absent from the original labels.[2]
The problem becomes clearer when a substantially different retriever is added:
BM25 \
dense |
hybrid +--> candidate pool --> LLM judge
ColBERT /
↓
relevance referenceIf the new system retrieves relevant documents that the original pool did not contain, those documents can change the relevance reference itself. Poor pool coverage can therefore favor systems that retrieve documents similar to the existing pool, rather than systems whose useful results differ from it.
Two-Level Stability
Does judgment variation change the decision?
Questions about judge consistency often involve two different kinds of stability.
Document stability asks whether individual documents receive similar judgments, or occupy similar positions in a judge-produced preference ranking, across repeated or alternative assessments.
System stability asks whether variation in those judgments changes the system decision.
These are related but distinct. Individual judgments can change substantially while producing little change in which system wins. The reverse is also possible: a small amount of document-level disagreement can change the decision when it occurs on the queries that separate closely matched systems. The amount of disagreement matters less than where it occurs.
This distinction has deep roots in IR evaluation. Voorhees found that substantial differences in individual relevance judgments could coexist with highly correlated rankings of retrieval systems.[5] Recent LLM-judge work makes the same point: individual judgment agreement does not fully describe the stability of the system-level decision.[4][1]
Consider two judging runs that disagree on many individual documents. That disagreement may reveal judge or procedural sensitivity, yet both runs may still support the same system decision after aggregation across queries. Conversely, two runs can agree on almost every document but disagree on a small set of queries that determines which of two closely matched systems wins.
What matters, therefore, is how judgment variation propagates into the decision you actually intend to make. Document-level variation is useful diagnostic evidence because it shows where uncertainty enters the measurement. But if the action is to ship a ranker, replace an embedding model, or migrate a retrieval architecture, system-level stability is the more consequential test.
Practical rule: measure stability at the level at which you act. Report both how much individual judgments vary and whether that variation changes the system decision.
Test Whether the Winner Holds
An LLM-based evaluation is more than its final score. To test whether the winner is robust, vary four parts of the setup: Sample, Order, Judge, and Pool.
Sample
The query sample determines which parts of the search problem contribute to the final metric. Retrieval effectiveness varies by query, so sample size and composition are major sources of uncertainty. Classic TREC analysis quantified how often changing the topic sample could change the system decision.[11]
A sample dominated by navigational or brand queries may conceal the behaviour of a method developed primarily for exploratory or long-tail search. The final metric can similarly obscure variation across markets, languages, query lengths, or other product-relevant slices.
Test: does the measured advantage, and the resulting system decision, survive defensible changes to the query sample?
Order
Presentation becomes part of the judging protocol whenever it can alter the judgment. For pairwise or listwise assessment, candidate position can directly influence preferences; recent work has shown position effects in both protocols.[12] Swapping sides or permuting candidate slates is therefore a natural robustness check.
Pointwise assessment has a different presentation surface, where relevant variables may include rubric ordering, field ordering, title-versus-passage placement, surrounding context, or the serialization of structured metadata.
Test: does changing how the evidence is presented materially change the judgments or the system decision?
Judge
Different changes to the judge answer different questions. Repeat the same judge to test run-to-run variation, use another capable model to test judge dependence, vary the prompt to test instruction sensitivity, and use targeted human audits for consequential or ambiguous cases. Studies of LLM relevance labelling show why these distinctions matter: capable LLM judges can correlate strongly with human system rankings while still exhibiting model-specific biases.[1]
Perfect agreement is usually too strict. The operational concern is whether credible judge variation changes the system decision.
Test: does the winner survive reasonable changes to the judge or judging procedure?
Pool
The candidate pool determines the evidence available for judgment. A useful pool test changes the contributing retrievers and measures whether the relevance reference or existing system decisions move materially.
This is especially informative when a new retrieval architecture is designed to surface a different candidate population from the incumbent systems.[7][8]
Test: does adding candidates from another credible retriever change the relevance reference or the system decision?
- Which queries create the measured gain?
- Which systems contributed candidates to the judged pool?
- How much of the top-k ranking is actually judged?
- Does the result survive another judge, prompt, or evidence presentation?
- Does the same conclusion hold when the challenger contributes new evidence?
If those answers are unclear, the evaluation can still guide exploration. It should not carry a production migration decision by itself.
The appropriate depth of testing depends on the system decision. An inexpensive evaluator used to discard weak ranking ideas has a lower reliability requirement than an offline evaluation used to justify a production migration.
Version the Evaluation Setup
While validating an evaluation setup, deliberately vary defensible inputs to discover its sensitivities. Once the setup is understood, freeze and version the parts that affect measurement: query sample, corpus snapshot, retrieval runs, pool construction, relevance criteria, document representation, ordering policy, judge model and revision, prompt, relevance scale, decoding settings, metric, and treatment of unjudged documents.
Production implementations already treat parts of this setup as versioned infrastructure. DoorDash, for example, describes versioning its relevance rubric and recalibrating against a golden set when evaluation criteria change, while tagging historical measurements with the rubric version used to produce them.[3]
The measured result can move while retrieval remains unchanged. Without versioning, a metric change may reflect retrieval changes, evaluation changes, or an interaction between the two.
When Is the Evaluation Good Enough for the Decision?
A team exploring many ranking ideas may use a fast evaluator to narrow the design space. Some judgment noise is acceptable when the evaluation is an early-stage filter.
But suppose one system scores 3% better offline and that result is being used to justify migrating production retrieval. That is a different standard. When two systems are close, small changes in the evaluation can change which one appears better. And when the decision is expensive or difficult to reverse, a wrong conclusion matters much more.
So two things should determine how much testing you do: how close the competing systems are, and how costly it is to make the wrong decision. The closer the systems are, or the more consequential the decision, the more you should test whether the result holds up.
In practice, you want to know whether you would reach the same decision if you changed reasonable parts of the evaluation: which examples you sampled, how they were presented, which judge evaluated them, or which candidates were included. If small, reasonable changes can flip the winner, you do not yet have strong evidence that one system is actually better.
LLMs make relevance assessment much easier to scale, and they allow more of the evaluation process to be implemented in software. That means the evaluator itself should be treated like an engineered system: control its inputs, measure what it is sensitive to, version its configuration, and check whether its conclusions survive reasonable changes to the evaluation setup.
At Searchplex, this is part of relevance engineering: inspect the evaluation setup, identify where the measured gain is sensitive, and separate retrieval improvement from measurement artifact before a team commits to a migration or ranking change.
References
On the Use of LLMs for Relevance Labelling. ACM Transactions on Information Systems
Improving retrieval with LLM-as-a-judge. Vespa Blog
LLM-as-a-Judge: Evaluating natural language search. DoorDash Engineering
JuStRank: Benchmarking LLM Judges for System Ranking. ACL 2025
Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. Information Processing & Management
Retrieval Evaluation with Incomplete Information. SIGIR 2004
Bias and the Limits of Pooling for Large Collections. Information Retrieval
On the Reliability of Test Collections for Evaluating Systems of Different Types. SIGIR 2020
Evaluating search relevance part 2 — Phi-3 as relevance judge. Elasticsearch Labs
On Information Retrieval Metrics Designed for Evaluation with Incomplete Relevance Assessments. Information Retrieval
The Effect of Topic Set Size on Retrieval Experiment Error. SIGIR 2002
Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. IJCNLP-AACL 2025