The Search Evaluation Ladder

It is tempting to use an LLM to evaluate how your search system is performing.

Give it a query, show it the results, ask it to act as a judge, and have it rate their relevance. The labels look clean. The explanations sound reasonable. You can aggregate the scores and compare one version of search with another.

But if you are diagnosing a search problem or testing a change, this is often the wrong place to start.

The LLM can judge only the results it receives. It can tell you whether those results look relevant. It cannot establish that the search system worked correctly before producing them.

Relevant documents may have been indexed incorrectly, filtered out, or missed during retrieval. Those failures require different tests and different engineering fixes.

In production search, several failures can produce the same visible symptom: a result looks wrong. A filter may have removed the right candidate, the candidate window may be too shallow, or the right documents may have reached ranking and been ordered poorly. The result page does not identify which boundary failed; diagnosis requires separate checks.

This article organizes the checks into five diagnostic layers: Deterministic Correctness, Candidate Coverage, Relevance Judgments, Retrieval & Ranking Evaluation, and Evaluation Audit. Together, they form the Search Evaluation Ladder.

This ladder focuses on offline diagnosis and evaluation. Production changes may also require online validation, such as an A/B test.

Each higher layer assumes that enough of the lower layers are working. Starting at the judge can produce a precise answer to the wrong question.

Search Must Behave as Configured

Deterministic Correctness establishes whether the search system processes the corpus and queries as intended.

This covers field types, tokenization, case and Unicode normalization, language analysis, stemming, synonyms, numeric and date handling, missing fields, filters, and permissions. A defect at this layer changes what the system can retrieve before ranking quality is measured.

Consider a product identifier such as AB-1234. If the configured analyzer splits it into ab and 1234, a search for the exact identifier may behave like a broad two-term query. An LLM can judge the returned products accurately while saying nothing about the configuration error that produced them.

Linguistic processing can create the same problem even when the implementation is behaving as designed. The original Porter stemmer, for example, can collapse distinctions between terms such as universe and university; later versions introduced handling to avoid such conflations. The algorithm may be operating correctly while collapsing a distinction the product needs to preserve.

Deterministic fixtures are the right instrument here.

A deterministic fixture checks both sides of the expected search behavior. The intended document must match; documents with the wrong identifier, tenant, language, or permissions must not. Analyzer output, indexed fields, query parsing, filters, and access-control predicates should be inspectable without a relevance judge.

A common diagnostic sequence is to inspect the indexed representation first, verify that the expected field is searchable, and only then test whether the query admits the document into the candidate set. This keeps a data or configuration failure from being mistaken for a candidate-retrieval problem.

An LLM judgment is an expensive and indirect way to discover a deterministic configuration error.

Can the Right Document Reach Ranking?

Once the data path is correct, the next question is candidate coverage: can a known relevant document reach the stage where ranking can act on it?

Candidate generation may involve lexical retrieval, dense nearest-neighbor search, hybrid branches, query expansion, filters, permissions, routing, and a fixed candidate depth. The previous layer checks whether filters and permissions behave as configured; this layer asks whether eligible relevant documents still receive adequate coverage. Approximate retrieval adds another variable: the engine may trade recall for latency by examining only part of the vector index.

An exact-versus-approximate comparison measures ANN recall: whether approximate search recovers the neighbors that exact vector search would return. Even 100% ANN recall does not show that the embedding places relevant documents near the query. Relevance Recall@K separately measures whether known relevant documents reach the candidate stage.

A candidate-coverage test does not ask whether the document finished first. It asks whether the document was available at the candidate depth consumed by the next ranking stage. For each known query-document pair, this is a binary inclusion check. Measuring relevance Recall@K across a query workload requires judgments identifying the relevant documents for each query.

This exposes failures caused by shallow candidate windows, low ANN recall, restrictive filters or permissions, routing errors, or a retrieval branch that never ran.

Ranking cannot recover a document that candidate generation never admitted.

Separate Recall Failures from Ranking Failures

A poor top-ten result set can reflect failures at two different boundaries, sometimes at both:

The visible symptom may be the same: relevant results are absent or too low. The corrective action differs.

Inadequate candidate coverage may require changes to query expansion, filters, ANN settings, retrieval routing, or candidate depth. Poor ranking means useful candidates were available but ordered poorly, and may require different features, weights, training data, or a stronger reranker. Both can occur in the same result set.

The candidate-coverage check therefore does more than measure retrieval coverage. It identifies which side of the retrieval–ranking boundary needs investigation. Tuning ranking when the necessary candidates were never available spends effort at the wrong boundary.

Where LLM Judgments Help

LLM judging becomes useful when the bottleneck is relevance labeling. A new corpus may have few judgments. A semantic retriever may surface candidates absent from an older judgment set. Human review may be too expensive to cover enough queries and documents for useful iteration.

An LLM can generate labels for those query-document pairs and make repeated offline comparison practical. OpenSearch Search Relevance Workbench, for example, retrieves the top results from a configured search, sends selected document fields and a prompt to an LLM, and stores the resulting judgments for later experiments.

That workflow solves a real problem by scaling judgment generation. It does not establish that the index is correct, that candidate recall is sufficient, or that the evaluation set represents the production workload.

The resulting judgments are also limited by the candidate set used to generate them. If candidates came from one retrieval method, the LLM can label only the evidence that method exposed. A different retriever may surface relevant query-document pairs that were never available for judgment.

The API call made judgment generation easier. The surrounding search and evaluation problems remain.

Evaluate Retrieval and Ranking Against Relevance Judgments

Deterministic tests establish whether expected search behavior holds. They cannot show whether one search configuration is better across a representative query set.

Relevance judgments support two complementary measurements: whether candidate retrieval finds relevant documents, and how well the final results satisfy the information need. Recall@K measures coverage of judged relevant documents at a specified depth. Metrics such as nDCG, MRR, and MAP measure the quality of ranked results and can reflect both missing relevant documents and their ordering.

To isolate a reranker's effect on ordering, compare ranking configurations over the same candidate set. When candidate sets differ between configurations, the comparison measures end-to-end search quality.

That requires offline relevance evaluation:

A candidate-coverage fixture can prove that document D was retrievable for query Q. It cannot prove that a new ranking function improved ordering across navigational, exploratory, exact-match, multilingual, and long-tail queries. Evaluating retrieval and ranking across that workload reveals improvements and regressions by query or slice.

This is the layer served by tools such as Rated Ranking Evaluator, Quepid, pyvespa, OpenSearch Search Relevance Workbench, and custom evaluation harnesses. They differ in workflow and platform support, but all support the same basic pattern: run controlled search configurations against a query set and compare the returned rankings using a defined set of relevance judgments.

Verification tells you whether an invariant holds. Relevance evaluation tells you whether one search configuration is better against a chosen reference.

When Evaluation Becomes Theater

LLM evaluation becomes evaluation theater when sophisticated judgments and precise-looking metrics are used to support a claim the experiment was not designed to establish.

Consider this chain:

The score may be calculated correctly. It still does not diagnose the analyzer failure. Repeating the run with a stronger model, a more detailed rubric, or more judged examples improves the top layer while leaving the underlying defect untouched. Those repeated runs consume inference budget and engineering time without producing evidence about the boundary that failed.

The test does not support the claim being made. A final-ranking metric can show that relevant documents are missing or poorly placed. It cannot tell you whether a missing document was excluded before ranking or simply ranked below the evaluated cutoff. A judge can label what it sees. It cannot audit evidence that never reached it.

LLMs remain useful here. The failure begins when judgment generation is treated as a substitute for retrieval diagnosis and evaluation design.

Audit Results That Carry Consequences

Once the lower layers have been checked and judgments exist, test the evaluation setup itself. A result used to discard an obviously weak idea needs less evidence than a small offline gain used to justify a production migration.

For consequential decisions, test whether the winner survives defensible changes to the query sample, candidate pool, evidence presentation, judging prompt, or judge model. Version the corpus, queries, retrieval runs, pool construction, judging configuration, and metric so a later score change can be attributed to something specific.

Our companion article, What Is Your LLM Judge Actually Measuring?, examines this layer in detail: relevance-reference drift, unjudged documents, Judge–Pool Coupling, judgment stability, and practical robustness tests.

The question here comes one step earlier: should this problem have reached a relevance judge yet?

One Change Across All Five Layers

The five layers answer different questions about the same search system. Consider a team introducing the synonym nyc → new york city for searches over municipal guidance. The same change creates a different evaluation question at each layer:

These tests are complementary, not interchangeable. A passing synonym fixture does not establish ranking quality. A higher nDCG score does not prove the synonym was applied correctly. An LLM label does not show whether a better document was excluded before judgment.

Match the Tool to the Question

Each failure calls for a different instrument:

A mature tool may support several evaluation jobs. The layer determines what its output can prove.

LLMs make a costly part of relevance assessment easier to scale and repeat. That makes broader judgment coverage and faster offline iteration practical.

Search quality still starts below the judge: in how data is represented, how queries are interpreted, which candidates survive retrieval, and how those candidates are ranked. Strong evaluation programs use deterministic verification, candidate-coverage tests, relevance judgments, retrieval and ranking metrics, and evaluation audits at the appropriate layer.

At Searchplex, this is part of relevance engineering: start at the lowest layer that can explain the failure, then use the judge for the question it can actually answer.