How do you evaluate an embedding model for search?

Nov 10, 2026 - 7 min read

Evaluate an embedding model on whether search returns the evidence your user needs within a realistic reading budget. Keep the searchable documents, queries, relevance judgments and result limit fixed across candidates. Separate finding one useful passage from finding everything needed to answer the question. Do not let a strong average hide failures on exact identifiers, exceptions or queries with no answer in the collection.

Nodus offers embedding models through its inference service, which is Beta. Availability depends on the current model list. That gives you candidates to compare, not proof that any candidate is the right choice for your search task.

What should count as a successful search?

Start with the decision the search supports. Finding a product manual, resolving an eligibility question and collecting evidence for a multi-part answer have different success conditions.

For a navigational query, the correct document near the top may be enough. For a policy question, success may require a rule and its exception. For research, several independently useful sources may matter. Write the acceptance rule before looking at candidate results.

Label evidence as useful for this exact question, not merely about the same subject. A passage about changing delivery addresses does not answer whether an order that has already shipped can be redirected. A current policy and an expired policy can share almost all their wording while supporting different decisions.

Keep two judgments separate:

Relevant result: this item contributes correct, applicable evidence.
Sufficient result set: the returned items collectively support the intended answer.

An embedding comparison becomes misleading when the first is used as a substitute for the second.

Build queries from information needs, not document titles

Use authorized, representative search requests where possible. Remove personal or confidential material you do not have permission to use. Record the collection version so every candidate faces the same available evidence.

Include natural paraphrases, exact product identifiers, abbreviated queries and questions that require exceptions. Keep realistic ambiguous requests instead of silently rewriting them into perfect queries. Identify questions whose required evidence is absent from the collection.

If you generate queries by reading a source, avoid copying distinctive phrases into every query. That creates a test of recognizing the author’s wording rather than meeting the user’s information need. Keep a held-out set that was not used to choose models or revise the comparison.

Public benchmarks help shortlist candidates. BEIR deliberately includes diverse retrieval tasks and supports several ranking measures. A benchmark result is useful context, not a guarantee for your documents, language mix or acceptance rule.

Freeze the comparison before changing the search experience

Choose how many results a person or downstream model can realistically inspect. Compare candidates at that same limit. Also check the amount of text: five short passages and five long documents are not the same reading burden.

Keep the document collection and the text presented as each search result unchanged. If one candidate gets rewritten documents, extra query context or a separate ranking stage, label it a comparison of complete search configurations rather than of embedding models alone.

Record each candidate’s documented input requirements and limits. If a candidate cannot handle a required query or document, report that limitation; do not quietly omit those cases. Keep errors and unavailable responses visible alongside completed searches.

This is a controlled buying decision, not a contest to make each candidate look good under a different set of conditions.

Worked example: one relevant hit is not a complete answer

Consider this fictional, illustrative search task. It is not a customer result or a measured model comparison.

The query is: “Can we redirect an order after dispatch?”

The collection contains these items:

Item	Content	Judgment
General policy	Address changes are allowed before dispatch.	Useful background, insufficient alone.
Current exception	After dispatch, redirection requires carrier approval.	Required evidence.
Approval conditions	Approval depends on destination and shipment status.	Required evidence.
Expired exception	A previous policy allowed a different process.	Not applicable to the current decision.

Suppose the fixed reading budget is three results. Candidate A returns the general policy, the current exception and a duplicate of that exception. Candidate B returns the general policy, the current exception and the approval conditions.

Both candidates found a useful result. Both found the current exception. Only B returned the complete required evidence within the budget. A polished answer written after A’s search cannot repair the missing conditions without obtaining more evidence.

Count duplicate passages as consuming reading space, not as independent coverage of a new requirement. Conversely, do not penalize two different passages merely because they support the same fact when either is a valid answer source. Define interchangeable evidence and distinct requirements before scoring.

Which retrieval metrics should you report?

Choose measures that match the search task, and state their denominators.

Hit rate at k asks what share of answerable queries have at least one judged-relevant result in the first k positions. It is useful when one suitable result is enough. It does not establish completeness.

Recall at k measures how much of the judged-relevant evidence is returned within those positions. State whether your judgments refer to documents, passages or distinct evidence requirements. If your reference set is incomplete, call it recall against that judged set rather than claiming exhaustive recall.

Complete-evidence rate at k can be an application-specific measure: the share of answerable queries for which every required evidence item, or an accepted equivalent, appears within the first k results. Define it explicitly; it is not a substitute name for standard recall.

Report ranking quality as well when early placement matters. BEIR supports measures including normalized discounted cumulative gain and reciprocal rank. A ranking measure still depends on the quality and meaning of the relevance labels.

Review a shared pool of results from all candidates, with candidate identity hidden where practical. Otherwise, judgments made only from one model’s results can unfairly favor it. Preserve a category for disputed or unjudged results instead of automatically treating them as irrelevant.

Diagnose missing answers before replacing the model

Keep these cases separate in the report:

The required source does not exist in the collection.
The source exists, but the candidate does not return it within the reading budget.
The returned evidence is sufficient, but the answer written afterward is wrong.
The result appears relevant but applies to a different product, date or eligibility condition.

Only some of these are failures of the embedding candidate. Increasing the result limit can improve evidence coverage while increasing reading time and downstream input. Compare that tradeoff explicitly rather than claiming an improvement at an unchanged cost.

For queries with no answer in the collection, do not compute ordinary recall against an empty relevant set. Evaluate the application’s behavior separately, including whether it admits insufficient evidence. An embedding model alone does not guarantee safe abstention.

Choose the candidate that meets your critical search requirements within the tested reading budget, response time and total operating cost. Include document-processing and query costs where applicable; do not select on a token price or public leaderboard alone.

For an embedding-model comparison on Nodus, check current availability and supported operations first. If you want to discuss fit for your document-search requirements, book a conversation with Nodus. No retrieval accuracy result is implied by model availability.

FAQ

Is a higher similarity score evidence of better search?

Not across different models by itself. Scores are not necessarily calibrated on a common scale. Compare judged results and task outcomes under the same conditions.

Should I score the final generated answer instead?

Score it separately. A correct answer can conceal missing retrieved evidence, and an incorrect answer can follow a successful search. Keep evidence retrieval and supported answer quality distinguishable.

Can I choose a model using only a public benchmark?

Use benchmarks to reduce the candidate list. Make the final choice on held-out queries and authorized documents that reflect your actual search task, including the failure categories you cannot accept.