# How do you evaluate citations in RAG answers?

> Evaluate RAG citations for claim support, missing evidence, and source quality without confusing working links with a trustworthy answer.

Source: https://www.nodus-compute.ai/blog/evaluate-rag-citations/
Build revision: ac3676a69c0cf64772501b25f4970adaf26dbd26

**Beta:** this feature may change.

**Evaluate RAG citations by checking whether the cited passages support the attached factual claims, whether material claims have supporting citations, and whether the answer covers the user’s question.** Keep these judgments separate. A working link proves access to a page, not support for a claim. A fully cited answer can still leave out the condition the user needed.

For engineers comparing open-weight models through [Nodus inference](https://www.nodus-compute.ai/docs/guides/inference/), citation quality belongs in the acceptance decision alongside answer quality. Inference is **Beta**. Serving a model does not establish that its citations are valid or that its sources are authoritative; those decisions require evaluation against your own questions and evidence.

## Judge the claim, not the appearance of a reference

Suppose an answer says a product is available in a particular region and includes a link to its documentation. The page might describe the product without saying anything about regional availability. The reference is relevant, but it does not establish the answer’s claim.

Review the smallest meaningful factual assertion, including its qualifiers. Numbers, dates, exceptions, comparisons, and words such as “all” or “only” can change what evidence is required. A source saying a feature supports one account type does not support an answer saying every account can use it.

Define citation scope before scoring. Does a marker apply to the preceding clause, the entire sentence, or several sentences? When that is ambiguous, record the ambiguity rather than assuming the most generous interpretation. Split compound sentences into separately reviewable claims when needed, and document that choice so the denominator stays consistent.

Research distinguishes these dimensions too. [ALCE, a benchmark for answers with citations](https://arxiv.org/abs/2305.14627), evaluates answer correctness and citation quality separately. Its statement-level support checks are not interchangeable with a finer-grained claim review. Choose a review unit deliberately rather than borrowing a metric name for a different procedure.

## Separate support coverage from citation usefulness

Two failure patterns require different fixes:

* An answer can attach many irrelevant references while leaving its factual assertions unsupported.
* An answer can cite one assertion correctly while making several other assertions without evidence.

For a practical claim-level audit, report **supported-claim coverage**: the number of material factual claims fully supported by their attached evidence, divided by all material factual claims in the answer. Count an uncited claim as unsupported for this measure, even if a reviewer can find a supporting passage elsewhere. That missing attachment is part of the user experience being tested.

Separately inspect each citation’s contribution. Does it support the attached claim, contribute necessary evidence when sources are combined, or add nothing relevant? Several passages may jointly support a statement. Do not require each passage to prove every clause on its own, and do not penalize a second independently supporting source merely because the first was sufficient.

Use full, partial, and no-support labels during review. The [human citation audit in *Evaluating Verifiability in Generative Search Engines*](https://arxiv.org/abs/2304.09848) uses these distinctions and discusses ambiguities caused by sentences containing multiple claims. Partial support is valuable diagnostic information, but it should not silently become full credit.

## Worked example: correct citations, incomplete support

The following policy, answer, and counts are **illustrative**, not customer records, measured model results, or Nodus product terms.

Imagine an internal travel guide with two passages. Passage A says a lodging allowance applies to domestic trips. Passage B says international trips need prior approval and a separately agreed allowance.

An assistant answers: “The lodging allowance applies to domestic trips. It also applies automatically to international trips. International travel needs prior approval.” It cites A after the first sentence and B after the third.

|Material claim|Attached evidence|Review|
|-|-|-|
|The allowance applies to domestic trips|Passage A|Fully supported|
|The allowance automatically applies internationally|None|Unsupported and contradicted by the guide|
|International travel needs prior approval|Passage B|Fully supported|

Both attached citations support their respective claims. Yet only two of the three material claims have full attached support: supported-claim coverage is **2 / 3**, about **67%**. Reporting only the quality of the two citations would hide the unsupported international allowance claim.

Now remove the second sentence. The remaining answer has full support for its two claims, but it still omits the need to agree on a separate allowance. If the user asked how to arrange an international trip, that omission matters. Citation coverage improved without the answer becoming complete enough to act on.

## Keep source suitability and answer completeness visible

A passage can support an assertion faithfully and still be outdated or inappropriate for the decision. A retired policy might explicitly state the old rule. A model repeating it with an accurate citation has passed an attribution check, not a current-policy check.

The [Attributable to Identified Sources research](https://arxiv.org/abs/2112.12870) treats source quality as separate from attribution. Apply that boundary in your evaluation. Record whether the cited material is current, authorized for the task, and applicable to the user’s circumstances. Do not let a support score certify those properties by implication.

Define expected answer elements before comparing candidates. For the illustrative travel question, those might include approval, the applicable allowance, and any unresolved condition requiring a reviewer. Score whether the answer covers those elements separately from whether its generated claims have support. Otherwise a terse answer can improve its citation score by avoiding difficult but necessary facts.

Include questions the available evidence cannot answer. The acceptable behavior may be to state the gap or request clarification. A confident answer with a related link is not an acceptable substitute for missing evidence.

## Compare models without changing the evidence underneath them

For a controlled citation comparison, give candidates the same questions and evidence passages, with the same instructions and response constraints. This tests how they use available evidence. It does not test whether the complete retrieval process finds the right material.

Then evaluate the complete application on representative questions. Keep failures to retrieve useful evidence separate from failures to use or cite evidence that was available. Preserve the evaluated source versions so later page changes do not make an old answer impossible to audit. A broken link is an access failure; an accessible passage that does not support the claim is a support failure.

Automatic support judgments can help prioritize review, but verify them against human-reviewed examples. Include partial support, multiple-source claims, changed dates, and exceptions. ALCE notes limitations in automated support evaluation; a judge’s confident label is not independent proof that a claim follows from a passage.

Set acceptance conditions by consequence. An unsupported eligibility condition may block release even when most descriptive claims are well cited. Compare recurring failure categories and whole-answer acceptance, not just a single average.

Nodus offers Beta inference for open-weight model evaluation; it does not replace your citation rubric or source review. For a model evaluation project you want to run on Nodus, [book a call](https://nodus-compute.ai/book/).

## FAQ

### Is a valid URL enough to count a citation as correct?

No. Check the cited passage against the attached claim, including its scope and qualifiers. Accessibility and support are separate checks.

### Is citation coverage the same as retrieval recall?

No. Citation coverage concerns evidence attached to generated claims. Retrieval recall concerns finding relevant evidence from the available collection. Neither establishes that the answer covers the whole question.

### Should every sentence have a citation?

Not necessarily. Define which factual assertions need verification. A sentence may contain several such claims, while a transition or clearly labeled hypothetical may require different treatment. Apply the rule consistently.

### Can several citations support one claim together?

Yes. Assess their combined support and each citation’s contribution. Flag irrelevant extras without requiring every useful source to establish the whole statement independently.
