How do you evaluate LLM summaries when omissions matter?

Nov 7, 2026 - 7 min read

Evaluate an LLM summary in two directions: check that every claim it makes is supported, and check that every fact required for the reader’s next decision survives compression. Define those requirements before viewing model outputs. Score critical omissions separately from invented claims, and compare summaries under the same reader purpose and length constraint.

A summary can contain only true statements and still be unusable. Leaving out an exception, an unresolved dependency, or who must act can change what the reader does next. Readability alone cannot establish that the handoff is safe.

Nodus inference (Beta) gives you access to available open-weight models and request usage and billing information. When comparing models for summarization, use that information alongside a task-specific omission review, not as a substitute for it.

Start with the reader’s next decision

“Summarize this document” leaves the acceptance criteria ambiguous. A leadership briefing and an operational handoff may legitimately preserve different facts from the same source.

State who reads the summary, what they must decide, and whether they will also have the source. Then define which information must survive. For a handoff, that might include the outstanding action, its owner, its deadline, and the condition that permits work to proceed.

Use only information present in the source. If an owner is unspecified, the required takeaway may be that ownership remains unresolved. Do not reward a model for filling the gap with a plausible name.

This purpose statement also limits the rubric. A summary is not defective simply because it omits detail irrelevant to the agreed decision. Requiring everything would test transcription rather than summarization.

Build a source-backed fact inventory before scoring

For each evaluation document, have a qualified reviewer identify the facts the intended reader needs. Record the supporting passage and distinguish three categories:

Decision-critical facts: omitting or changing them could alter the next action.
Useful context: helpful to interpretation but not independently necessary for that action.
Dispensable detail: safe to remove for this reader and purpose.

Keep related facts together. “Approved” and “approved only after inspection” are not interchangeable. The qualification belongs to the approval, not in an optional context category.

Write acceptable paraphrases rather than one mandatory reference summary. The model should preserve meaning, not mimic a reviewer’s wording. Resolve disagreements about criticality against the reader purpose before comparing models. Otherwise, a reviewer may unknowingly favor the facts highlighted by a preferred output.

Worked example: an accurate summary that fails the handoff

The following is a fictional, illustrative example, not a customer incident or a measured model result.

A source packet says:

The replacement pump passed its bench test. Installation remains blocked until the site supervisor confirms the isolation permit. Mina owns that confirmation and must report its status by Tuesday afternoon. The old pump remains in service under the existing operating restriction.

The reader is the incoming shift lead, deciding whether to schedule installation.

Candidate summary	What the review should notice
“The replacement pump passed testing. Mina will provide an update Tuesday.”	Both statements are supported, but the installation gate and the nature of Mina’s action are missing.
“The replacement pump is ready for installation Tuesday.”	This invents readiness and turns an update deadline into an installation date.
“Bench testing passed, but installation is blocked pending the site supervisor’s isolation-permit confirmation. Mina owns the confirmation and must report by Tuesday afternoon. The old pump remains under its operating restriction.”	The required gate, ownership, deadline, and current operating state survive.

The last candidate is acceptable for this defined handoff, not automatically for every audience. A maintenance archive might need additional test detail. A shorter display might need to link to the full handoff rather than pretend every necessary fact fits.

This distinction matters when comparing models. The first candidate is not merely less detailed; it withholds information the reader needs to avoid scheduling blocked work.

Score omissions and unsupported claims separately

Review each summary against the inventory and the source. A practical scorecard keeps these outcomes distinct:

Critical coverage: Did each required fact survive with its meaning intact? Track the missed fact and its category, not just an average across all facts.

Claim support: Is each factual assertion supported? Check names, dates, quantities, status, ownership, and whether uncertainty has become certainty.

Relationship preservation: Does the summary retain which condition governs which action? Listing the right nouns does not preserve an approval dependency.

Reader usefulness: Can the intended reader identify the next action without guessing? Note whether returning to the source is expected, an acceptable fallback, or a failure for this product.

Brevity and readability: Assess these only alongside the factual checks. A longer summary may cover more facts while failing the actual space constraint.

For a decision-critical handoff, make required-fact preservation and absence of action-changing distortions acceptance gates. Keep less important quality measures separate so strong prose cannot compensate for a missing gate. Report document-level acceptance as well as fact-level coverage; many correct minor facts should not hide one consequential omission.

Test compression without changing the task

Use matched summaries of the same documents at the output lengths your product actually permits. Hold the reader purpose and required information constant. Check whether shortening removes background first or drops conditions and unresolved actions.

Distinguish an impossible brief from a model failure. Before evaluating, ask a reviewer to produce an acceptable summary within the constraint. If the essential information cannot fit, change the product requirement: allow more space, narrow the reader’s decision, or make the summary an explicitly incomplete preview.

Include documents with different omission risks: one dominant finding, several independent actions, a negative result, a conditional approval, and unresolved responsibility. Evaluate long source packets separately from short ones. Do not let success on simple updates obscure failures on dependency-heavy handoffs.

Use realistic source material you are authorized to process. Remove unnecessary personal and confidential details before moving material into an evaluation service.

Compare models on acceptable handoffs, not attractive prose

Run each candidate on the same held-out documents, instructions, and output constraints. Blind the model identity during review where practical. For repeated model runs, retain all responses rather than selecting the most complete answer afterward.

Report critical omissions, unsupported claims, and acceptable-summary outcomes by document type. Also measure review effort: a model whose summaries routinely require checking the original document may not reduce the intended handoff effort, even when its prose is excellent.

On Nodus inference (Beta), check each model’s current capabilities and availability before testing. The public guide documents model access plus request token usage and charges. Compare those observations with your reviewed acceptable outputs. They do not establish summary quality, and Nodus does not supply the domain judgment in this evaluation framework.

Choose the model and summary format that satisfy the decision requirements at an acceptable cost and review burden. Keep human review where the consequences of a missed condition demand it.

FAQ

Is comparison with a reference summary enough?

No. A reference can help reviewers, but matching its wording does not establish critical coverage or factual support. Use source-backed requirements and allow meaning-preserving paraphrases.

Should every omission count as a failure?

No. Omissions are the purpose of summarization. Treat missing required information as a failure for the defined task; assess optional context separately. Decide what is required before seeing outputs.

Can an LLM judge review summaries automatically?

It can assist with flagging potential omissions and unsupported claims, but validate it against qualified human review on the error types you care about. A judge that also overlooks conditional language cannot reliably clear that risk. See calibrating an LLM judge.

What if the source itself is ambiguous?

Require the summary to preserve the ambiguity that matters to the reader. Do not give credit for guessing the missing owner, approval, or date. Clarifying that a dependency remains unresolved may be the most important part of the summary.

If you are comparing summarization models and need to discuss Nodus inference fit, book a conversation. Bring the intended reader, a representative source packet you can share, and the information your summary must preserve.