# How do you evaluate LLM extraction when documents contain conflicting values? > Test whether an LLM extracts the authoritative value from conflicting documents, preserves uncertainty, and supports its answer with the right evidence. Source: https://www.nodus-compute.ai/blog/evaluate-llm-extraction-conflicting-documents/ Build revision: 06296236aea5db1dcc6c4b7d9b4aba763435f9e7 **Beta:** this feature may change. Evaluate extraction from conflicting documents by scoring the **value, its authority, and its supporting evidence together**. Define which source controls the answer before testing, include cases where no source resolves the conflict, and use matched document pairs to check that answers change only when the governing evidence changes. An amount that matches by accident is not a dependable extraction result. Nodus Inference (Beta) offers hosted-model inference for testing document-based tasks. The acceptance method below is a customer-side evaluation workflow, not a claim that Nodus supplies document adjudication, automatic citations, or a built-in conflict-resolution score. ## The target is an authoritative answer, not a plausible number Consider a purchasing workflow that asks for the currently approved order total. A packet might contain an original order, a proposed amendment, an approved amendment, and correspondence quoting an older amount. All can contain genuine numbers. Only some answer the question. The extraction target needs a rule for authority. For example: an approved amendment replaces the original order for the fields it explicitly changes; an unsigned proposal does not. A message mentioning a total does not necessarily amend the order. That rule must come from the business process and authorized reviewers, not from the model’s preference for a recent date or a prominent number. Keep the requested field precise. “Original order total,” “latest proposed total,” and “currently approved total” can legitimately have different answers in the same packet. Scoring them against one reference value would measure an unclear task rather than extraction quality. ## Make the reference decision inspectable For each packet, have a qualified reviewer record the requested field, authoritative value if determinable, controlling document and passage, and the reason competing values do not control. Preserve currency, units, and relevant date or version qualifiers. A value without those qualifiers can still be unusable. Use three reference outcomes: * **Determinate:** the documents and authority rule support one answer. * **Conflicting:** the available sources disagree and the rule does not resolve them. * **Insufficient:** required evidence is missing, even if a plausible candidate appears. These are evaluation labels, not claims about Nodus response features. Your application must decide how to represent and handle each outcome. Reviewers should mark genuine ambiguity rather than manufacture certainty to complete the answer key. Adjudicate consequential disagreements before comparing models. Give the model only information available to the production task. A reference based on a private approval record is unfair if the application receives only the document packet. Either make that evidence available through an authorized workflow or expect the model to report that the answer cannot be determined. ## Worked example: the same two amounts, different correct decisions The following packet and amounts are **illustrative**, not customer records, measured results, or Nodus prices. An original order is approved for $12,000. A later document proposes $14,000. The question is: “What is the currently approved total?” |Packet variation|Reference decision|Why| |-|-|-| |Later document is an unsigned proposal|$12,000|A proposal does not replace the approved order| |Later document is an approved amendment|$14,000|The amendment explicitly replaces the total| |Later document has unclear approval status|Unresolved|The packet cannot establish whether the amendment controls| |Approved amendment changes only delivery date|$12,000|Approval of another field does not change the total| These variations test a useful distinction: sensitivity to material evidence without sensitivity to irrelevant changes. A model that always chooses the later number fails the proposal and delivery-date cases. One that always copies the original order fails the approved amendment case. Also inspect the evidence. Returning $12,000 while citing an unrelated historical invoice should not earn full acceptance, even if the amount matches. Require the selected value and evidence to support the same reference decision. ## Build matched pairs that expose the shortcut Start with authorized or deliberately constructed packets that represent the intended workflow. Create a development set and a separate untouched audit set. Keep related variants together when assigning those sets so familiarity with one packet does not masquerade as generalization. For each development packet, vary one meaningful condition while holding the rest stable. Change proposal status to approved status. Change which field an amendment replaces. Remove the evidence establishing approval. Include a conflicting amount in a quotation that has no governing authority. Then add changes that should not affect the decision: reorder the packet, move the governing passage, or insert an unrelated document. Judge each variation under the actual business rule. Reordering documents should not change authority unless order itself has a defined meaning in that workflow. Record pair outcomes as well as individual accuracy. Does the system change its answer when the governing evidence changes? Does it preserve the answer when only presentation changes? These are different properties. A model can score well on ordinary packets while failing the exact transition that matters in production. ## Accept the whole extraction, not isolated fields Evaluate the output the receiving reviewer will actually use. For a consequential field, acceptance can require the correct normalized value, correct unit or currency, applicable qualifier, and supporting passage from the controlling source. If the packet is unresolved, a confident amount is a failure even when it happens to equal a later reviewer decision made with additional information. Report separate error categories rather than one blended score: |Observed outcome|What to investigate| |-|-| |Wrong amount, correct source|Reading, calculation, or normalization error| |Plausible amount, wrong source|Authority selection or evidence error| |Definite answer on unresolved packet|Unsupported certainty| |Unresolved answer on determinate packet|Unnecessary escalation| |Correct value and governing evidence|Candidate for end-to-end acceptance| Do not let an average conceal a field whose error would trigger a payment or contractual action. Set acceptance requirements by consequence. A workflow may tolerate manual review for an uncertain description while requiring verified authority before accepting a changed total. Include missing outputs and processing failures in the original packet count. Keep document preparation constant when comparing models, then test the complete workflow with the document formats users actually submit. Good extraction from clean text does not establish reliable results on scans or damaged files. ## Compare models on the same decision contract Freeze the authority rule, document preparation, requested fields, and acceptance rubric before comparing candidates. Use the same audit packets and inspect disagreements, especially changes in governing evidence and unresolved cases. Record review effort alongside accepted extractions: fewer escalations are useful only if unsupported answers do not rise. The [Nodus inference guide](https://www.nodus-compute.ai/docs/guides/inference/) documents Beta inference, current model availability, and request receipts. Verify the selected model’s capabilities before testing. Receipts can support usage accounting; they do not establish that an extracted value is authoritative. Keep task labels, evidence review, and acceptance decisions in your own evaluation record. For a document-extraction project you want to evaluate on Nodus, [book a call](https://nodus-compute.ai/book/). ## FAQ ### Is exact-match accuracy enough? No. The right amount from the wrong source can match accidentally. Evaluate authority and evidence as well as the normalized value, especially where the same number appears in several documents. ### Should the latest document always win? Only if the governing business rule says so. A later proposal, quotation, or unrelated amendment may not supersede an earlier approved value. Dates are evidence, not a universal authority rule. ### Should unresolved packets be excluded from the test? No. They test whether the workflow avoids unsupported decisions. Report unresolved-case behavior separately from determinate extraction accuracy, and define the appropriate review path before deployment. ### Does model confidence prove the selected source is authoritative? No. Confidence is not approval evidence. Test the authority decision against reviewed packets, including matched variations that change approval status without changing the candidate amounts.