How do you compare two LLMs with the same overall accuracy?

Nov 14, 2026 - 6 min read

Compare two LLMs with the same overall accuracy by inspecting their answers to the same requests. Count improvements and regressions separately, identify which users or decisions each change affects, and apply critical-failure rules before choosing on cost or speed. Equal accuracy means the net change is zero on that test set. It does not mean the models behave interchangeably.

Nodus Inference is Beta and lets you try available open-weight models in the Playground. That can support a candidate comparison, but availability is not evidence that one model can replace another. The decision belongs to your evaluation, including the errors hidden behind its average.

Start with matched requests and a fixed decision

Define the replacement decision first. Perhaps you want lower serving cost without losing an important capability. Perhaps you want better answers to difficult requests without slowing routine ones. Write down which tradeoffs are acceptable before seeing the candidate’s results.

Give both models the same authorized inputs, relevant conversation history and task instructions. Record model identities and the response settings that affect the comparison. If your actual proposal changes the prompt too, evaluate that complete candidate setup and describe the result as a setup comparison, not a pure model comparison.

Use a fixed rubric. For a binary score, state what makes an answer pass: factual correctness, required evidence, permitted actions or another observable outcome. A fluent answer is not automatically a correct one. Resolve uncertain labels rather than quietly treating them as wins.

Keep the final decision set separate from examples used to choose prompts or candidates. Otherwise, the comparison measures adaptation to a familiar test as well as behavior on unseen requests.

Build the disagreement table

For each request, record whether the current model passed and whether the candidate passed. This produces four groups:

Current model	Candidate	Meaning
Pass	Pass	Retained success
Fail	Pass	Improvement
Pass	Fail	Regression
Fail	Fail	Unresolved failure

The two disagreement groups explain the accuracy change:

Net accuracy change = (improvements - regressions) / total requests.

But keep both counts visible. A net gain can conceal important losses, and a net tie can conceal a substantial change in who receives a correct answer. The shared-failure group also matters: replacing the model may leave your most important problem untouched.

Worked example: a tie hides sixteen changed outcomes

All numbers and scenarios in this example are illustrative, not measured model performance or Nodus results.

Suppose you compare a current model and a candidate on the same 100 support requests. Both score 90% under your binary rubric.

Paired outcome	Requests
Both pass	82
Current fails, candidate passes	8
Current passes, candidate fails	8
Both fail	2
Total	100

Each model passes 82 shared successes plus eight unique successes. The net accuracy change is (8 - 8) / 100 = zero percentage points. Yet sixteen requests change outcome.

Now inspect those changes. Suppose the candidate’s eight improvements correct routine setup guidance. Six regressions provide incorrect setup guidance, while two would disclose account information to an unauthorized requester. These are invented case labels for the example, not reports of an actual incident.

If the predeclared rule prohibits that disclosure, the candidate fails the release gate even if it is faster or less expensive. Routine improvements do not cancel a prohibited action. Conversely, if the regressions are acceptable under your actual rules and the efficiency benefit is verified, the candidate may merit further testing. The table identifies the tradeoff; it does not decide your tolerance for you.

Separate business weights from hard constraints

For errors that you can reasonably trade, estimate their importance using the decision they affect. Extra review time, an abandoned task and an incorrect recommendation need not have equal costs. Use documented assumptions and show how the conclusion changes when those assumptions change.

Do not assign a convenient finite weight to an outcome your policy forbids. Keep that as a separate constraint. Nor should a single observed violation be dismissed because its contribution to overall accuracy is small.

A challenge set can deliberately contain many difficult cases. Its unweighted average does not estimate incoming traffic performance unless that mix represents incoming traffic. Report challenge-set results separately from results on a representative sample. Apply traffic weights only when you have a defensible basis for them.

Audit disagreements before choosing a winner

Review both improvements and regressions without showing reviewers which model produced each answer when practical. Use the same rubric, adjudicate ambiguous cases and retain enough context to explain the final label. Do not let an automatic judge’s preference for one writing style become the release decision.

Group disagreements by meaningful use case, language, input source or decision severity. Report each group’s denominator. A small group with no observed regression is limited evidence, not proof that it is protected. Inspect shared failures as well, especially when the purchase decision is supposed to fix one of them.

For a formal binary paired comparison, McNemar’s test tests equal marginal outcomes using the disagreement counts. It can address evidence of an overall difference under its assumptions. It does not establish equivalence, assign business importance or certify a critical-error rate. An inconclusive test is not proof that the models are the same.

Check whether the result repeats

A response can change across runs. Recheck important disagreements under the response settings you intend to use, and report whether the same cases keep failing. Repeated answers to one request are not new independent customer requests. Keep those two sources of variation separate.

After changing a prompt in response to a regression, treat it as a new candidate setup. Reuse known failures for development, then make the final decision on held-out cases. Preserve the original comparison so a repaired example does not erase the history of what changed.

Compare serving cost and latency only after quality gates are met. Use equivalent task inputs and completion requirements; shorter but incomplete answers are not a valid efficiency win. Nodus model availability and capabilities can change, so check the current catalog before planning a candidate trial. For help finding a fit for a defined comparison, talk to Nodus.

FAQ

Is the cheaper model the right choice when accuracy ties?

Only if the paired regressions are acceptable, critical constraints pass and the cost advantage holds for comparable completed tasks. The tie alone supplies none of those conclusions.

Do I need to inspect every shared success?

Audit a sample to check that the rubric and labels are reliable. Prioritize disagreements for understanding the replacement tradeoff, but do not assume matching scores prove matching reasoning or correctness.

Can I compare scores from two different test sets?

You can describe each result with its population and limits. You cannot build a paired disagreement table from unmatched requests or attribute the score difference solely to the model. Run a matched comparison for the replacement decision.

Does no significant difference mean the models are equivalent?

No. Equivalence requires a separately designed analysis with a justified tolerance and enough evidence. A business decision also needs explicit treatment of critical failures and the users represented by the test.