Can an agent's pass rate rise while it completes fewer tasks?

Oct 15, 2026 - 6 min read

Yes. A pass rate calculated only from scored cases can rise while the number of confirmed successful tasks falls. Read it alongside scoring coverage and confirmed successes across the original batch, or a better percentage can conceal a worse operational result.

This distinction matters in Nodus, whose documented agent-evaluation workflow is in Beta. Its results view separates scored cases from execution and grading failures. Those categories answer different questions; combining them into one headline number loses information you need to decide what to fix next.

How can a 100% pass rate be less useful than 78%?

Consider two evaluations of the same fixed set of 100 tasks. These are illustrative numbers, not a benchmark or a Nodus customer result.

Outcome	Run A	Run B
Correct, scored answers	70	60
Incorrect or invalid, scored answers	20	0
Unscored cases	10	40
Pass rate among scored cases	70 / 90 = 77.8%	60 / 60 = 100%
Scoring coverage	90 / 100 = 90%	60 / 100 = 60%
Confirmed successes across the full batch	70 / 100 = 70%	60 / 100 = 60%

Run B has the better scored pass rate. It also has ten fewer confirmed successes and four times as many cases without a score.

That does not prove its model is worse. The missing cases might have passed, failed, or exposed a different problem. It proves that the headline pass rate cannot settle the comparison.

If every unresolved case in Run A eventually passed, it would finish with 80 successes. Run B could finish anywhere from 60 to 100. Those are accounting bounds on unresolved outcomes, not statistical confidence intervals. There is not enough evidence here to rank the models by full-batch correctness.

For someone waiting for completed work, the immediate fact is simpler: Run B has delivered fewer verified successes so far.

Which failures belong in the score?

A wrong answer and a failed evaluation are not the same observation.

A wrong answer is evidence about task performance when the evaluation ran correctly. An invalid response can also be a scored failure if the task requires a particular response format and the rubric specifies that rule.

A grader that could not run provides neither a correct nor an incorrect verdict. Turning that absence into a zero reward confuses measurement failure with model failure. Removing it without reporting the missing case creates the opposite problem: the score looks cleaner because part of the workload disappeared.

The label alone does not establish the cause. A timeout might reflect a broken dependency, an unrealistic task deadline, or an agent that used its entire allowance without finishing. Inspect the record before deciding whether the change belongs in the model, the environment, or the evaluation setup.

For operational reporting, keep every original case visible. For model reporting, state which cases received valid scores and why the rest did not. A case can count as an unsuccessful delivery without pretending that a missing correctness judgment is a measured wrong answer.

What should you compare before changing the model?

Compare like-for-like tasks, then inspect what became unscored.

If Run B lost the harder tasks, its 100% score describes an easier surviving subset. If it lost a random slice, the smaller scored set still contains less evidence. Either way, comparing 100% with 77.8% alone overstates what you learned.

Keep the task set, starting conditions, model configuration, scoring rules and time limits fixed for a controlled comparison. When one of those must change, label the new result as a different evaluation condition rather than silently placing it on the same trend line.

Repeated runs answer another question: whether success is consistent. The original τ-bench paper evaluates task outcomes against a target state and introduces a measure of reliability across repeated trials. WebArena likewise emphasizes the functional correctness of completed tasks, not merely a convincing final explanation.

Those benchmarks support an important distinction: completing a task once, completing it consistently, and producing a valid measurement are separate accomplishments. A single successful trace does not establish all three.

What can you inspect in Nodus today?

Nodus’s documented Beta agent-evaluation workflow exposes the distinctions needed to investigate this problem. A fixed evaluation batch records the selected environment version, task split and seed. The results view shows scored cases, pass rate, mean reward, pending cases, and execution or grading failures separately.

Start with the case counts rather than the aggregate percentage. Then inspect the cases that changed category between runs. The per-case result links back to the agent run and includes grading evidence, so the investigation can move from a percentage to a specific attempt.

The documented behavior also preserves an important boundary: a wrong or invalid answer is scored; a failed run, missing answer or failed grader is unscored. A completed answer that cannot be graded before the evaluation deadline is reported as an evaluation timeout rather than silently counted as correct.

This is product behavior described in the current documentation, not a claim that Nodus automatically diagnoses every failure or improves the model. It gives you records to inspect. You still need a valid task set, a suitable rubric, and judgment about what the failures mean.

Keep the evidence you need before deleting an evaluation group. The current documentation says deletion removes its runs and records; per-case payload retention is also limited. A score screenshot is not a substitute for the underlying cases.

How should you report a result that has missing cases?

Lead with the result and the missing evidence together.

For the illustrative Run B, a useful summary is: “60 of 100 tasks have confirmed successful outcomes. All 60 scored cases passed; 40 cases remain unscored.” Follow it with the reason breakdown and the settings used for the run.

That statement is less impressive than “100% pass rate,” but much more useful. It tells an engineer where to investigate and tells a stakeholder how much of the original workload has actually been verified.

Do not quietly replace the first run with its best retry. Keep the original record, identify the retry, and say whether it used the same conditions. Otherwise the report measures a changing process while appearing to measure a stable agent.

What else should you know?

Should every timeout count as an incorrect answer?

Not automatically. For task delivery, a missed deadline can count as an unsuccessful outcome. For a correctness benchmark, follow the predefined rubric and distinguish a scored failure from a case that never received a valid judgment.

Can you rerun only the unscored cases?

Yes, for diagnosis or completion. Report those retries separately, including any changed settings. If you want a fresh model comparison, use the same full task set and a stated retry policy for both candidates.

Does a fixed seed make an agent evaluation deterministic?

No. Fixing task selection does not necessarily fix model sampling, external services, or every execution condition. Preserve those settings and repeat runs when the decision depends on consistency.

Does a higher mean reward solve the denominator problem?

No. If a mean is calculated only over scored cases, changes in which cases receive scores can change the mean too. Report coverage and the outcome distribution beside it.

Product behavior checked October 14, 2026. Agent evaluation remains in Beta. No experiment was run for this article.

If you are evaluating an agent and want to see how Nodus presents completed work and unresolved cases, book a conversation with the Nodus team.