How do you evaluate an LLM classifier when positive cases are rare?

Nov 12, 2026 - 7 min read

Evaluate a rare-event LLM classifier by checking whether it catches enough positive cases without producing more flags than people can review. Measure recall and false-positive rate separately, combine them with the expected positive rate and traffic volume, and test the resulting review demand. A high accuracy score or a strong precision score on a deliberately balanced test set does not establish that the classifier fits your operation.

Nodus Inference is Beta and lets you try available open-weight models in the Playground. Use that access to compare candidate behavior on your labeled examples. Model availability is not evidence of classification quality, calibrated confidence or a built-in human-review process.

Define the action, not just the label

Suppose a classifier screens incoming messages for cases needing escalation. A positive prediction sends a message to a reviewer; a negative prediction does not. That action makes two errors costly in different ways: a missed positive delays attention, while a false alarm consumes review capacity.

Define what genuinely requires escalation before comparing models. Resolve ambiguous examples with the people responsible for the decision. Keep authorized data, remove unnecessary private information, and record the period and source of the examples.

For this screening decision:

Recall is the fraction of actual positives the classifier flags.
False-positive rate is the fraction of actual negatives it incorrectly flags.
Precision is the fraction of flagged messages that are actual positives.

Review is a separate step. Screening recall does not prove that reviewers resolve the flagged cases correctly or on time.

Why can good test precision disappear in production?

Precision depends on how common positives are. The same recall and false-positive rate can produce very different flag quality when the positive rate changes.

Consider an explicitly illustrative calculation, not an observed result or a Nodus benchmark. Assume 20,000 messages, 90% recall and a 2% false-positive rate:

Positive share	Actual positives	Correct flags	False flags	Precision
10%	2,000	1,800	360	83.33%
1%	200	180	396	31.25%

At 1% prevalence, the classifier faces many more negatives for each positive. Even a small false-positive rate can therefore dominate the flagged set. The lower-prevalence population produces fewer total flags here, but a much larger fraction of those flags are false alarms.

This comparison assumes recall and false-positive rate stay unchanged. That is a planning assumption, not a promise: a different channel, language mix or case difficulty can change both rates too.

A test set enriched with rare positives helps inspect those cases. Its raw precision describes the enriched mixture, not automatically the incoming stream. Preserve class counts and conditional error rates, and validate the decision on representative held-out traffic. Do not relabel enriched-set precision as production precision.

Convert error rates into review demand

For a binary screen where every flag goes to review, use these quantities:

N: messages per day.
p: fraction of messages that are genuinely positive.
r: recall among positives.
f: false-positive rate among negatives.

Then the planning calculation is:

Daily flags = N × [p × r + (1 − p) × f]

Precision = (p × r) ÷ [p × r + (1 − p) × f]

Precision is undefined if there are no flags. These formulas describe expected counts under the stated rates; they are not measured forecasts. Use a range for uncertain prevalence rather than making a precise capacity promise from a weak estimate.

Review capacity also depends on effort. A count budget assumes comparable review time per message. If difficult cases take longer, use measured review minutes instead. Include abstentions, audits and other queues competing for the same reviewers. A daily average does not establish that a burst of cases will meet a response deadline.

Worked decision: neither candidate fits

Continue the illustrative example at 20,000 messages per day and 1% positives. Compare two tested screening policies. They could be two candidate configurations, or two score cutoffs when a suitable score actually exists.

Policy	Recall	False-positive rate	Correct flags	False flags	Total reviews
A	90%	2%	180	396	576
B	70%	0.5%	140	99	239

Suppose the requirement is at least 80% screening recall and at most 250 reviews per day. Those requirements are illustrative too, not recommended defaults.

Policy A meets the recall requirement but exceeds review capacity. Policy B fits the review budget but misses too many positives. Neither tested policy qualifies.

Policy B has better precision, approximately 58.58% versus 31.25%, but that does not repair its recall failure. Choosing it solely because its flagged set looks cleaner would silently change the acceptance rule.

This result does not prove every possible policy fails. Test another operating point or model, increase review capacity, or explicitly revisit the requirement. Do not assume an untested midpoint between A and B achieves both goals. Changing the escalation definition is another decision, not a free improvement to the score.

Select an operating point without inventing confidence

An LLM that returns only a category does not automatically expose a meaningful continuous score. A model-written confidence percentage is not validated merely because it looks like a probability. If a candidate exposes scores, verify what they mean and test their relationship to your labels before treating them as useful decision signals.

When a score supports a cutoff, select the cutoff on development examples and evaluate the locked policy on untouched examples. Keep the model, prompt, label definition and decision rule fixed during that final comparison. Choosing the cutoff after inspecting final-test errors turns the final set into development data.

Report counts as well as rates. A recall estimate based on very few positives cannot support the same confidence as one based on many independent positives. Show uncertainty and failure groups; do not hide a weak channel behind the overall average.

Check representative traffic after release. If prevalence changes, revisit expected precision and review demand. If case mix changes, remeasure conditional error rates rather than only updating the arithmetic.

Which summary metric should you use?

Use precision-recall views to inspect candidate tradeoffs, then select against the actual operating constraints. Saito and Rehmsmeier’s study explains why precision-recall plots can be more informative than ROC plots for imbalanced binary classification. It does not establish how any particular LLM performs.

Neither one area-under-curve number nor F1 contains your daily review budget. F1 combines precision and recall, but a better F1 does not guarantee that both your recall floor and capacity ceiling are satisfied. Compare summary scores only with the dataset mixture and evaluation conditions visible.

The scikit-learn threshold guide distinguishes estimating scores from choosing actions. Apply that distinction to the buying decision: a promising model still needs an acceptable operating policy.

Frequently asked questions

Is accuracy useful when positives are rare?

It is descriptive but insufficient. In the illustrative 1%-positive population, predicting every message negative gives 99% accuracy and zero positive recall. Report the errors that drive the action instead of relying on the headline percentage.

Should the final test set be balanced?

A balanced diagnostic set can expose positive-case failures. For expected flag quality and review demand, also use a representative set or a clearly justified prevalence-adjusted estimate. Keep the diagnostic and operational conclusions separate.

Does a higher threshold always solve review overload?

A stricter cutoff can reduce flags when the score supports that ordering, but it can also miss positives. Recheck both constraints. Label-only candidates need separately evaluated policies, not an imaginary threshold sweep.

What should you compare on Nodus?

Try currently available models against the same authorized labels and acceptance rule. Keep model quality, score usefulness and human-review capacity as separate questions. Nodus Inference remains Beta; this framework does not imply Nodus supplies the evaluation or review system. To discuss model fit for your application, book a conversation.