# How do you calibrate an LLM judge before trusting its scores? > Test an LLM judge against human decisions, inspect false approvals and slice-level errors, and reserve an untouched audit before using its scores to select a model. Source: https://www.nodus-compute.ai/blog/calibrate-llm-judge/ Build revision: d90e9c7b540ab1af530cb77220dea181db6f37e3 **Beta:** this feature may change. Calibrate an LLM judge by comparing its decisions with independently reviewed human labels on representative answers, measuring the errors that matter to your release decision, and testing the final rubric on an untouched audit set. Do not trust an agreement percentage without checking how often the judge approves unacceptable answers. If you use Nodus Training (Beta), trained weights and evaluation results support your model-selection workflow; the judge calibration described here is a customer-side acceptance step, not a built-in Nodus judge or certification. A model judge can make review faster, but it can also make a mistaken definition of quality repeatable. The important question is not whether it sounds like a careful reviewer. It is whether its decisions are reliable enough for the particular job you intend to delegate. ## Define the decision before choosing the judge Start with what the score will control. Screening obvious failures for human review, ranking two model candidates, and automatically approving answers are different decisions. A judge suitable for triage may be unsuitable as a release gate. For a hypothetical document assistant, an acceptable answer might need to include the requested facts, support each claim with the supplied document, and avoid inventing missing information. Fluency alone would not satisfy that contract. Write a rubric that distinguishes factual support, completeness, and presentation rather than letting polished prose compensate for unsupported claims. Use objective checks where the requirement is objective. A required field, an exact extracted value, or a verifiable task result should not become a subjective preference question merely because a language model can discuss it. Reserve judgment for the parts that actually require judgment. ## Build human reference labels that deserve comparison Collect authorized examples from the intended task distribution or construct representative test cases. Include ordinary successful answers, clear failures, and difficult boundary cases. Preserve the question and the evidence available to the answerer so reviewers judge the same information the application had. Have reviewers apply the written rubric without seeing the judge’s decision. Where practical, hide the identity of the model that generated the answer. Independently label an overlapping subset to reveal disagreements between people, then adjudicate consequential differences and document why they occurred. A human label is a reference decision under a rubric, not infallible ground truth. If reviewers disagree about whether a missing qualification matters, clarify the acceptance rule before asking the judge to imitate it. Do not erase genuine ambiguity by forcing every case into an apparently certain label. Mark unresolved cases separately and decide how production should route them. Keep development examples separate from the final audit. Rewriting the judge’s instructions after reading audit errors turns those examples into development data. Another untouched audit is then needed for a fresh estimate. ## Worked example: 90% agreement can mean no useful screening The following numbers are **illustrative**, not a measured experiment, a Nodus benchmark, or a customer result. Assume reviewers have resolved the reference labels for 200 answers: 180 acceptable and 20 unacceptable. A judge that approves every answer agrees with the reference on 180 of 200 cases, or 90%. It also misses every unacceptable answer. That can look impressive in a summary and be useless for rejection screening. Consider a second illustrative judge on those same answers: |Human reference|Judge approves|Judge rejects| |-|-|-| |Acceptable answer|171|9| |Unacceptable answer|4|16| This judge agrees on 171 + 16 = 187 cases, or 93.5%. More importantly, the table exposes the decisions that an average hides: * **Missed failures:** 4 of the 20 unacceptable answers are approved, or 20%. * **False alarms:** 9 of the 180 acceptable answers are rejected, or 5%. * **Contamination among approvals:** 4 of 175 approved answers are unacceptable, about 2.3%. These percentages answer different questions. If rejected answers go to reviewers, review volume and missed failures matter. If approved answers are published without review, what slips through the approval bucket matters. None of these figures establishes a universal safe threshold. Twenty unacceptable examples provide limited evidence about rare failures. A deliberately failure-heavy test set can help diagnose weaknesses, but its approval contamination does not directly estimate deployment contamination when the real mix differs. Evaluate a representative sample as well, and keep targeted stress tests visible as a separate view. ## Test whether irrelevant changes move the verdict The research paper [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) documents limitations including position and verbosity biases. Its findings are a reason to test your own judge, not a promise of agreement in your application. For pairwise comparisons, reverse which answer appears first while preserving the answers themselves. Track how often the preference changes. If presentation order changes the winner, a single ordering is weak evidence for choosing a model. For individual-answer review, compare concise and expanded versions that preserve the same task-relevant facts. Check whether additional polish changes acceptance despite unchanged correctness. Also include a fluent answer with a consequential unsupported claim. A judge that rewards length or confidence may prefer it over a shorter, properly bounded answer. Treat candidate text as material to evaluate, not as instructions for the evaluator. Include cases where an answer asks the judge for a favorable score. A successful test on these examples is useful evidence, not proof that all adversarial answers are covered. Keep these diagnostic sets separate from ordinary task reporting. Otherwise a test designed to expose a specific bias can be mistaken for an estimate of how often that bias appears in production. ## Approve a scope, not a judge in the abstract Break down errors by the distinctions that matter: language, answer length, task type, missing evidence, and consequential failure category. An acceptable overall result can conceal a weak slice. Avoid overinterpreting tiny slices; collect more examples where a decision depends on them. Record the judge version, instructions, rubric, reference labels, and reviewed answer set. Changing any of these can change the measurement. If you replace the judge, revise the rubric, or move to a new application, rerun the relevant audit rather than carrying forward an old agreement claim. On [Nodus Training (Beta)](https://www.nodus-compute.ai/docs/guides/training/), validation-based checkpoint selection and downloadable model outputs are documented customer capabilities. The same guide explicitly separates validation loss from task quality on an untouched test set. Use that distinction here: selecting weights and deciding whether their answers are acceptable are separate steps. Nodus does not make an uncalibrated external judge trustworthy simply by completing a training run. ## FAQ ### What agreement percentage is good enough? There is no universal percentage. Choose tolerances for missed failures and false alarms according to what the decision controls, then inspect those errors on relevant examples. An average alone cannot establish readiness. ### Can the model judge its own answers? It can produce a critique, but that is not independent evidence of correctness. Test agreement with independently reviewed reference decisions and check whether the judge favors answers associated with itself. Keep consequential cases under appropriate human review. ### Do I need a judge when an answer is objectively checkable? Usually, start with the direct check. Subjective review can assess additional qualities, but should not override a failed factual or task requirement because an answer reads well. ### When should I recalibrate? After a meaningful change to the judge, rubric, candidate answer distribution, or application requirements, and when ongoing human audits reveal new failure patterns. Calibration applies to a defined decision and task distribution, not permanently to a model name. If you are planning a training and evaluation workflow, [book a conversation with Nodus](https://nodus-compute.ai/book/) to discuss your evaluation goals and current product fit. Bring the acceptance decision you need to make, not just the score you want to increase.