How do you test an LLM's robustness to paraphrased prompts?
The same request can arrive in different words. Test whether the decision survives the rewrite without rewarding a model for giving the same wrong answer.
Test paraphrase robustness by grouping different wordings of the same request, verifying that each wording preserves the facts and expected decision, and scoring every response against that decision. Report both individual-response accuracy and the share of case families answered correctly across every tested wording. Agreement alone is insufficient: a model can give the same wrong answer consistently.
Nodus Inference is Beta and lets you try available open-weight models in the Playground. If you are comparing candidates for a customer-facing assistant, use realistic variations of the requests customers bring. Access to a model does not establish that its decisions survive a rewrite; you need evidence from your own task.
Decide what should stay the same
Section titled “Decide what should stay the same”A paraphrase test asks whether irrelevant wording changes alter a meaningful result. It does not require identical prose. Two support answers can use different sentences while reaching the same correct conclusion and preserving the same important qualifications.
Define that conclusion before writing variants. For a product-return assistant, it might include eligibility, the applicable exception and whether more information is required. Decide whether omitting an exception makes an answer wrong, even if the main eligibility label is correct.
Keep these properties fixed within a meaning-preserving family:
- The relevant facts and any supplied reference material.
- The action the user requests and their authority to request it.
- Important conditions, qualifications and uncertainty.
- The standard used to judge the answer.
Then vary realistic expression: sentence order, ordinary synonyms, a direct question versus a polite request, or a concise description versus a longer equivalent one. Do not add clues that effectively explain the answer in only one version.
This distinction follows a useful idea from CheckList: behavioral evaluation should test capabilities and expected relationships between inputs, rather than rely on a single aggregate accuracy score. The method here applies that idea to your application’s request families; it is not a claim that a historical benchmark establishes any current model’s performance.
Verify equivalence before blaming the model
Section titled “Verify equivalence before blaming the model”A rewrite can change the task unintentionally. “Can I return this?” is not necessarily equivalent to “Will I receive a cash refund?” A return may be accepted for store credit without qualifying for cash. Likewise, removing a purchase date can turn an answerable question into one requiring clarification.
Have a reviewer compare each variant with the original and the expected decision. For consequential use, have another reviewer resolve disputed cases before model scoring. Record whether the input is equivalent, changed in a meaningful way, or ambiguous.
Keep changed-fact tests in a separate set with their own expected answers. Contrast-set research uses small, meaningful edits that typically change the correct label. Those edits test whether a model responds to relevant changes, not whether it ignores them. A good assistant needs both capabilities: stability under equivalent wording and sensitivity to changed facts.
Exclude unresolved equivalence disputes from the paraphrase score and report how many you excluded. Do not quietly keep the variants that make a candidate look best.
Worked example: the same return request, four wordings
Section titled “Worked example: the same return request, four wordings”Illustrative example, not a customer policy or measured model result. Suppose a fictional shop accepts ordinary returns within 30 days, but a final-sale item is ineligible unless it arrived damaged. The supplied facts say that the item was marked final sale, arrived undamaged and was purchased 12 days ago.
These requests preserve the same facts:
| Wording | Expected decision |
|---|---|
| “I bought this final-sale item 12 days ago. It arrived undamaged. Can I return it?” | Ineligible under the supplied policy |
| “Does the return window cover my undamaged final-sale purchase from 12 days ago?” | Ineligible under the supplied policy |
| “Please check return eligibility: purchased 12 days ago, final sale, no damage on arrival.” | Ineligible under the supplied policy |
| “My purchase was 12 days ago and arrived without damage. It was final sale. Is a return allowed?” | Ineligible under the supplied policy |
The response should apply the exception, not simply notice that 12 is less than 30. It must not invent a goodwill refund or claim that a return has been arranged.
Now change “arrived undamaged” to “arrived damaged.” That is a separate contrast case. Under this fictional policy, the eligibility decision should change. Rewarding an unchanged answer would reward ignoring the most important fact.
For an illustrative evaluation, create ten independent scenarios with four equivalent wordings each. Suppose six scenarios receive four correct answers, while four scenarios receive three correct answers and one wrong answer. Individual accuracy is 36 of 40, or 90%. But only six of ten families, or 60%, pass across all tested wordings.
Neither percentage estimates every possible rewrite a customer might use. The family score exposes a weakness hidden by the individual score, while remaining specific to the variants you actually tested.
Measure correctness, stability and baseline difficulty separately
Section titled “Measure correctness, stability and baseline difficulty separately”Report three views rather than compressing the test into one number.
Individual accuracy counts correct responses across all tested wordings. If families contain different numbers of variants, a large family contributes more responses. Use equal family sizes for a simple comparison, or report family-weighted results explicitly.
Complete-family success counts families for which every tested variant is correct. Always disclose the number and type of variants. Adding harder or more numerous rewrites can lower this score without any change to the model, so compare candidates on the same fixed families.
Decision consistency checks whether equivalent inputs produce equivalent conclusions. It is diagnostic, not a replacement for correctness. A consistently incorrect model can score perfectly on agreement.
Also report original-wording accuracy. If a model fails the original and every rewrite, that is evidence of task difficulty, not a special success on robustness. Inspect families where the original passes but a variant fails, and families where the reverse happens. Do not assume the original wording is inherently the fairest wording.
Keep repetition and test selection honest
Section titled “Keep repetition and test selection honest”Run each variant in a fresh conversation unless conversation history is part of the intended test. Otherwise an earlier answer can influence a later answer and change what you are measuring.
Use the same instructions, supplied evidence and supported response settings across the comparison. When responses vary on repeated identical inputs, repeat each wording under a predefined protocol. Report within-wording variation separately from differences across wordings. One inconsistent pair does not isolate wording as the cause.
Keep every variant of a scenario together when separating development and final evaluation data. Tuning on one wording and testing on its close rewrite can overstate generalization to new scenarios. Choose final families before comparing candidate results, and do not repair failed cases in the final test while retaining its original headline score.
Use observed customer language where you have permission, removing private details. Generated variants can broaden coverage, but human review still needs to establish equivalence and relevance. This tests prompt robustness, not broad security, multilingual quality or every possible user behavior.
If you are choosing an inference model, book a conversation with Nodus to discuss available options and the behavior your application needs. Bring the case families and decision criteria, not just a leaderboard score.
Must robust responses have identical wording?
Section titled “Must robust responses have identical wording?”No. Judge the decision and required qualifications. Surface differences can be harmless; identical wording can conceal a consistently wrong conclusion.
Should changing a number count as a paraphrase?
Section titled “Should changing a number count as a paraphrase?”Usually not. A number may change eligibility, urgency or the required answer. Review it as a changed-fact case unless task-specific evidence establishes equivalence.
Does passing every variant prove robustness?
Section titled “Does passing every variant prove robustness?”No. It establishes success on the tested families and wordings under the recorded conditions. Report coverage and limitations rather than claiming universal stability.
Can an LLM review whether its own rewrites are equivalent?
Section titled “Can an LLM review whether its own rewrites are equivalent?”It can suggest candidates for review, but its agreement is not independent validation. Human-reviewed facts and expected decisions should anchor the final evaluation.
More from the blog
gpu9 min read
How do you fine-tune Gemma with LoRA on a cloud GPU?
Gemma 4 LoRA and QLoRA by model size: VRAM, the GPU each size needs, chat-template and attention traps, and a recipe that survives restarts on Nodus.
Read the postdistributed9 min read
FSDP vs DDP: when do you need sharded data parallel?
Use DDP until params, grads and AdamW state stop fitting on one GPU, about 3.75B params on an 80 GB H100. Then shard with FSDP. The memory math, worked.
Read the post