# How do you evaluate an LLM for multilingual customer support? > Compare support models by correct resolutions in each language, not fluency alone. Use matched intents, native-language cases, local policy checks and explicit release gates. Source: https://www.nodus-compute.ai/blog/evaluate-multilingual-support-llm/ Build revision: d78dff32a1649c5255309de8ff0d98133c2f5edf **Beta:** this feature may change. **Evaluate a multilingual support LLM by whether it resolves the customer’s problem correctly in each supported language, under the policy and account context that actually apply.** Score fluency separately from factual correctness, permitted actions and appropriate escalation. Combine matched intents across languages with cases written naturally in each language. Do not let a strong aggregate score authorize a language you have barely tested. Nodus offers hosted-model inference in **Beta**. You can use the available models as candidates for a support evaluation, but language coverage and support quality are properties you must establish for your support tasks. The [inference guide](https://www.nodus-compute.ai/docs/guides/inference/) describes the offering and current catalog availability; it does not establish a multilingual support benchmark. ## Define the resolution before judging the reply A support answer needs more than grammatical sentences. Establish what an acceptable resolution means for each task: the correct policy, an authorized next step, necessary qualifications and a customer who can understand what happens next. For a cancellation request, a friendly explanation can fail if it promises a refund the policy does not permit. A cautious answer can also fail if it unnecessarily sends an eligible cancellation to a person. Keep both errors visible rather than rewarding confident prose or universal escalation. Give reviewers distinct questions: * Does the response understand the customer’s intent and relevant details? * Does it apply the supplied policy for the correct product, account and market? * Does it avoid invented actions, promises and account facts? * Is the language understandable, appropriate and faithful to those decisions? * When a person must take over, does the handoff preserve the customer’s issue? Define critical failures before comparing candidates. A promise of an unauthorized refund may be unacceptable even when the rest of the answer is excellent. Do not average that failure away with a high politeness score. ## Separate language from market and policy Language is not a reliable substitute for location, currency or eligibility. A Spanish-speaking customer might have an account governed by a policy written for a different market from the one a reviewer initially assumes. Represent these as separate facts in the evaluation: the language of the conversation, the account’s applicable policy and the contextual details required to answer. Include matched cases where only the language changes, and separate cases where only the applicable market changes. In the first comparison, the permitted resolution should remain equivalent. In the second, it may legitimately differ. Without that separation, a model that changes policy whenever it changes language can look localized while making the wrong decision. Dates, address conventions and amounts deserve the same care. Test whether the answer preserves an explicitly supplied date or amount without silently interpreting it through an assumed locale. When the customer’s meaning is genuinely ambiguous, define what clarification is necessary instead of accepting an unsupported guess. ## Use matched intents and native-language cases Build two complementary groups. **Matched-intent cases** hold the customer goal, account facts and policy constant across languages. Have qualified reviewers check that each version preserves the same meaning and difficulty. These cases help reveal whether a correct decision survives a language change. **Native-language cases** start with how customers actually express problems in that language. Use authorized, appropriately de-identified conversations or have fluent domain reviewers author realistic cases. Include shorthand, spelling variation and mixed-language messages when they belong to the intended conversations. Translations alone can miss these patterns. Conversely, entirely different cases in each language make it hard to tell whether a score difference reflects language or task difficulty. Keep both groups and report their results separately. Keep evaluation cases out of prompt selection and tuning. If several versions come from the same underlying conversation, keep that family together when separating development cases from the final test. Treat changes to the policy packet or customer population as reasons to revisit the evaluation. ## Worked example: a cancellation request The following is an **illustrative evaluation scenario, not a customer story or a measured model result**. A fictional subscription business supplies a policy: an unused plan may be canceled before activation; an activated plan requires a support review. The test packet explicitly states that the account has not activated its plan. The customer asks to cancel. Create matched versions in the languages you intend to support. An acceptable response should identify the applicable cancellation path, avoid claiming that an action has already occurred and explain the next step clearly. It must not invent a refund policy that the packet does not contain. Then vary the case deliberately: |Variation|What a correct response must preserve or change| |-|-| |The same account facts, another language|Preserve the cancellation decision and its qualifications.| |The account is now activated|Change to the support-review path.| |Activation status is missing|Ask for the missing fact or follow the specified review policy.| |The customer mixes languages|Preserve the intent and account details; follow the defined response-language policy.| |The customer requests a refund as well|Do not invent refund eligibility from the cancellation rule.| Judge meaning, not identical wording. A natural reply can differ substantially across languages while preserving the same decision. Record which requirement failed when a polished answer chooses the wrong path. ## Review in the customer’s language Use reviewers who can assess both the language and the support domain. A translation back into a shared language can help investigate a failure, but it should not be the sole basis for acceptance. It can hide register problems or alter the meaning being judged. Have reviewers assess candidate replies without seeing the model name. Provide the same policy packet and acceptance rules. Resolve disagreements by identifying the disputed requirement, not by choosing whichever wording sounds more professional. An automated judge can assist triage after calibration against human judgments in each relevant language. Do not assume calibration in one language transfers to another. Review difficult disagreements and critical failures directly; a second fluent model is not independent proof that the answer is right. ## Make a release decision for each language Report correct resolutions, policy violations, unnecessary escalations and language-quality failures for each supported language. Show the number and mix of cases behind every rate. A small set with no observed failures is limited evidence, not proof of reliability. Also show a traffic-weighted aggregate if it helps planning, but do not use it as the only release gate. The highest-volume language can dominate that number while a smaller language fails important tasks. Examine task categories and markets within each language where the consequences warrant it. Compare candidate models on the same cases and inspect paired disagreements. Keep policy context, answer constraints and support permissions constant. Track review effort and response time alongside quality, without turning a faster wrong answer into a successful resolution. A reasonable outcome can be a narrower launch: selected languages and intents automated, with the remaining cases handed to people. Label that boundary honestly and retest it as customer requests change. ## Where Nodus fits Nodus Inference provides access to hosted models in **Beta**. Check current model capabilities and availability before selecting candidates. Request receipts can help account for inference charges during comparisons; they do not replace support-quality review or establish the full cost of human handling. If you are evaluating hosted models for a multilingual support application, [book a call](https://nodus-compute.ai/book/) to discuss fit and current availability. ## FAQ ### Is a translated English test set enough? No. It is useful for matched-intent comparisons, but add naturally written cases that represent the language and customer behavior you intend to support. Check translations for equivalent meaning and difficulty. ### Should every language use the same acceptance threshold? Set requirements from task consequences and customer commitments, not from whichever threshold makes a model pass. Different observed uncertainty may justify different launch scopes, not weaker protection against consequential errors. ### Can one model handle every language? Possibly, but that is an evaluation outcome, not an assumption. Establish quality per language and task before deciding whether a single candidate meets the full scope. ### What matters more: fluency or resolution accuracy? Both are necessary. Score them separately and require the response to meet both standards. Neither fluent misinformation nor an incomprehensible correct answer is an acceptable customer resolution.