# How do you decide which synthetic examples belong in a fine-tuning dataset?

> Accept synthetic examples for factual support and useful task coverage, not fluency. A worked audit separates invalid answers from correct but redundant data.

Source: https://www.nodus-compute.ai/blog/audit-synthetic-fine-tuning-data/
Build revision: 7b37c4bab16ee8776cdf5b8839e021253d37edbe

**Beta:** this feature may change.

Keep a synthetic fine-tuning example only when its target answer is supported by authorized evidence, can be produced from the information the deployed model will receive, and teaches a decision your dataset needs. Correctness is necessary, but it does not make another paraphrase valuable. Test the accepted additions against a trusted-data baseline before treating a larger dataset as an improvement.

[Nodus Training (Beta)](https://www.nodus-compute.ai/docs/guides/training/) supports supervised fine-tuning on prepared prompt-and-answer data. That makes dataset acceptance a separate decision from completing the run: successful training does not certify the generated targets. The practical question is which examples deserve to influence the model’s behavior.

## Start with the behavior, not the generated count

Suppose a document assistant must extract a deadline from an attached agreement, ask for the missing attachment when necessary, and decline to invent a date. Generating more well-written deadline answers is not automatically useful. The missing skill might be recognizing when no answer is supported.

Describe the decisions the model should learn before requesting synthetic examples. For this hypothetical assistant, the important distinctions are a stated date, a conditional date, an absent date, and an unavailable document. Those distinctions determine what to collect and what evidence a reviewer needs.

A polished answer to an underspecified question can teach exactly the wrong behavior. If the prompt lacks the document but the target gives a confident date, the example rewards guessing. It is not rescued by the fact that the date exists somewhere in the generator’s source material.

## Apply three acceptance tests

**Is the target defensible?** Review the answer against the permitted source and the task’s acceptance rule. Check consequential details such as dates, quantities, qualifications and exceptions. A plausible rationale is not evidence. When the task has an objectively checkable answer, prefer that direct check to a general impression of quality.

**Could the deployed model know this?** Inspect only the information the model will actually receive. A generated answer may rely on an omitted passage, a previous conversation or privileged background material. Either supply appropriate context that will genuinely be available at use time, rewrite the task, or reject the example. Do not teach access to information the application will not provide.

**Does it add a needed decision?** Compare the example with the trusted dataset. Another wording of an already common easy case may add less than a carefully reviewed exception. Count coverage of meaningful task conditions, not just distinct strings. Different wording does not necessarily mean different evidence, reasoning or behavior.

These tests serve different purposes. Unsupported targets are quality failures. Correct but redundant examples are an allocation decision. Keeping the two categories separate helps you improve the request for examples instead of treating every unused row as equally bad.

## Worked audit: 600 generated examples do not mean 600 useful additions

The following figures are **illustrative**, not a measured experiment, a Nodus benchmark or a customer result. Assume an engineer has 600 generated examples for the document assistant. Each candidate is assigned one primary disposition, so the counts below do not overlap.

|Audit disposition|Examples|Decision|
|-|-|-|
|Target contradicts or exceeds the supplied evidence|90|Reject|
|Target requires context missing from the prompt|60|Repair and re-review, or reject|
|Repeats an already accepted task with cosmetic changes|150|Exclude from this addition|
|Supported, answerable and not redundant with accepted tasks|300|Consider for the training mix|

The 300 remaining candidates contain 240 routine extractions, 40 clarification cases and 20 cases where the document does not support an answer. The trusted dataset already has many routine extractions. An illustrative selection might keep 80 routine cases plus all 40 clarification and 20 no-answer cases, for 140 additions.

The other 160 routine cases are not necessarily incorrect. They are deferred because this experiment is aimed at missing decisions, not maximizing row count. The selected proportions are not a universal recipe or an estimate of production traffic. They deliberately emphasize a learning gap.

That distinction matters during evaluation. Report results on a representative task set, and separately inspect clarification and unsupported-answer cases. Otherwise an intentionally enriched training mix can be confused with the real frequency of those situations. Review whether additional refusals fix guessing without causing the model to decline answerable questions.

## Do not let the generator certify itself

A generator can critique its own answers, but agreement with that critique is weak evidence when both share the same misunderstanding. Ask reviewers to inspect the task and source rather than merely endorse the explanation that came with the candidate.

If another model screens the data, validate its decisions against independently reviewed examples. Yesterday’s reliable screening rule may not cover a newly generated task category. For subjective acceptance, use a documented rubric and retain unresolved cases outside the approved training set until their intended behavior is clear.

Keep provenance sufficient to revisit decisions: the permitted source, generator settings, review outcome and reason for rejection or acceptance. Use only material you are authorized to process and train on. A public page is not, by itself, a complete rights or privacy assessment. Do not send confidential source material to an unapproved service used to produce examples.

## Test the addition, not just the final checkpoint

Compare a trusted-data-only baseline with a candidate that includes the reviewed synthetic examples. Use the same starting model, task evaluation and declared training allowance, and document material differences in the training mix. If adding data changes exposure to trusted examples, that is part of the experiment, not an invisible detail.

Choose the candidate on development evidence, then evaluate it on untouched, independently sourced task cases. Avoid having the generator produce both the training examples and the only test of success. A shared template or shared misconception could make the result look reassuring without testing the intended behavior.

Inspect where the model improved and what it traded away. For the document assistant, measure supported answers, invented details, unnecessary clarification and inappropriate refusal separately. Better presentation is useful only if it does not conceal worse evidence use. If the addition fails, revise the specific category responsible rather than assuming the solution is more generated rows.

Nodus’s Beta training guide documents validation-based checkpoint selection and downloadable model outputs. It also states that validation loss guides selection while a separate untouched test set must establish task quality. Use those capabilities for the training experiment, not as a substitute for the dataset audit. The product documentation does not establish that your synthetic examples are correct or that their addition improves your application.

## FAQ

### Is a stronger model’s answer automatically a good training target?

No. It can still be unsupported, use unavailable context or repeat an unneeded pattern. Judge the example against the task contract, not the generator’s reputation.

### What percentage of the dataset should be synthetic?

There is no universal percentage. Start with the specific behavior gap, accept examples that address it, and compare the addition with a trusted-data baseline. Track useful coverage and task outcomes, not a target ratio.

### Should I discard every duplicate?

Not necessarily. Repetition changes emphasis and may be intentional. Distinguish deliberate weighting from accidental duplication, and do not present repeated versions of one case as independent coverage.

### Can I use synthetic examples in the test set?

They can support targeted diagnostics, especially for rare conditions. They should not be the only evidence of generalization when training data comes from the same generator. Keep representative, independently sourced evaluation cases as well.

If you are preparing a fine-tuning experiment, [book a conversation with Nodus](https://nodus-compute.ai/book/) to discuss the behavior you need to improve and whether the current Beta training offering fits.