# How should you split fine-tuning data when prompts repeat?

> Split related prompts together, not individual rows. Learn when exact-prompt grouping is enough, when to hold out whole documents, and what Nodus supports.

Source: https://www.nodus-compute.ai/blog/fine-tuning-data-split-repeated-prompts/
Build revision: 5003a113a82ee95ed420f52eff43ba3490d31e58

**Beta:** this feature may change.

**Split by the independent unit you want the model to generalize to, not automatically by row.** Keep repeated prompts and their answers together. If several prompts come from the same document, conversation or underlying case, consider holding out that entire group. Use validation to choose a model, then use a separate untouched test set to judge the chosen model.

[Nodus’s training workflow](https://www.nodus-compute.ai/docs/guides/training/) is in **Beta** and supports validation-based checkpoint selection that keeps repeated prompt identities together. That protects against exact prompt overlap. It does not establish that paraphrases, related documents or repeated business cases are independent. That distinction determines what your evaluation can honestly claim.

## Why is a random row split risky for fine-tuning?

One question can produce several training rows: alternative acceptable answers, preference pairs, edited responses or conversation excerpts. Those rows are not necessarily independent examples of the task.

If an answer to a question appears in training and another answer to the same question appears in validation, the validation result partly measures performance on a familiar question. That can be appropriate if familiar questions are your actual target. It is weak evidence for performance on new questions.

The problem is not randomness itself. It is randomizing at the wrong level. A random split of independent question groups can be useful; a random split of correlated rows can undermine the separation you intended.

The same reasoning appears in [scikit-learn’s guidance on grouped cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data): related observations should remain in the same group when independence between training and evaluation matters. For language-model data, deciding what counts as related requires knowledge of how the examples were collected.

## What does leakage look like in a small example?

Consider a dataset with 100 distinct questions and five acceptable answer variants per question, for 500 rows total. These are **illustrative numbers, not an experiment or a Nodus customer result**.

Compare two possible assignments:

|Assignment|Training set|Validation set|What validation tests|
|-|-|-|-|
|Split each question’s answer variants|Four variants for each of 100 questions: 400 rows|One variant for each of those same questions: 100 rows|Familiar questions with held-out answer variants|
|Hold out entire questions|All five variants for 80 questions: 400 rows|All five variants for 20 different questions: 100 rows|Questions absent from this fine-tuning set|

The row counts are identical. The evidence is not. The first assignment has complete question overlap; the second has none between these two sets.

This table is a constructed comparison, not a prediction that every random row split produces exactly that first assignment. It also says nothing about whether the base model encountered those questions before fine-tuning.

Grouping does not remove repeated examples or fix uneven weighting. A question with many answer variants can still dominate a row-averaged metric. Report the number of independent questions as well as rows, and inspect whether frequently repeated questions overwhelm the cases you actually care about.

## Should you group by prompt, document or customer case?

Choose the group based on the intended deployment claim.

|Claim you want to test|Candidate holdout unit|Remaining concern|
|-|-|-|
|Handle unseen exact questions|Prompt identity|Paraphrases can cross the split|
|Answer questions about unseen source material|Source document or document family|Related revisions can remain too similar|
|Handle a new support issue|Conversation or underlying case|Excerpts can reveal the same resolution|
|Generalize to future requests|Later time period, with related cases kept together|The future distribution can still change|

These are experimental design choices, not promises that Nodus automatically identifies those groups.

For a document-answering dataset, holding out exact question strings may be insufficient. Training could contain one question about a report while validation contains a slightly reworded question about the same passage. If your claim is generalization to new reports, keep related questions and report revisions together before creating the partitions.

Conversely, holding out entire document families can test a harder problem than your application needs. If production repeatedly queries a fixed approved corpus, evaluate that use case explicitly. There is no universally correct split independent of the question you are trying to answer.

## What does Nodus support, and where is the boundary?

The current **Beta** training guide describes validation-based selection for supervised fine-tuning, pretraining, preference training and distillation. Repeated prompt identities stay together, including different answers to the same prompt. The validation fraction counts identities, so the proportion of rows can differ when some prompts have more examples.

That means a nominal validation fraction is not a guarantee of the same fraction of rows or tokens. Inspect the resulting composition before treating the split as balanced.

The documented minimum is at least 20 rows and enough distinct identities to make both partitions nonempty. That is an eligibility requirement, not evidence that such a small validation set is statistically adequate for your workload.

The guide explicitly limits the protection to exact prompt leakage, not semantic near-duplicates. If you need document-level, case-level or time-based separation, prepare and verify that separation yourself. Do not assume a built-in prompt holdout enforces those broader boundaries.

With validation selection configured, Nodus evaluates the starting model and exports the best evaluated checkpoint by validation loss. Without that configuration, the guide describes final-checkpoint export instead. Neither behavior turns validation loss into proof of better task performance.

## How should you decide whether the selected model is better?

Keep three questions separate:

1. **Did training fit its examples?** Training loss describes the optimization target on the training material.
2. **Which candidate should you choose?** Validation supports checkpoint and recipe selection under the split you defined.
3. **Does the chosen model improve the real task?** An untouched test set and a task-appropriate rubric answer that question.

Compare the chosen model with the starting model on the same held-out tasks and under the same evaluation conditions. Inspect formatting failures and important workload slices, not just one average.

If you repeatedly change the recipe after inspecting test results, that set is participating in selection. Treat it as development evidence and reserve new independent cases for the next final comparison. A better split cannot repair a test set that has become part of the tuning loop.

## Frequently asked questions

### Is exact deduplication enough?

No. Removing identical rows does not necessarily remove repeated prompts with different answers, paraphrases, or multiple excerpts from the same source. Decide which relationships would make your evaluation too easy, then keep those examples together.

### Does grouping guarantee a worse validation score?

No. It changes the question the evaluation asks. Scores might rise, fall or remain similar. Do not claim a measured leakage effect without running a controlled comparison.

### Should I use a fixed validation percentage?

A percentage alone is insufficient. Consider independent groups, workload coverage, example lengths and the precision needed for your decision. A large row count can still contain very few independent cases.

### Can lower validation loss justify deployment?

Not on its own. The Nodus training guide calls for a separate untouched test set to establish task quality. Use validation to select; use representative task outcomes and your acceptance criteria to decide whether to deploy.

If you want to discuss whether Nodus’s **Beta** training workflow fits your fine-tuning and evaluation process, [book a conversation](https://nodus-compute.ai/book/). Bring the outcome you want to measure and how your examples are related, not only the dataset’s row count.