How do you choose an embedding similarity threshold for deduplication?
Similar documents are not necessarily interchangeable. Audit the documents you would lose before choosing a deduplication threshold.
Choose an embedding similarity threshold by measuring whether the documents it would remove are genuinely replaceable by the documents you keep. Start with a labeled sample from your own collection, compare candidate cutoffs, and review the proposed removals for lost facts, versions and minority cases. A similarity score is a candidate signal, not permission to delete.
For practitioners preparing data with Nodus Jobs, the important boundary is straightforward: running a document-processing command does not establish that its output is safe to train on. Decide what counts as redundant before scaling the run. This guide covers that decision, not a built-in Nodus deduplication feature.
Define what the retained document must preserve
Section titled “Define what the retained document must preserve”Two documents can discuss the same subject without being substitutes. Product instructions might share almost every sentence but differ in a safety warning. Support articles might describe different subscription plans. A revised policy might change one date that determines the answer.
Write the removal rule in terms of the downstream task:
- For a search collection, will the retained document still answer the queries the removed one could answer?
- For a training collection, does removal erase a distinct example, language, source or behavior you need represented?
- For an archive, are you allowed to discard historical versions at all?
The same pair can be redundant for one purpose and essential for another. Where retention obligations or permissions are uncertain, resolve them separately; similarity cannot decide them.
Research such as SemDeDup studies semantic duplication beyond exact matching. Its existence is not evidence that a published cutoff transfers to your documents, embedding model or task. Treat a cutoff as a local decision that needs local evidence.
Label difficult neighbors, not just obvious copies
Section titled “Label difficult neighbors, not just obvious copies”Begin with proposed duplicate pairs spanning the score range you might use. Include easy copies, borderline matches and tempting false matches. Keep a separate random audit sample of the actual proposed removals so an intentionally difficult test does not masquerade as a population estimate.
Give reviewers a question more precise than “are these similar?” Ask whether a named retained document preserves the information that justifies removing the other document. Allow “uncertain” rather than forcing an arbitrary yes.
Include cases with different dates, quantities, negations, product editions and languages. Inspect boilerplate-heavy pages whose shared navigation or standard introduction overwhelms the distinctive content. Review short documents separately from long ones when their failure patterns differ.
Also examine likely duplicates below the candidate cutoff. Reviewing only accepted pairs measures how often the system is right when it proposes a removal; it cannot establish how many duplicates it misses. A separate labeled duplicate set helps estimate that second error within the set’s scope.
Resolve reviewer disagreements against the written removal rule. If reviewers cannot agree on replaceability, changing the threshold will not repair the definition.
Worked example: fewer removals can be the better result
Section titled “Worked example: fewer removals can be the better result”The following numbers are illustrative, not a benchmark or a Nodus customer result. Suppose a practitioner has a fixed review set of 200 candidate pairs. Reviewers label 120 pairs safely redundant and 80 not safely redundant. Each candidate removal has a named retained counterpart.
Two candidate cutoffs produce these outcomes on that same review set:
| Illustrative outcome | Stricter cutoff | Looser cutoff |
|---|---|---|
| Proposed removals | 80 | 120 |
| Safely redundant removals | 78 | 102 |
| Unsafe removals | 2 | 18 |
| Labeled duplicate pairs missed | 42 | 18 |
| Safe-removal precision | 97.5% | 85% |
| Recall within this labeled set | 65% | 85% |
Safe-removal precision is safely redundant removals divided by all proposed removals. Recall here is safely redundant removals divided by the 120 known redundant pairs in this review set.
The looser cutoff finds 24 additional redundant pairs but also proposes 16 additional unsafe removals. Whether that exchange is acceptable depends on what disappears. Losing a unique exception is not equivalent to retaining another copy of a generic paragraph.
Do not present these percentages as collection-wide estimates if the review set deliberately overrepresents difficult cases. Use the random audit of proposed removals to estimate their unsafe-removal rate, and report the sample size and uncertainty. Neither sample proves that every remaining document is safe.
Audit the final retained collection, not only the pairs
Section titled “Audit the final retained collection, not only the pairs”A pairwise decision is incomplete until you know what actually remains. A document marked as a replacement is no help if it is also removed. Require every proposed deletion to point to a retained substitute that still passes the replaceability rule.
Review groups containing several versions together. Shared topic alone can hide a chain of differences: one document preserves an old eligibility condition, another a new exception, and a third a translated explanation. A tidy-looking group does not prove one member preserves everything worth keeping.
Compare the collection before and after deduplication by source, language, time period and task-relevant category. An unchanged overall document count ratio can conceal the disappearance of a small but important slice. Review retained examples from those slices, not just totals.
For search, test questions requiring information found in the proposed removals. For training, compare downstream performance using the same held-out evaluation and training budget. Removing text is not itself evidence of better answers or lower total cost at equal quality.
Choose an operating rule before scaling compute
Section titled “Choose an operating rule before scaling compute”Separate three actions: automatic removal for well-supported cases, manual review for ambiguous cases, and retention where evidence is insufficient. There is no requirement to force the whole collection through one irreversible cutoff.
Record the embedding model, document treatment, collection version, removal rule and review results with the decision. Recheck when any of these changes. A score from one setup is not a universal unit of replaceability.
Keep the original collection recoverable and preserve the removal decisions for audit. Before a larger run, decide which observed losses would stop the process. Prefer an explicit accepted tradeoff over picking the cutoff that produces the largest percentage reduction.
Nodus Jobs can run your existing batch command, let you watch its progress and retrieve declared outputs. That is the execution step; your review determines the acceptable data loss. If you want to discuss running your document-preparation process on Nodus, book a call.
Is a cosine similarity of 0.9 enough to call two documents duplicates?
Section titled “Is a cosine similarity of 0.9 enough to call two documents duplicates?”Not by itself. A score needs a specific embedding model, document treatment and removal rule. Validate candidate cutoffs on your own collection. Two documents with a high score can still disagree on the only fact that matters.
Should you optimize precision or recall for deduplication?
Section titled “Should you optimize precision or recall for deduplication?”Start from the cost of each error. Unsafe removal destroys useful coverage; a missed duplicate leaves redundancy. For destructive automatic removal, demand stronger evidence of replaceability and send uncertain cases to review. The acceptable balance depends on the task.
Can a pairwise audit prove the final collection is safe?
Section titled “Can a pairwise audit prove the final collection is safe?”No. Also check that substitutes actually remain, important slices survive and downstream tasks still succeed. Pairwise precision is useful evidence, but it is not a guarantee about the final retained collection.
Do you need to remove every semantic duplicate?
Section titled “Do you need to remove every semantic duplicate?”No. A conservative cleanup can be worthwhile without maximum reduction. Keep examples whose removal lacks a defensible substitute, and measure downstream quality rather than treating a smaller dataset as the goal.
More from the blog
pricing7 min read
What is the best budget GPU for AI, and when should you rent?
Budget AI GPUs ranked by list price per GB of VRAM, what 12 and 16 GB fit, and the hours-per-month rule for when per-hour jobs on Nodus beat buying.
Read the posth1008 min read
What are the best Denvr Dataworks alternatives for training?
Denvr Dataworks alternatives for H100 training: Nodus, Together AI, Runpod, Lambda and CoreWeave priced on the same 72-hour, 8-GPU job.
Read the post