Getting started with Smart Dedupe
Smart Dedupe finds exact and semantically similar blocks across notes so you can decide whether repeated material should remain separate.
A 100% similarity or Exact hash match label does not mean that one note should be removed. Two identical passages can serve different purposes.
Before you start
Confirm that:
- Smart Dedupe Pro is installed and enabled
- Smart Environment has finished preparing source blocks
- one meaningful note with repeated or potentially repeated material is open
Use neutral test material for the first scan. Do not begin with customer, credential, legal, or other high-risk notes.
1. Activate the persistent Duplicate Detector
Open the Command Palette and run:
Smart Dedupe Pro: Open Duplicate Detector
The Duplicate Detector keeps the scan settings and results in one view. Start with Current note so the first result set is easy to understand.
The Detector opens with the scan boundary visible before anything runs. This frame proves the current controls, not a completed scan or a useful match.
Choose a scan mode:
| Mode | Use it when... | Start the scan | Learn more |
|---|---|---|---|
| Current note | You want to compare one open note with the rest of the vault. | Choose Current note, then Run note scan. | Scan the current note |
| Full vault | You are ready to review matches across the whole vault. | Choose Full vault, review the settings, then Run vault scan. | Scan the full vault |
See Open the persistent Duplicate Detector for more detail.
2. Configure a current-note scan
Choose Current note, then review:
- Similarity threshold
- Max results
- Minimum length
- Skip exact matches
- Exclude frontmatter matches
For the first scan, keep exact matches visible. Start with the visible threshold of 0.85 and a bounded result count, then adjust from the quality of the candidates. These are review choices, not quality guarantees.
See Configure the scan before changing threshold, result-count, exact-match, frontmatter, or minimum-length controls.
3. Run the note scan
Choose Run note scan and wait for the Detector to settle.
A short scan may finish before you can read the progress state. That is normal; continue with the completed results.
4. Review one candidate pair
Check:
- the threshold and result count
- whether the candidate is exact or semantic
- the Source and Duplicate identities
- the surrounding text and line range
Use Open when the visible excerpt is not enough. Use Copy only to compare or preserve text deliberately; neither action merges the notes.
See Review matches side by side and Copy or open a result for more detail.
5. Make a human decision
After reviewing both sources, choose one of these outcomes:
- keep both passages because their purposes differ
- rewrite one passage to clarify the distinction
- consolidate useful material manually
- archive a genuinely redundant note through your normal Obsidian workflow
- ignore the candidate
Smart Dedupe does not execute those decisions for you.
You know it worked when
- one real candidate pair was visible in the persistent Detector
- both source identities and excerpts were reviewed
- the match label guided inspection without replacing judgment
- any subsequent note change was deliberate and reversible
Troubleshooting
| Symptom | First adjustment |
|---|---|
| No candidates | Lower the threshold slightly or confirm block readiness. |
| Too many weak candidates | Raise the threshold, increase minimum length, or stay with Current note. |
| Boilerplate dominates | Enable Exclude frontmatter matches and raise minimum length. |
| Exact repeats hide paraphrases | Review exact matches first, then try Skip exact matches deliberately. |
| Scan finishes before progress is readable | Review the settled result; active-scan controls may disappear too quickly to inspect. |
| You expected automatic cleanup | Review and change notes manually; Dedupe is not an automatic merge/delete tool. |