Deduplication
Finding the copies of the same thing and keeping just one.
Deduplication is spotting records or pieces of data that are really the same thing and collapsing them into one. It shows up in two quite different places: in business data, where the same customer was typed in several times, and in storage, where the same file or block of bytes is saved over and over.
In business data the hard part is that duplicates rarely match exactly. "María González", "Maria Gonzalez" and "M. González" with the same phone number are probably one person. Deduplication tools normalize text (accents, capitals, spaces), compare fields with fuzzy matching, and score how likely two records are the same. Above a threshold they merge; in a gray zone a human decides. This is part of data cleaning and matters a lot before counting customers or sending a mailing.
In storage and backups, deduplication works on raw bytes. The system cuts files into chunks, computes a hash of each one, and stores a chunk only the first time it sees that hash; later copies just point to it. That is how backup services can keep daily snapshots of thousands of laptops without storing every copy of the same operating system files. In data pipelines there is a third flavor: making sure an event delivered twice is not counted twice.
Keep in mind
- Collapses copies of the same thing into one.
- In business data, duplicates are fuzzy and need careful matching.
- In storage, identical chunks are detected by their hash and stored once.
- Wrong merges can be worse than duplicates, so thresholds matter.
