# Deduplication

> Finding the copies of the same thing and keeping just one.

- Level: Intermediate
- Topics: Data
- URL: https://tidbit.legios.com.ar/deduplication/
- In Spanish: [Deduplicación](https://tidbit.legios.com.ar/es/deduplication/)

**Deduplication** is spotting records or pieces of data that are really the same thing and collapsing them into one. It shows up in two quite different places: in business data, where the same customer was typed in several times, and in storage, where the same file or block of bytes is saved over and over.

In business data the hard part is that duplicates rarely match exactly. "María González", "Maria Gonzalez" and "M. González" with the same phone number are probably one person. Deduplication tools normalize text (accents, capitals, spaces), compare fields with fuzzy matching, and score how likely two records are the same. Above a threshold they merge; in a gray zone a human decides. This is part of data cleaning and matters a lot before counting customers or sending a mailing.

In storage and backups, deduplication works on raw bytes. The system cuts files into chunks, computes a hash of each one, and stores a chunk only the first time it sees that hash; later copies just point to it. That is how backup services can keep daily snapshots of thousands of laptops without storing every copy of the same operating system files. In data pipelines there is a third flavor: making sure an event delivered twice is not counted twice.

## An analogy

Like cleaning up your phone's contacts and finding "Mom", "Mamá" and "Mom cell" with the same number. You merge them into one card and keep the best details from each.

## Keep in mind

- Collapses copies of the same thing into one.
- In business data, duplicates are fuzzy and need careful matching.
- In storage, identical chunks are detected by their hash and stored once.
- Wrong merges can be worse than duplicates, so thresholds matter.

## Go deeper

- [Data deduplication on Wikipedia](https://en.wikipedia.org/wiki/Data_deduplication)
- [Record linkage on Wikipedia](https://en.wikipedia.org/wiki/Record_linkage)

## Related terms

- [Bit and byte](https://tidbit.legios.com.ar/bit-and-byte/): The smallest unit of digital information, and the group of eight that computers count in.
- [Data cleaning](https://tidbit.legios.com.ar/data-cleaning/): Fixing errors, gaps and duplicates in data before anyone trusts it.
- [Backup](https://tidbit.legios.com.ar/backup/): A separate copy of your data, kept so you can get it back when the original is lost.
- [Hashing](https://tidbit.legios.com.ar/hashing/): A one-way fingerprint of data: easy to compute, not meant to be reversed.
- [Laptop](https://tidbit.legios.com.ar/laptop/): A whole computer folded into one portable case that runs on a battery.
- [Operating system](https://tidbit.legios.com.ar/operating-system/): The base software that runs a device and lets every app share its hardware.
- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.