# Data cleaning

> Fixing errors, gaps and duplicates in data before anyone trusts it.

- Level: Basic
- Topics: Data
- URL: https://tidbit.legios.com.ar/data-cleaning/
- In Spanish: [Limpieza de datos](https://tidbit.legios.com.ar/es/data-cleaning/)

**Data cleaning** is the work of finding and fixing problems in a dataset so it can be used safely. Real data is messy: the same customer typed twice with slightly different names, dates written as `03/04/2025` in one system and `2025-04-03` in another, an age of 250, a blank email, prices with a comma in one country and a dot in another.

Typical steps are removing duplicates, standardizing formats (dates, currencies, country names), handling missing values (fill them, flag them or drop the row), catching values that cannot be true, and checking that references match, for example that every order belongs to a customer who exists. Each decision is a judgment call, so good teams write the rules down and run them as code instead of fixing cells by hand.

It is famously the least glamorous and most time-consuming part of working with data, and also one of the most important. A dashboard or a machine learning model is only as good as the data it is fed; "garbage in, garbage out" is the oldest saying in the field. In a pipeline, cleaning usually happens in the transform step of ETL, and the checks keep running every time new data arrives.

## An analogy

Washing and sorting vegetables before cooking: you throw out the bruised ones, remove the dirt and cut everything the same size. Skip it and even a great recipe tastes off.

## Keep in mind

- Fixes duplicates, inconsistent formats, missing values and impossible numbers.
- Every rule is a decision; write it down and automate it.
- Usually part of the transform step in a data pipeline.
- Garbage in, garbage out: reports and AI models inherit every error.

## Go deeper

- [Data cleansing on Wikipedia](https://en.wikipedia.org/wiki/Data_cleansing)
- [What is data cleansing? (AWS)](https://aws.amazon.com/what-is/data-cleansing/)
- [What is data cleaning? (IBM)](https://www.ibm.com/think/topics/data-cleaning)

## Related terms

- [Machine learning](https://tidbit.legios.com.ar/machine-learning/): Teaching computers by showing them examples instead of writing every rule.
- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.
- [Artificial intelligence](https://tidbit.legios.com.ar/artificial-intelligence/): Getting computers to do things that, until recently, only people could do.