# Data engineering

> The work of building the pipes that move data to where people can use it.

- Level: Intermediate
- Topics: Data
- URL: https://tidbit.legios.com.ar/data-engineering/
- In Spanish: [Ingeniería de datos](https://tidbit.legios.com.ar/es/data-engineering/)

**Data engineering** is the discipline of collecting data from the places it is born, cleaning and reshaping it, and delivering it somewhere reliable where others can use it. A company's data starts scattered: the app's database, the payments provider, the CRM, spreadsheets, event logs. A **data engineer** builds and runs the pipelines that bring all of that together, usually into a data warehouse or a data lake.

Day to day, that means writing ETL or ELT jobs in SQL and Python, scheduling them with tools like Airflow, modeling tables with dbt, and handling large volumes with engines like Apache Spark. It also means the unglamorous part: making jobs rerun safely when they fail, noticing when a source changes its format, keeping costs under control and making sure yesterday's numbers arrive before the 9 a.m. meeting.

The role sits upstream of the analysts and data scientists. They ask the questions; data engineers make sure the answers rest on data that is complete, on time and consistent. As companies feed more of their data into AI systems, this plumbing has become more visible, because a model is only as good as the data that reaches it.

## An analogy

The water utility of a city. Nobody thinks about the pipes, pumps and treatment plants until the tap runs brown; data engineers build and maintain that network so clean data comes out wherever someone opens a tap.

## Keep in mind

- Moves data from many sources into a warehouse or lake, cleaned and consistent.
- Core tools: SQL, Python, orchestrators like Airflow, dbt, Spark.
- Much of the job is reliability: failures, late data, changed sources.
- Works upstream of analysts, data scientists and machine learning teams.

## Go deeper

- [Data engineering on Wikipedia](https://en.wikipedia.org/wiki/Data_engineering)
- [Apache Airflow documentation](https://airflow.apache.org/docs/)

## Related terms

- [Database](https://tidbit.legios.com.ar/database/): An organized store of data that programs can search and update safely.
- [Spreadsheet vs database](https://tidbit.legios.com.ar/spreadsheet-vs-database/): Why a company that runs on Excel eventually needs a real database.
- [Data warehouse](https://tidbit.legios.com.ar/data-warehouse/): A database built for analysis: years of history, from many sources, in one place.
- [Data lake](https://tidbit.legios.com.ar/data-lake/): Cheap storage where all your data is kept as raw files, to be sorted out later.
- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.
- [SQL](https://tidbit.legios.com.ar/sql/): The language you use to ask a database questions.
- [Python](https://tidbit.legios.com.ar/python/): A popular programming language known for being easy to read and quick to write.
- [dbt](https://tidbit.legios.com.ar/dbt/): A tool for writing data transformations as SQL files, with tests and docs, like software.