# Data orchestration

> Running data jobs in the right order, at the right time, and handling it when they fail.

- Level: Advanced
- Topics: Data
- URL: https://tidbit.legios.com.ar/data-orchestration/
- In Spanish: [Orquestación de datos](https://tidbit.legios.com.ar/es/data-orchestration/)

A real data platform is dozens or hundreds of jobs: pull orders from the shop database, pull payments from the bank, clean both, join them, rebuild the data mart, refresh the dashboards. Each step depends on others finishing first. **Data orchestration** is the layer that knows those dependencies, starts each job when its inputs are ready, and keeps track of what ran, what failed and what is late.

Most orchestrators describe a pipeline as a DAG, a directed acyclic graph: boxes for tasks and arrows for "this must finish before that starts", with no loops. You write the DAG as code, the orchestrator schedules it (every night at 2 a.m., or whenever a new file lands), runs independent tasks in parallel, retries the ones that fail, and alerts someone when a retry is not enough. If the payments extract breaks, nothing downstream runs on half the data.

**Apache Airflow** is the best-known tool. It started at Airbnb in 2014, became an Apache project, and pipelines in it are written in Python. Newer tools like Dagster and Prefect put more weight on the data each task produces rather than only the tasks, and clouds offer managed versions such as Amazon MWAA. Orchestration does not transform the data itself; it calls the tools that do, such as dbt, Spark or plain SQL, and makes sure they run in order.

## An analogy

Like the conductor of an orchestra. The conductor plays no instrument, but decides who comes in when, keeps everyone in time, and notices immediately when a section is missing.

## Keep in mind

- Runs data jobs in dependency order, on a schedule or on events.
- Pipelines are usually DAGs: tasks plus "runs after" arrows, no loops.
- Handles retries, alerts, backfills and a history of every run.
- Apache Airflow is the classic tool; Dagster and Prefect are newer options.

## Go deeper

- [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html)
- [Apache Airflow on Wikipedia](https://en.wikipedia.org/wiki/Apache_Airflow)
- [What is data orchestration? (IBM)](https://www.ibm.com/think/topics/data-orchestration)

## Related terms

- [Database](https://tidbit.legios.com.ar/database/): An organized store of data that programs can search and update safely.
- [Data mart](https://tidbit.legios.com.ar/data-mart/): A small, focused slice of the company's data, shaped for one team.
- [Dashboard](https://tidbit.legios.com.ar/dashboard/): One screen of charts and numbers that shows how something is doing at a glance.
- [Airbnb](https://tidbit.legios.com.ar/airbnb/): A website and app where people rent out their homes, or a room, to travelers.
- [Python](https://tidbit.legios.com.ar/python/): A popular programming language known for being easy to read and quick to write.
- [Amazon](https://tidbit.legios.com.ar/amazon/): An online store with its own delivery network, and the company that runs AWS.
- [dbt](https://tidbit.legios.com.ar/dbt/): A tool for writing data transformations as SQL files, with tests and docs, like software.
- [Apache Spark](https://tidbit.legios.com.ar/apache-spark/): An open source engine that splits work on huge datasets across many machines.