dbt
A tool for writing data transformations as SQL files, with tests and docs, like software.
dbt (originally "data build tool") is an open source tool for the transform step of a data pipeline, the T in ELT. The data has already been loaded into a data warehouse such as Snowflake, BigQuery, Redshift or Databricks; dbt helps you turn those raw tables into clean, trusted ones. Each transformation is a model: a plain .sql file with a SELECT statement, saved in a git repository.
Models can refer to each other with {{ ref('orders') }}, so dbt knows the order in which to build them and draws the whole chain as a graph. Running dbt run sends each model's SQL to the warehouse, which does the heavy work; dbt itself moves no data. dbt test checks rules like "this column is never empty" or "every order has a customer", and dbt docs generates a website describing every table and column, with lineage.
Its impact was cultural as much as technical: it brought software habits like version control, code review, testing and reusable pieces to analysts who already knew SQL. That gave rise to the role of analytics engineer. dbt Labs, the company behind it, also sells a hosted platform, but the core tool is open source and runs from the command line.
Keep in mind
- Each model is a SELECT in a .sql file; dbt builds them inside the warehouse.
- ref() links models into a dependency graph with automatic build order.
- Built-in tests and generated documentation with lineage.
- Brought git, code review and testing to SQL analytics work.
