# Databricks

> A cloud platform for processing, analyzing and training AI on a company's data.

- Level: Intermediate
- Topics: Tech companies, Data
- URL: https://tidbit.legios.com.ar/databricks/
- In Spanish: [Databricks](https://tidbit.legios.com.ar/es/databricks/)

**Databricks** is an American company founded in 2013 by the researchers at UC Berkeley who created Apache Spark, among them Ali Ghodsi, its CEO, Matei Zaharia and Ion Stoica. It sells a platform where companies process huge amounts of data, query it with SQL, and build and run machine learning and AI models, all in one place. It runs on top of the big clouds: AWS, Microsoft Azure and Google Cloud.

Its central idea is the **lakehouse**: keep all data as open-format files in cheap cloud storage, the way a data lake does, but add the reliability and fast queries of a data warehouse on top. Two open source projects it started make that work: **Delta Lake**, which adds transactions and version history to those files, and MLflow, which tracks machine learning experiments. **Unity Catalog** controls who can see which tables, files and models.

In practice a bank, a retailer or a hospital uses Databricks to pull data from dozens of systems, clean it, build reports and train models without copying everything between separate tools. Its main rival is Snowflake, which started from the warehouse side; over the years the two have converged, and both now pitch themselves as the place where companies build AI on their own data. Databricks expanded into that area by buying MosaicML in 2023.

## An analogy

A huge shared kitchen built next to the warehouse where all the raw ingredients arrive. Instead of trucking ingredients to different restaurants, every cook, the baker, the analyst and the AI chef, works right there from the same shelves.

## Keep in mind

- Founded in 2013 by the creators of Apache Spark.
- Popularized the lakehouse: data lake storage with data warehouse features.
- Runs on AWS, Azure and Google Cloud; customers pay for the compute they use.
- Its main competitor is Snowflake.

## Go deeper

- [Databricks (official site)](https://www.databricks.com/)
- [Delta Lake (open source project)](https://delta.io/)
- [Databricks on Wikipedia](https://en.wikipedia.org/wiki/Databricks)

## Related terms

- [Apache Spark](https://tidbit.legios.com.ar/apache-spark/): An open source engine that splits work on huge datasets across many machines.
- [SQL](https://tidbit.legios.com.ar/sql/): The language you use to ask a database questions.
- [Machine learning](https://tidbit.legios.com.ar/machine-learning/): Teaching computers by showing them examples instead of writing every rule.
- [Artificial intelligence](https://tidbit.legios.com.ar/artificial-intelligence/): Getting computers to do things that, until recently, only people could do.
- [Microsoft](https://tidbit.legios.com.ar/microsoft/): The company of Windows and Office, now just as much a cloud company with Azure.
- [Google Cloud](https://tidbit.legios.com.ar/google-cloud/): Google's cloud business: computing, storage, data and AI services rented by the hour.
- [Data lake](https://tidbit.legios.com.ar/data-lake/): Cheap storage where all your data is kept as raw files, to be sorted out later.
- [Cloud computing](https://tidbit.legios.com.ar/cloud-computing/): Renting computers over the internet instead of buying your own.