Databricks

Databricks is an American company founded in 2013 by the researchers at UC Berkeley who created Apache Spark, among them Ali Ghodsi, its CEO, Matei Zaharia and Ion Stoica. It sells a platform where companies process huge amounts of data, query it with SQL, and build and run machine learning and AI models, all in one place. It runs on top of the big clouds: AWS, Microsoft Azure and Google Cloud.

Its central idea is the lakehouse: keep all data as open-format files in cheap cloud storage, the way a data lake does, but add the reliability and fast queries of a data warehouse on top. Two open source projects it started make that work: Delta Lake, which adds transactions and version history to those files, and MLflow, which tracks machine learning experiments. Unity Catalog controls who can see which tables, files and models.

In practice a bank, a retailer or a hospital uses Databricks to pull data from dozens of systems, clean it, build reports and train models without copying everything between separate tools. Its main rival is Snowflake, which started from the warehouse side; over the years the two have converged, and both now pitch themselves as the place where companies build AI on their own data. Databricks expanded into that area by buying MosaicML in 2023.