Apache Spark
An open source engine that splits work on huge datasets across many machines.
Apache Spark is an open source engine for processing data that is too big, or too slow, for one computer. You describe what you want (read these files, filter, join, group, write the result) and Spark splits the data into partitions, sends them to many machines in a cluster, runs the steps in parallel and gathers the answer. You can write it in SQL (Spark SQL), Python (PySpark), Scala, Java or R.
It started at UC Berkeley around 2009 and became an Apache project, designed to improve on Hadoop MapReduce, which wrote intermediate results to disk between every step. Spark keeps as much as it can in memory and plans the whole job before running it, which makes multi-step work much faster. A central driver builds that plan; executors on the worker machines do the actual work, and if one fails, Spark recomputes only the lost pieces.
Today Spark is a workhorse of data engineering: nightly ETL jobs over a data lake, cleaning billions of events, preparing training data for machine learning, and processing streams in small batches. You rarely install it by hand; it runs as a managed service in Databricks, Amazon EMR, AWS Glue or Google Cloud Dataproc. For data that fits on one machine, simpler tools are often faster and cheaper.
Keep in mind
- Splits data into partitions and processes them in parallel on a cluster.
- Keeps intermediate data in memory, which made it much faster than MapReduce.
- Used from SQL, Python (PySpark), Scala, Java or R.
- Usually run as a managed service, such as Databricks, EMR or Glue.
