# Apache Spark

> An open source engine that splits work on huge datasets across many machines.

- Level: Advanced
- Topics: Data
- URL: https://tidbit.legios.com.ar/apache-spark/
- In Spanish: [Apache Spark](https://tidbit.legios.com.ar/es/apache-spark/)

**Apache Spark** is an open source engine for processing data that is too big, or too slow, for one computer. You describe what you want (read these files, filter, join, group, write the result) and Spark splits the data into **partitions**, sends them to many machines in a cluster, runs the steps in parallel and gathers the answer. You can write it in SQL (**Spark SQL**), Python (**PySpark**), Scala, Java or R.

It started at UC Berkeley around 2009 and became an Apache project, designed to improve on Hadoop MapReduce, which wrote intermediate results to disk between every step. Spark keeps as much as it can in memory and plans the whole job before running it, which makes multi-step work much faster. A central **driver** builds that plan; **executors** on the worker machines do the actual work, and if one fails, Spark recomputes only the lost pieces.

Today Spark is a workhorse of data engineering: nightly ETL jobs over a data lake, cleaning billions of events, preparing training data for machine learning, and processing streams in small batches. You rarely install it by hand; it runs as a managed service in Databricks, Amazon EMR, AWS Glue or Google Cloud Dataproc. For data that fits on one machine, simpler tools are often faster and cheaper.

## An analogy

Counting all the votes of a country. One person would take months; instead, each polling station counts its own box at the same time and phones the total to a central office that adds them up. Spark is the system that organizes the stations and the phone calls.

## Keep in mind

- Splits data into partitions and processes them in parallel on a cluster.
- Keeps intermediate data in memory, which made it much faster than MapReduce.
- Used from SQL, Python (PySpark), Scala, Java or R.
- Usually run as a managed service, such as Databricks, EMR or Glue.

## Go deeper

- [Apache Spark (official site)](https://spark.apache.org/)
- [What is Apache Spark? (AWS)](https://aws.amazon.com/what-is/apache-spark/)
- [Apache Spark on Wikipedia](https://en.wikipedia.org/wiki/Apache_Spark)

## Related terms

- [Open source](https://tidbit.legios.com.ar/open-source/): Software whose code anyone can read, use, change and share.
- [SQL](https://tidbit.legios.com.ar/sql/): The language you use to ask a database questions.
- [Python](https://tidbit.legios.com.ar/python/): A popular programming language known for being easy to read and quick to write.
- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.
- [Data lake](https://tidbit.legios.com.ar/data-lake/): Cheap storage where all your data is kept as raw files, to be sorted out later.
- [Training and inference](https://tidbit.legios.com.ar/training-and-inference/): Training builds the model once; inference is using it, millions of times a day.
- [Machine learning](https://tidbit.legios.com.ar/machine-learning/): Teaching computers by showing them examples instead of writing every rule.
- [Amazon](https://tidbit.legios.com.ar/amazon/): An online store with its own delivery network, and the company that runs AWS.