# Batch processing

> Collecting data for a while and then processing it all in one big run.

- Level: Intermediate
- Topics: Data, Software
- URL: https://tidbit.legios.com.ar/batch-processing/
- In Spanish: [Procesamiento batch](https://tidbit.legios.com.ar/es/batch-processing/)

**Batch processing** means handling data in large chunks on a schedule, instead of one item at a time as it arrives. A bank collects the day's transactions and settles them overnight. A company computes payroll once a month. A data team loads yesterday's sales into the data warehouse every morning at 2 AM so the dashboards are fresh by 9.

The idea is older than screens: early computers ran stacks of punched cards as one batch. It survives because it is simple and efficient. A single big run can use machines fully and then shut them down, it is easy to rerun if something fails, and the results are consistent because everything was computed from the same snapshot. Most ETL pipelines and tools like Apache Spark are typically run this way, often triggered by a cron job or an orchestrator.

The cost is delay: a batch report is only as fresh as the last run. When minutes or seconds matter, such as fraud detection or live tracking, teams switch to stream processing, which handles each event as it comes. Many companies use both: streams for what must be immediate, batches for heavy, complete calculations.

## An analogy

Like doing laundry once a week: you let clothes pile up and wash a full load, which is efficient, but if you need a specific shirt today you are out of luck.

## Keep in mind

- Processes accumulated data in scheduled runs, often nightly.
- Efficient and easy to rerun, but results are only as fresh as the last run.
- The opposite approach is streaming, which handles each event as it arrives.
- Typical uses: payroll, billing, nightly ETL loads, monthly reports.

## Go deeper

- [What is batch processing? (AWS)](https://aws.amazon.com/what-is/batch-processing/)
- [Batch processing on Wikipedia](https://en.wikipedia.org/wiki/Batch_processing)

## Related terms

- [Data warehouse](https://tidbit.legios.com.ar/data-warehouse/): A database built for analysis: years of history, from many sources, in one place.
- [Dashboard](https://tidbit.legios.com.ar/dashboard/): One screen of charts and numbers that shows how something is doing at a glance.
- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.
- [Apache Spark](https://tidbit.legios.com.ar/apache-spark/): An open source engine that splits work on huge datasets across many machines.
- [Cron job](https://tidbit.legios.com.ar/cron-job/): A command a computer runs on its own on a schedule, like every night at 3 a.m.
- [Data streaming](https://tidbit.legios.com.ar/data-streaming/): Processing data as a continuous flow of events, seconds after they happen.