# Data streaming

> Processing data as a continuous flow of events, seconds after they happen.

- Level: Advanced
- Topics: Data, Cloud
- URL: https://tidbit.legios.com.ar/data-streaming/
- In Spanish: [Streaming de datos](https://tidbit.legios.com.ar/es/data-streaming/)

**Data streaming** means handling data as a continuous flow of small events instead of in big periodic batches. Each card payment, click, GPS position or sensor reading is sent the moment it happens, and programs downstream react to it within seconds. A nightly ETL answers 'what happened yesterday?'; a stream answers 'what is happening right now?'.

At the center there is usually a **log** of events, such as Apache Kafka or Amazon Kinesis. Producers append events to the end, the log keeps them in order for days, and any number of consumers read at their own pace, each remembering where it left off. Because events are kept, a new consumer can start from the beginning, and one that crashes picks up where it stopped without losing anything.

Streams power things that cannot wait: a bank blocking a suspicious card payment before it is approved, a delivery app moving the courier on your map, a factory raising an alert when a machine overheats. The cost is complexity. Events arrive late or out of order, the same event can be delivered twice, and a question like 'sales in the last five minutes' needs **windows** over a flow that never ends. When an answer every hour is good enough, a batch pipeline is simpler and cheaper.

## An analogy

Reading the morning paper versus following a live radio broadcast. The paper (batch) is tidy but a day late; the radio (stream) tells you as it happens, and you can tune in at any moment.

## Keep in mind

- Events are processed one by one, seconds after they happen, instead of in nightly batches.
- A durable log such as Kafka or Kinesis keeps events in order and lets many consumers read them.
- Late, duplicated and out-of-order events are normal and must be handled.
- Use it when minutes matter; otherwise a batch ETL is simpler.

## Go deeper

- [What is streaming data? (AWS)](https://aws.amazon.com/what-is/streaming-data/)
- [Introduction to Apache Kafka](https://kafka.apache.org/intro)
- [Stream processing on Wikipedia](https://en.wikipedia.org/wiki/Stream_processing)

## Related terms

- [ETL](https://tidbit.legios.com.ar/etl/): Copy data out of its sources, clean it up, load it where it can be analyzed.