Data streaming
Processing data as a continuous flow of events, seconds after they happen.
Data streaming means handling data as a continuous flow of small events instead of in big periodic batches. Each card payment, click, GPS position or sensor reading is sent the moment it happens, and programs downstream react to it within seconds. A nightly ETL answers 'what happened yesterday?'; a stream answers 'what is happening right now?'.
At the center there is usually a log of events, such as Apache Kafka or Amazon Kinesis. Producers append events to the end, the log keeps them in order for days, and any number of consumers read at their own pace, each remembering where it left off. Because events are kept, a new consumer can start from the beginning, and one that crashes picks up where it stopped without losing anything.
Streams power things that cannot wait: a bank blocking a suspicious card payment before it is approved, a delivery app moving the courier on your map, a factory raising an alert when a machine overheats. The cost is complexity. Events arrive late or out of order, the same event can be delivered twice, and a question like 'sales in the last five minutes' needs windows over a flow that never ends. When an answer every hour is good enough, a batch pipeline is simpler and cheaper.
Keep in mind
- Events are processed one by one, seconds after they happen, instead of in nightly batches.
- A durable log such as Kafka or Kinesis keeps events in order and lets many consumers read them.
- Late, duplicated and out-of-order events are normal and must be handled.
- Use it when minutes matter; otherwise a batch ETL is simpler.
