What Is Stream Processing?

Stream processing, is a data processing technique that collects, transforms, and analyzes streams of data in real time. Stream processing systems are designed to handle large amounts of high-velocity events (e.g., IoT sensor readings, financial market data, server usage metrics) and provide real-time insights.

Stream processing visualization
Stream processing processes transaction in real time.

Stream vs. batch processing​

Traditional data pipelines use batch processing where data is analyzed in batches. This incurs latency penalty due to the lag between the event taking place and the data being processed.

Batch processing visualization
Batch processing collects events and processes them in batch.

Contrary to batch processing, stream processing analyzes data on the fly. This means that analysis occurs as soon as data is ingested, and therefore stream processing systems enable use cases where real-time decision-making is useful. Examples include anomaly detection, fraud prevention, and predictive analytics.

Popular stream processing systems include:

Types of stream processing​

There are two major types of stream processing:

  • Stateless: Stateless systems process elements individually without any batching or historical context. IoT monitoring systems may opt for a stateless stream processing model for real-time detection of faults or failures (e.g., the temperature is too high). The data may be analyzed later for forecasting purposes further downstream.
  • Stateful: Stateful stream processing models, on the other hand, process elements in groups, and apply historical information when making decisions or operating on the data. These types of models are useful when context or temporal relationships are important. For example, an e-commerce website may rank products based on hourly popularity, or the recent history of returning customers. Or an IoT system might want to report not only the current temperature for a sensor, but the rate change and average for the last 5 minutes.

In general, stateful stream processing systems are more complex as they must handle state updates and failures gracefully.

While stateful stream processing systems can provide more insight, they are typically insufficient for advanced analytics and are used in conjunction with databases that store the output of stateful processing for further exploration or to power graphical dashbo

Use cases​

As the need for real-time analytics grows, stream processing is growing more popular across many industries:

  • Financial markets: credit fraud monitoring, real-time trading engines
  • IoT: supply chain metrics, smart grids, smart homes, predictive maintenance
  • Telecommunications: networking monitoring, traffic analysis Application metrics: personalized recommendations, inventory management

Additional Resources​