Data science teams are not waiting for the nightly job anymore. Models are being pushed into production paths where a stale feature is a wrong decision, and a wrong decision costs money, trust, or in some sectors, regulatory standing. The question of whether to build real-time data science pipelines on event streaming or keep running batch is no longer purely academic: it shapes your infrastructure costs, your operational complexity, and how much latency you can actually promise to the business.
The answer is not “streaming always wins.” It is more nuanced than that, and getting the architecture wrong in either direction is expensive. Here is how to think through it honestly.
What batch pipelines actually cost you
Batch processing has a deserved reputation for simplicity. You schedule a job, it runs, you get results. Debugging is straightforward because the data is static at the time of execution, and compute can be scheduled during off-peak windows. For model training on historical data, batch is still the right default.
The problem emerges when the business asks the model to respond to something happening right now. A fraud detection system that scores yesterday’s transactions in bulk has already let the fraudulent ones clear. A recommendation engine that refreshes features every four hours is serving suggestions based on who the user was four hours ago. The gap between when an event happens and when your model knows about it is called feature latency, and in real-time applications it is the primary driver of model degradation in production.
Batch architectures also carry hidden costs that tend to be underestimated: large compute spikes at job time, storage for intermediate outputs, and the engineering overhead of backfill logic when upstream data arrives late. When those costs are accounted for honestly, the economics of streaming look considerably more competitive than the conventional wisdom suggests.
Where event streaming architecture earns its complexity budget
Event streaming becomes the right backbone when the model needs to act on the current state of the world, not a recent snapshot of it. The canonical use cases are well-established: fraud detection, dynamic pricing, real-time personalisation, and operational monitoring. In each of these, the value of a prediction degrades sharply with latency. A fraud signal delivered in 150 milliseconds can block a transaction; the same signal delivered in 15 minutes is a post-mortem.
Feature engineering is where event streaming earns its complexity budget: windowed aggregations, event enrichment with live entity data, and derived metrics computed continuously are the transformations that turn raw events into ML-ready features at sub-second freshness.
Apache Kafka has become the de facto standard for this layer because it provides durable, replayable event storage with the throughput to handle high-cardinality event streams. Replayability matters specifically for ML: you need to reconstruct the exact feature state the model saw at prediction time for debugging, retraining, and audit. A stream you cannot replay is a stream you cannot trust in production.
Processing engines like Apache Flink and Spark Structured Streaming sit downstream of the broker, handling stateful transformations: computing a user’s transaction count over the last hour, joining a clickstream event with a product catalog, or deriving a rolling percentile. Spark’s micro-batch model introduces latency in the 100-millisecond-plus range, which is sufficient for many use cases. Flink’s true event-time processing pushes that lower when strict per-event execution is genuinely required. Databricks has also introduced a real-time execution mode for Spark that targets consistent sub-second latencies on the same API surface, which matters for teams that have already standardised on Spark for batch training.
The dual-store problem in real-time feature engineering
One of the less-discussed architectural requirements of Kafka for machine learning is the dual-store pattern. Training needs historical feature access, typically as Parquet or Delta Lake files queried in batch. Serving needs the current feature value for a given entity, retrieved in single-digit milliseconds. These are fundamentally different access patterns, and a single store cannot serve both well.
The standard solution is a feature store that maintains an offline store for training and an online store, typically Redis, Cassandra, or ScyllaDB, for serving. Platforms like Feast, Tecton, and Hopsworks orchestrate the dual-write pattern: the stream processor writes to both stores simultaneously, keeping them consistent. Wix runs this pattern in production, using Kafka and Flink together to power real-time personalisation at scale. The architecture works, but it adds operational surface area: you now have a broker, a stream processor, an online store, and an offline store, each with its own failure modes.
When streaming is the wrong answer
Stream processing vs. batch processing is not a binary where streaming is always superior. Continuous systems require continuous compute, which means continuous cost. They require stronger observability because failures are not isolated to a job window: a broken enrichment join silently corrupts features for every event until someone notices. State management becomes a discipline in itself.
For dashboards, compliance reporting, log aggregation, and model training, micro-batching at sub-minute intervals is almost always sufficient and considerably simpler to operate. The rule worth applying: if the business cannot describe a specific decision that degrades meaningfully when the feature is 60 seconds old rather than 60 milliseconds old, you probably do not need a full streaming pipeline. The complexity budget should be spent where latency genuinely matters.
Network architecture and why it matters for real-time pipelines
Real-time feature engineering is latency-sensitive end to end, not just inside the processing engine. The round-trip between your event source, your broker, your stream processor, your online feature store, and your model serving endpoint adds up, and that round-trip is largely determined by where those components sit relative to each other and to the network.
We operate 44 points of presence across Europe and the Middle East, with direct peering to AWS, Azure, GCP, Oracle, and Cloudflare at all sites. For teams running hybrid architectures where training workloads stay on a hyperscaler but feature computation and model serving run on sovereign EU infrastructure, that peering eliminates the transit hops that would otherwise add tens of milliseconds to every feature lookup. For pipelines where end-to-end latency is the product requirement, those milliseconds are not rounding errors.
Running event streaming on EU-sovereign infrastructure also keeps the data residency question clean. Events flowing through a Kafka-compatible broker hosted inside Europe, processed by stream processors on European compute, writing features to an online store that never leaves European jurisdiction: that is a materially simpler compliance posture than one where the event stream transits through a US-headquartered cloud provider’s European region.
Making the architecture decision
The practical framework is straightforward. Start with the latency requirement: if your model’s value degrades meaningfully when features are more than a few seconds old, streaming is justified. If a minute is fine, micro-batch. If an hour is fine, batch. Then map the complexity cost against the team’s operational maturity: a streaming pipeline that is poorly observed is worse than a batch pipeline that runs reliably.
Where streaming is the right call, invest in the feature store architecture early. The dual-store pattern is not optional for production ML; it is the mechanism that makes streaming features usable for both training and serving without rebuilding the pipeline twice. Choose your processing engine based on the latency target and your existing tooling, not on what the architecture diagram looks like.
Event streaming is a powerful backbone for real-time data science pipelines when the latency requirement is real. The mistake is treating it as an upgrade to batch rather than as a different operational model with different costs and different failure modes. Get that distinction right and the architecture decision becomes considerably clearer.
