Kafka bare metal deployment: architecture patterns that cut latency and cost

Apache Kafka 4.0, released in March 2025, ships without ZooKeeper entirely. KRaft is now the only way to run a cluster, and teams still on ZooKeeper mode must migrate through Kafka 3.9 before they can upgrade. That is a meaningful operational shift, and it lands at the same moment many platform teams are asking a harder question: whether running Kafka on shared cloud tiers still makes sense at the throughput and latency requirements they actually have.

The honest answer is: it depends on your P99 requirements, your data volume, and how much of your bill you want to hand to a managed service. This post works through the architecture decisions that matter most when moving Kafka-compatible workloads onto dedicated hardware, where the physics of the problem become your advantage rather than someone else’s margin.

We run NATS JetStream as our own production event-streaming backbone across four European datacenters. That means we have direct operational experience with the tuning decisions below, not just the theory.

Network latency follows the speed of light in fiber, roughly 5 microseconds per kilometer, plus routing overhead. Same-rack latency sits between 0.1 and 0.5 ms one-way; cross-AZ within a region adds another 0.5 to 2 ms. Cross-region hops across a continent run 15 to 40 ms. These are physical floors, not configuration problems. When a managed Kafka service routes your replication traffic across availability zones, you are paying that cross-AZ tax on every write with acks=all. On bare metal, you choose the topology: brokers in the same rack, on the same switch, with replication traffic that never leaves your cage.

What actually bounds your partition throughput

A Kafka topic partition is a log file written sequentially to disk on a broker. Its throughput ceiling is set by three things: the broker’s NIC, the disk’s write IOPS, and the replication network traffic to follower replicas. On a shared cloud instance, you are competing for all three with other tenants. On dedicated hardware with NVMe storage, you own those resources entirely.

Kafka clusters on three-to-five-broker configurations with NVMe storage routinely sustain single-digit GB/s. The Apache Kafka project’s own benchmark suite shows Kafka 4.x holding above 100 MB/s per broker on commodity hardware. Redpanda Labs published figures in 2024 showing 1.4 million messages per second across a five-node cluster on commodity hardware. These numbers are achievable on bare metal; they are much harder to reproduce consistently on burstable cloud instances where disk and network performance fluctuates.

The distinction between “usually fast” and “always fast” matters most at the tail. Kafka holds tight latencies up to around the 94th percentile before disk persistence costs surface. NVMe drives eliminate the mechanical seek penalty and give you predictable IOPS under sustained write load, which is what keeps P99 from diverging from median in production.

Replication, durability, and the latency trade-off

The replication factor and acknowledgement mode together set your durability-latency contract. With acks=all, latency equals your base write time plus the round-trip to the slowest in-sync replica. With acks=1, latency equals your base write time plus the local disk flush only. Neither choice is universally correct; the right answer depends on what you are streaming and what losing a message costs.

Durability is not free, and latency is not negotiable. The architecture question is which one you are paying for, and whether you control the exchange rate.

On bare metal, you control where your replicas land. Placing all three replicas of a critical topic on brokers in the same rack, connected by a dedicated replication VLAN, keeps your ISR round-trip under 0.5 ms. Placing them across cloud availability zones for “resilience” adds 1 to 2 ms to every acknowledged write, and you pay for that network traffic by the GB.

Partition count, hot partitions, and parallelism limits

A higher partition count increases parallelism and throughput, but it also increases broker workload: more open file handles, more replication connections, more metadata that the KRaft controller must track. The practical rule is to size partitions to your target consumer parallelism, not to a theoretical maximum. Kafka’s enforced constraint is that a partition can be consumed by only one consumer within a group, so your partition count sets the ceiling on how many consumers can work in parallel.

Hot partitions are a keying problem, not a hardware problem, but bare metal makes them cheaper to diagnose. When certain partitions receive far more traffic than others, the brokers holding those partitions become resource-constrained while others sit idle. Poor keying strategy is the usual cause: using a low-cardinality field as the message key, or routing all null-key messages to a single partition. On a dedicated cluster, per-broker CPU and disk utilisation is visible without cloud-provider abstractions in the way, so imbalances surface faster.

Tiered storage: separating retention cost from throughput cost

Kafka’s tiered storage, introduced through KIP-405, lets you offload older log segments to object storage while keeping recent data on local NVMe for low-latency tail reads. This matters economically at scale: retaining 30 days of high-volume topic data on NVMe is expensive; retaining it on object storage is not. The architecture keeps your hot path fast and your retention cost manageable.

On a self-hosted bare metal deployment, you choose the object store. That means your tiered storage target can be a sovereign European object store under your own contractual control, not a US-headquartered cloud provider’s bucket. For financial services, healthcare, and defence workloads, that distinction is not academic.

Kafka 4.0 operational implications for bare metal teams

The KRaft-only architecture in Kafka 4.0 removes the ZooKeeper ensemble from your operational surface entirely. That is a genuine simplification: fewer processes to monitor, fewer failure modes to handle, and a single metadata system to reason about. The KRaft controller manages cluster metadata through a Raft consensus log, and the migration path from ZooKeeper mode runs through Kafka 3.9 as the recommended bridge release.

Java 17 is now required for brokers, Connect, and tools in Kafka 4.0, with Java 11 required for clients and Streams. This is a breaking change for teams running older JVM versions on bare metal, and it should be part of your upgrade planning before you touch the Kafka binaries.

KIP-848, the new consumer group protocol now generally available in 4.0, significantly improves rebalance performance. Rebalances on large consumer groups have historically been a source of latency spikes and availability gaps; the new protocol reduces the blast radius of a single consumer joining or leaving the group. On bare metal, where you control the network topology and can colocate consumers with brokers, the combination of KIP-848 and low-latency replication cuts the tail impact of rebalances substantially.

The self-hosted cost structure

Managed Kafka pricing is typically structured around partition-hours, data transfer, and storage, with the provider’s margin embedded in each. At low volumes, that model is fine. At high volumes, the arithmetic reverses: you are paying per-message overhead on traffic that dedicated hardware could handle for the cost of the hardware and the engineering time to run it.

The crossover point varies by workload, but teams running sustained throughput above a few hundred MB/s consistently find that self-hosted Kafka on bare metal is cheaper over a three-year horizon, even accounting for operational overhead. The infrastructure cost is fixed; the managed service cost scales with your data volume. At DIVERSITY, our bare metal fleet runs under Tier III availability guarantees, carrier-neutral interconnects, ISO 27001 certification, and a contractual 99.9% uptime SLA. Those are not marketing lines; they are the terms your procurement and legal teams can review and hold us to.

The architecture patterns above are not hypothetical. They are the decisions you make when you own the hardware, the network, and the operational responsibility. The managed cloud tier abstracts those decisions away from you, and charges you for the privilege. Whether that trade is worth it depends on your scale, your latency requirements, and how much operational control you need over your event streaming infrastructure.