The easiest mistake in event streaming is treating partition count as something you can fix later. You cannot, at least not on a keyed topic without breaking per-key ordering guarantees permanently. The second easiest mistake is sizing brokers around peak throughput numbers from a benchmark that does not match your hardware, your replication factor, or your consumer topology. Both mistakes tend to surface at the worst possible moment: during a traffic spike, not during a quiet Tuesday afternoon test.

This post works through the three sizing decisions that matter most before you push a production workload onto a Kafka-compatible cluster: partition count, consumer group design, and broker resources. The underlying principles apply whether you are running vanilla Kafka or a compatible managed service; we will note where NATS JetStream, which we run as our own production event-streaming backbone, behaves differently and why those differences matter for capacity thinking.

Partition sizing: the decision you cannot easily undo

The canonical approach to partition sizing, which Jun Rao articulated at Confluent in 2015 and which still underpins official Confluent documentation, is straightforward in principle: divide your target throughput by the per-partition throughput your hardware can sustain, once for the produce path and once for the consume path, then take the larger of the two results. The formula forces you to commit to a per-partition throughput constant before you know your actual workload, which is where most plans go wrong.

The range of published per-partition throughput figures is wide enough to be dangerous. Netflix’s Keystone pipeline uses 0.5 to 1 MB/s per partition as a conservative planning constant for their specific workload. Conduktor’s 2025 guidance for well-tuned Kafka 3.0+ clusters on modern NVMe hardware puts the ceiling at 50 to 100 MB/s per partition. These numbers are not contradictory: Netflix’s figure is a deliberate underestimate built for headroom; Conduktor’s is a hardware ceiling achievable only with careful tuning. Use the wrong constant for your context and you will either over-partition wildly or under-partition badly.

A practical starting point for most production workloads on commodity hardware is 1 to 3 MB/s per partition as a planning constant, then stress-test that assumption against your actual disk and network before you finalise topic configuration. Confluent’s own guidance recommends staying between 100 and 200 partitions per broker as a baseline, with a hard warning against exceeding 4,000 partitions per broker because the controller overhead at that scale can destabilise the cluster. Conduktor notes that Kafka 4.0 clusters with adequate CPU, memory, and disk I/O can exceed that ceiling, but that is a tuned-hardware exception, not a default assumption.

Adding partitions to a keyed topic after creation is a one-way door. The default partitioner uses murmur2(key) % partition_count to route messages, so changing partition count silently re-routes keys that were previously co-located, breaking ordering for any downstream consumer that assumed a stable assignment. Over-provision at topic creation and do not alter keyed topics.

The over-provisioning imperative is not theoretical caution: it is the documented production pattern at LinkedIn, Shopify, Cloudflare, and Confluent. LinkedIn’s clusters run more than 7 million partitions across 4,000+ brokers handling over 7 trillion messages per day as of 2023. They built custom tooling to emit per-partition throughput metrics because Kafka does not expose them natively, and they run Cruise Control on every cluster for partition rebalancing. If your tooling does not give you per-partition visibility, you are flying blind on the metric that matters most.

One detail that frequently causes ordering bugs in production: null keys. In Kafka 2.4 and later, null keys trigger the sticky partitioner, which batches messages to a single partition for a configurable window before switching. In older versions, null keys round-robin across all partitions. Either way, null-key messages carry no ordering guarantee across partitions, and mixing keyed and null-key messages on the same topic is a reliable way to introduce subtle race conditions in downstream processing.

File descriptor limits are a broker-level concern that partition count directly drives. The standard formula is partitions multiplied by partition size divided by segment size. The common recommendation from both Cloudera and LinkedIn is to start with at least 100,000 file descriptors per broker and adjust upward as partition count grows. Missing this during provisioning produces cryptic errors under load rather than clean capacity failures.

Consumer group throughput: the partition ceiling and how to work around it

Kafka enforces a hard ceiling on consumer parallelism: a consumer group can have at most one active consumer per partition. Adding an eleventh consumer to a group assigned to a ten-partition topic gives you one idle standby, not more throughput. This coupling between partition count and consumer parallelism is why partition sizing decisions made at topic creation time constrain your scaling options months later.

The standby consumer is not useless: it provides fast failover if an active consumer dies. But if your throughput requirement grows beyond what your current partition count supports, your only options are to increase partitions (with the key-ordering caveat above), use a multi-threaded consumer model, or restructure the topic. Kafka 4.0 introduced Share Groups as a new consumer model intended to break the one-consumer-per-partition constraint, but the GA status and production semantics of Share Groups in 4.0 were not confirmed in the sources available to us; treat that as a feature to watch rather than a current planning input.

Rebalancing is the other consumer-side capacity trap. Kafka pauses processing for all consumers in a group during a rebalance while it recalculates partition assignments. In environments with auto-scaling or ephemeral consumers, frequent membership changes mean frequent pauses. The cooperative sticky assignment strategy reduces rebalance cost by maintaining existing assignments where possible and only migrating partitions that need to move, which is the correct default for any deployment where consumer count changes dynamically.

On the tuning side, three parameters dominate consumer throughput. fetch.min.bytes tells the broker to wait until it has accumulated at least that much data before responding, trading latency for batch efficiency. fetch.max.wait.ms sets the upper bound on that wait. max.poll.records controls how many records the consumer processes per poll loop; raising it increases throughput but also increases the processing time per loop, which can trigger the session timeout and trigger a rebalance if your processing logic is slow. These three parameters interact, and tuning one without the others is a common source of unexpected rebalances.

Broker sizing: the three constraints that bound your cluster

Broker sizing reduces to three independent throughput ceilings, and the minimum of the three is your actual cluster limit. AWS’s documented formula for this accounts for storage throughput multiplied by broker count divided by replication factor; network throughput between brokers and storage (relevant for network-attached storage) multiplied by broker count divided by replication factor; and EC2 or host network throughput multiplied by broker count divided by consumer groups plus replication factor minus one. Work out all three and your cluster is bounded by whichever is smallest.

Memory sizing follows a simple heuristic: buffer for 30 seconds of write throughput. A 64 GB RAM broker is a reasonable starting point for moderate workloads, but the more important principle is that Kafka relies heavily on the Linux page cache for read performance. Under-provisioning RAM degrades read throughput before it shows up as CPU pressure, which makes memory the resource most frequently under-sized in initial deployments.

Network throughput is typically the first bottleneck in practice, not CPU. A topic with replication factor 3 and 5 active consumer groups multiplies your raw ingest throughput by 8 on the broker network: 3 for replication and 5 for fan-out. Size your broker NICs and your inter-broker network accordingly, not just for ingest.

We operate Kafka-compatible managed infrastructure under the same Tier III, ISO 27001, and contractual 99.9% uptime SLA that covers the rest of our platform, with sovereignty as a contract-level guarantee rather than a marketing line. That means the capacity headroom we build into broker provisioning is auditable and consistent, not a function of shared-tenancy overcommit ratios that shift under load. The sizing decisions above are ones our engineers make for every new streaming deployment; the principles hold regardless of where you run.

Putting it together: a pre-launch checklist

  • Commit to partition count at topic creation. Use your target throughput divided by a conservative per-partition constant (1 to 3 MB/s on commodity hardware) and round up generously for keyed topics.
  • Never alter a keyed topic’s partition count in production. If you must scale, create a new topic and migrate consumers.
  • Match consumer count to partition count per group. Extra consumers are standby failover, not throughput.
  • Use cooperative sticky assignment in any environment where consumer group membership changes dynamically.
  • Tune fetch.min.bytes, fetch.max.wait.ms, and max.poll.records together, not independently.
  • Size broker memory for page cache first, not just JVM heap.
  • Calculate all three throughput ceilings (storage, storage network, host network) and design to the minimum.
  • Set file descriptor limits to at least 100,000 per broker before you hit production partition counts.
  • Instrument per-partition throughput. Kafka does not expose it natively; build or adopt tooling that does before you need it.