Most object storage performance problems are not hardware problems. The disks are fast, the network cards are capable, and the CPUs are largely idle. The bottleneck is almost always the kernel’s default configuration, which was tuned for a general-purpose workload that looks nothing like sustained, high-concurrency object I/O. Getting from “the hardware should be faster” to “the hardware is faster” is a matter of a handful of sysctl knobs, an I/O scheduler choice, and understanding why each one matters.

This post walks through the specific parameters that move the needle: network buffers sized for real bandwidth-delay products, TCP congestion control that stops punishing you for shallow-buffer paths, and I/O scheduler selection matched to your storage media. None of this requires new capacity. It requires tuning.

Start with the bandwidth-delay product

The fundamental constraint on TCP throughput is the bandwidth-delay product (BDP): the amount of data that must be in flight simultaneously to fill a pipe. The formula is simple: max throughput = TCP window size / RTT. If your buffers are smaller than the BDP, the sender stalls waiting for acknowledgements, and you never see line rate regardless of what the NIC reports.

On a 10 Gbps link with a 50 ms RTT, the BDP is 62.5 MB. The Linux kernel default socket buffer ceiling is around 212 KB. That gap explains a lot of disappointing iperf3 numbers. The standard sizing rule is to set your maximum buffer to at least 2x the BDP, then let the kernel’s auto-tuning (net.ipv4.tcp_moderate_rcvbuf, enabled by default) scale within that range based on actual conditions.

The critical detail most guides miss: net.core.rmem_max and net.core.wmem_max are system-wide ceilings that apply to every socket, not just TCP. If you raise net.ipv4.tcp_rmem to 128 MB but leave rmem_max at the default 212 KB, the kernel silently caps at 212 KB. Both must move together.

The sysctl settings that matter for object storage throughput

For a 10 Gbps or higher environment, a reasonable starting configuration for kernel parameters affecting object storage throughput looks like this:

net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 1048576 134217728
net.ipv4.tcp_wmem = 4096 1048576 134217728
net.core.rmem_default = 262144
net.core.wmem_default = 262144

The three-value tuples in tcp_rmem and tcp_wmem are minimum, default, and maximum. The kernel allocates twice the value requested via setsockopt(), so the figures in /proc will look larger than the effective window. Keep that in mind when interpreting diagnostic output.

Object storage nodes typically handle large numbers of concurrent connections. Two parameters that are easy to overlook:

  • net.core.somaxconn = 65535 and net.ipv4.tcp_max_syn_backlog = 65535: the default listen queue depth is far too small for high-connection-rate services. Exhausting it causes silent connection drops that look like application errors.
  • net.core.netdev_max_backlog: the default of 1000 will drop packets under sustained load on a busy NIC. Raise it to at least 5000, more on high-packet-rate interfaces.
  • fs.file-max = 2097152: each object operation opens a file descriptor. Running out is a hard wall.
  • net.ipv4.tcp_tw_reuse = 1: safe for outbound connections and reduces TIME_WAIT accumulation on nodes that open many short-lived connections to backends.

Persist everything in /etc/sysctl.d/99-objectstore.conf and apply with sysctl --system. Changes in /etc/sysctl.conf directly are fine but harder to audit and override.

The kernel’s defaults are conservative by design. They were set for a world of 100 Mbps links and modest connection counts. Object storage in 2025 runs on 10 Gbps or 25 Gbps interfaces with thousands of concurrent connections. Closing that gap is not optional; it is the first thing to do before any capacity conversation.

TCP congestion control: the case for BBR

CUBIC, Linux’s default congestion control algorithm, is loss-based: it interprets packet loss as a signal of congestion and backs off. That works well on wired LAN paths with deep buffers. It works poorly on paths with shallow buffers, modest random loss, or meaningful RTT variation, all of which are common in object storage deployments that span availability zones or serve remote clients.

BBR (Bottleneck Bandwidth and Round-trip propagation time) takes a different approach. Rather than reacting to loss, it builds an explicit model of the path by measuring delivery rate and RTT continuously, then uses that model to set both the sending rate and the volume of data in flight. The practical result is higher throughput on paths where CUBIC would have backed off unnecessarily, and lower queuing delay on paths with deep buffers.

Enabling it requires two parameters together:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

The fq (fair queuing) qdisc is not optional here; BBR relies on it for pacing. Setting BBR without fq undermines the algorithm’s behaviour.

Two caveats worth stating honestly. First, BBR v1 (the version in most distribution kernels) is known to compete unfairly with CUBIC and HTCP on shared network paths; ESnet does not recommend it on networks where other congestion control algorithms are also in use. If your object storage nodes share a fabric with other workloads, test before committing. Second, recent ACM SIGCOMM research published in 2025 shows that BBR performance can deteriorate under virtualized scheduling conditions, which is relevant if you are running storage nodes as VMs rather than bare metal. BBRv3, announced in 2023, is targeting these issues, but upstream inclusion is still in progress.

I/O scheduler selection

The Linux multi-queue block layer offers three schedulers. The right choice depends entirely on the storage medium:

  • none: no scheduling; requests go directly to the device driver queue. Correct for NVMe SSDs, which have their own deep internal queues and do not benefit from kernel-side reordering. Adding a scheduler here adds latency without benefit.
  • mq-deadline: enforces deadline guarantees for both reads and writes, preventing write starvation. Good for SATA/SAS SSDs and mixed-media configurations where predictable latency matters alongside throughput.
  • bfq (Budget Fair Queueing): designed for fairness across processes and low latency for interactive workloads. Generally the wrong choice for object storage, where you want to maximise aggregate throughput rather than per-process fairness.

Check the current scheduler per device with cat /sys/block/nvme0n1/queue/scheduler. Set it temporarily with echo none > /sys/block/nvme0n1/queue/scheduler and make it persistent via a udev rule targeting the device type. Kernel parameters affecting object storage throughput at the I/O layer often get less attention than network tuning, but on a node with twelve NVMe drives, leaving them on mq-deadline or bfq visibly caps sequential throughput.

Readahead and dirty page writeback

Two additional knobs matter for large sequential object reads and writes. vm.dirty_ratio and vm.dirty_background_ratio control how much dirty data the kernel accumulates before forcing writeback. The defaults (20% and 10% of RAM respectively) are designed for desktop workloads. On a storage node with 256 GB of RAM, 20% is 51 GB of dirty data before the kernel applies backpressure. That can cause write latency spikes rather than smooth throughput. Lowering vm.dirty_background_ratio to 5 and vm.dirty_ratio to 10 keeps writeback continuous rather than bursty.

For large sequential reads, per-device readahead (/sys/block/sdX/queue/read_ahead_kb) should be sized to your typical object size. The default of 128 KB is appropriate for small random I/O; for objects in the multi-megabyte range, 2048 KB or higher reduces the number of I/O operations required to satisfy a single read.

Putting it together

At AMS1, where dense AMS-IX peering means object storage traffic often traverses multiple autonomous systems before reaching clients across Benelux and the UK, we run BBR with fq on storage nodes precisely because the path characteristics vary in ways that CUBIC handles poorly. The network buffer settings above are the baseline; actual values are validated with iperf3 against the real RTT of each traffic class.

Linux kernel tuning for storage is not a one-time configuration exercise. Workloads change, object sizes change, and client distribution changes. The parameters above are a starting point with understood reasoning behind each one. Measure before and after, keep changes in version control alongside your other infrastructure configuration, and treat any parameter you cannot explain as a risk rather than a setting.