Linux ships with networking defaults calibrated for general-purpose servers, not for cloud instances pushing sustained gigabit traffic with latency-sensitive workloads. Those defaults made sense when RAM was scarce and 100 Mbps was fast. On a modern VPS handling real-time data pipelines, high-frequency API traffic, or cross-border database replication, they quietly cap your throughput and inflate your tail latency. The good news is that three layers of tuning, TCP buffer sizing, interrupt affinity, and queue discipline, give you most of the gain, and all of them are reachable without kernel patches or exotic drivers.

TCP buffer sizes: stop leaving bandwidth on the table

The core concept is the Bandwidth-Delay Product. BDP is simply bandwidth multiplied by round-trip time, and it tells you how much data can be in flight before the first byte is acknowledged. At 10 Gbps with a 1 ms RTT, that is 1.25 MB. Stretch that to a cross-continent path at 100 ms and you need 125 MB of buffer to keep the pipe full. Now look at what the kernel ships with.

Typical Ubuntu and Debian stock values are net.ipv4.tcp_rmem = 4096 131072 6291456 and net.core.rmem_max = 212992. The maximum receive buffer ceiling of roughly 208 KB means the kernel cannot grow buffers beyond that regardless of what the connection needs. On a long-RTT path that ceiling alone can reduce effective throughput by an order of magnitude.

Raising the ceiling is the first move. A widely adopted starting point for 10 Gbps environments sets net.core.rmem_max and net.core.wmem_max to 134217728 (128 MB), and net.ipv4.tcp_rmem to 4096 1048576 134217728. The three values in tcp_rmem are minimum, default, and maximum. Be careful with the default (middle) value: Red Hat’s RHEL 10 documentation explicitly warns that setting it too high can cause buffer collapsing and latency spikes, recommending no more than 524288 bytes (512 KiB) for the default. Let auto-tuning, controlled by net.ipv4.tcp_moderate_rcvbuf, do the work of growing buffers toward the maximum only when a connection genuinely benefits.

The right maximum for your workload is two to three times your calculated BDP. Measure your actual RTT first with ping or ss --info, then size accordingly rather than copying a generic value from a blog post. Persist every change in /etc/sysctl.d/99-network-tuning.conf and apply with sysctl --system.

Raising the buffer ceiling costs nothing when auto-tuning is on: the kernel only allocates what the connection actually needs. The risk runs the other way — leaving the ceiling too low silently caps every long-RTT flow.

Interrupt affinity: get traffic off CPU0

By default, most Linux systems route all network interrupts to a single core, typically CPU0. At low traffic volumes this is invisible. Under sustained high-throughput load, CPU0 saturates while the rest of the machine sits idle, and your interrupt latency climbs.

The fix is SMP IRQ affinity. Each receive queue on a multi-queue NIC gets its own IRQ, and each IRQ has an smp_affinity bitmask stored in /proc/irq/IRQ_NUMBER/smp_affinity. Setting bit N in that mask allows CPU N to handle the interrupt. Spreading queue IRQs across cores, one queue per physical core is a reasonable starting point, can have a dramatic impact on multi-threaded throughput.

Before touching those files, stop irqbalance. It will overwrite manual settings while it runs, and its heuristics optimise for power management rather than network throughput. Once stopped, either use the set_irq_affinity script from your driver package or pin queues manually. For a four-queue NIC on an eight-core machine, spreading queues across cores 0, 2, 4, and 6 leaves the odd cores free for application threads sharing the same NUMA node.

RPS and RFS for single-queue VPS NICs

Many VPS instances expose a single-queue virtual NIC. Hardware IRQ spreading is not available, but the kernel’s software equivalents, Receive Packet Steering and Receive Flow Steering, cover most of the same ground.

RPS distributes incoming packets across CPUs based on a hash of the flow, preventing a single core from handling all receive processing. It does not increase the hardware interrupt rate, though it does introduce inter-processor interrupts. Configure it per queue in /sys/class/net/DEVICE/queues/rx-0/rps_cpus by writing a CPU bitmask. RFS goes a step further: it steers packets to the CPU core where the consuming application thread is actually running, improving data-cache hit rate. Together they give single-queue VMs a meaningful share of the multi-queue benefit at zero hardware cost.

Queue discipline: what happens before the packet leaves

The qdisc layer sits between the kernel’s IP stack and the NIC transmit ring. Every packet passes through it before transmission, and the algorithm you choose determines how the kernel handles bursts, drops, and scheduling under congestion.

The default on most distributions is pfifo_fast, a simple three-band priority FIFO. It is fine for light workloads. For latency-sensitive traffic sharing a link with bulk transfers, it is not.

Two alternatives are worth knowing in depth:

  • fq_codel combines Fair Queuing with CoDel (Controlled Delay) AQM. CoDel is an adaptive, no-knobs algorithm designed to address the failure modes of earlier AQM schemes like RED. It measures packet sojourn time rather than queue length, dropping packets only when they have been sitting in the queue too long. Fair Queuing isolates flows so that a single bulk transfer cannot crowd out interactive traffic. This is the right default for most mixed workloads.
  • fq (Fair Queue) without CoDel is preferred when you have a high-bandwidth, low-RTT path and you want per-flow pacing. It allows the kernel to pace TCP flows to their estimated rate, smoothing out bursts that would otherwise cause retransmits.

Switch the default qdisc with:

tc qdisc replace dev eth0 root fq_codel

To make it persistent across reboots, set net.core.default_qdisc = fq_codel in your sysctl file. Verify what is currently active with tc qdisc show dev eth0.

Putting it together: a practical sequence

Tuning in isolation produces noise. The sequence that gives clean signal is: measure first with iperf3 or netperf, apply one layer of changes, measure again, then move to the next layer. TCP buffer sizing has the largest impact on long-RTT bulk transfers. IRQ affinity matters most under high packet-rate workloads with many small flows. Queue discipline shapes the experience of mixed traffic where latency and throughput coexist on the same link.

At AMS1, where traffic regularly spans Benelux, the UK, and Western European markets over AMS-IX, the RTT variance between endpoints means BDP-based buffer sizing is not optional, it is the difference between a connection that saturates the link and one that stalls waiting for acknowledgements. The same logic applies to any VPS where your users or downstream services are not in the same city as the instance.

None of these settings require root on bare metal, a custom kernel, or vendor support tickets. They are all available in the standard Linux networking stack on any VPS. The defaults are conservative for a reason; tuning them is how you collect the headroom that was always there.