Pushing a quantized 8B parameter model to the edge sounds straightforward until your P99 latency blows past 200 ms on a load spike you did not cause. The culprit is rarely the model, the inference framework, or the network path. It is almost always the compute tier underneath it, and the distinction between a VPS and a VDS is exactly where that problem lives.

As teams deploy open-weights models like Meta Llama 3.1 8B Instruct, Qwen3-8B, and GLM-4-9B to edge nodes for real-time inference, the VPS-vs-VDS decision stops being a budget line item and becomes a genuine architecture choice. The wrong answer does not just cost money; it breaks latency SLAs in ways that are genuinely difficult to diagnose.

This post explains the mechanism, gives you the numbers to reason against, and tells you where the boundary is between tiers that can hold a latency target and tiers that cannot.

What VPS and VDS actually mean (setting the terms)

Provider marketing has made these terms nearly interchangeable, so let us be precise about what we mean here. In a VPS, multiple virtual machines share the same physical host. You may receive dedicated RAM and a vCPU quota, but CPU cycles, storage I/O queues, and often NIC bandwidth remain shared across tenants. In a VDS, a full virtualisation hypervisor (KVM, Xen, or Hyper-V) assigns a fixed, exclusive slice of hardware to a single VM. The VDS behaves like a physical server: no other tenant’s workload touches its CPU scheduler, its memory bus, or its storage queue.

That distinction matters enormously for inference, and here is why.

CPU steal time: the silent latency killer

The technical mechanism behind VPS latency variance has a name: CPU steal time. It is the wall-clock time your VM spends waiting for the physical CPU while the hypervisor is serving another VM. In a quiet environment, steal time is negligible. When a neighbor runs a backup job, absorbs a traffic spike, or simply has a noisy cron schedule, steal time climbs fast.

A rule of thumb worth internalising: sustained CPU steal above roughly 10 percent is a sign of host contention. For a model serving process where a single forward pass might take 30 to 80 ms, adding 10 to 20 percent unpredictable wait on top of that is the difference between hitting and missing a 100 ms real-time target.

Storage contention compounds the problem specifically for inference workloads. Model weights are large. A quantized 8B model in 8-bit precision is roughly 8 GB on disk. On a VPS backed by a shared NVMe pool or SAN volume, the latency of loading those weights at startup, or at container restart after a crash, is subject to the same I/O queue contention as CPU steal. Cold starts on GPU instances already take 60 to 180 seconds for model loading under ideal conditions; shared storage pressure makes that worse and less predictable.

Provider overselling amplifies both effects. Marketing materials promise guaranteed resources, and that promise holds only when the host is not oversold. When demand is high across the physical machine, every tenant feels it simultaneously, and there is no mechanism in a standard VPS arrangement to protect your VM from the aggregate.

What edge AI inference actually requires

Latency requirements for edge inference vary by application, but the published targets are instructive. Real-time computer vision for AR/VR applications requires end-to-end latency under 100 ms. Radio resource allocation via deep learning inference must complete within the channel coherence time, which can be as short as 10 ms. On-device inference on modern mobile hardware has crossed a sub-20 ms threshold for production computer vision models. LoRA-adapted models on optimised edge configurations have demonstrated inference latency around 45 ms.

Even at the more relaxed end of that range, the math is tight. If your model’s forward pass takes 40 ms on a clean host, a VPS under contention that adds 15 to 25 ms of steal and I/O jitter pushes you past a 60 ms budget before network transit is even counted. A VDS, with no contention source, holds the 40 ms baseline consistently.

The framing that edge inference is faster than cloud round-trip (directionally, edge P99 under 100 ms versus 300 to 800 ms for a cloud round-trip) only holds when the edge compute itself is not introducing the variance. A poorly provisioned edge node is not faster than a well-provisioned cloud endpoint; it is just differently slow and harder to debug.

Where VPS is still the right answer

Not every inference workload needs a VDS. A VPS is appropriate when:

  • The model is small and quantized to the point where a forward pass completes in under 10 ms, leaving enough headroom to absorb steal variance.
  • The use case tolerates P99 latency in the 200 to 500 ms range, such as asynchronous document processing or batch classification.
  • Traffic is highly predictable and the host is lightly loaded, meaning steal time stays consistently low.
  • The deployment is exploratory and optimising for cost while validating the model architecture before committing to production.

The problem is that teams often start on a VPS for cost reasons and discover the contention problem only after they are in production, at which point the migration is urgent rather than planned.

Where VDS becomes non-negotiable

Switch to a VDS when the workload has any of the following characteristics:

  • Hard latency SLAs. If the product contract specifies a response time, shared-resource variance is an uncontrolled risk. A VDS eliminates the contention variable.
  • 7B to 9B parameter models at real-time serving rates. Models like Llama 3.1 8B Instruct, even after 8-bit quantization, consume enough compute per token that steal time is not absorbed by spare capacity; it stacks on top of inference time.
  • GPU inference at the edge. GPU scheduling in a shared environment introduces a second contention surface beyond CPU. Dedicated GPU allocation is the only way to hold a consistent token-per-second rate under concurrent request load.
  • Frameworks like llama.cpp or MLC-LLM in hybrid CPU/GPU mode. These frameworks are designed to extract maximum throughput from available hardware. They benefit from dedicated resources because they are written to use everything available; contention from other tenants directly caps their ceiling.
  • Stateful serving with warm model caches. Keeping weights resident in memory to avoid cold starts requires that memory to be genuinely reserved. On a VPS, memory pressure from neighbors can force eviction even when your allocation looks sufficient on paper.

Benchmarking your actual tier before you commit

Do not take a provider’s label at face value. Before deploying a production inference workload, measure steal time under synthetic load using the %st field in top or vmstat. Run your model loading sequence and record wall-clock time for weight loading, not just forward-pass time. Measure P50, P95, and P99 latency under concurrent request load, not single-thread throughput. A single headline throughput number hides the distribution that actually determines whether you hit or miss a latency SLA.

We run GPU inference and training workloads across our European infrastructure and the pattern is consistent: teams that benchmark their tier before committing to a serving architecture catch the contention problem in staging, not in production. The VPS-vs-VDS question is answerable with a few hours of measurement; it should never be discovered by a customer complaint.

The architecture conclusion

For edge AI inference with open-weights models in the 7B to 9B parameter class, serving real-time traffic against a latency SLA, a VDS is not a premium option; it is the minimum viable compute tier. The VPS saves money until the first production incident, at which point the cost of diagnosis, migration, and customer impact exceeds any savings. The better trade is to size the tier correctly from the start, measure the actual latency distribution in staging, and treat the compute tier as part of the inference architecture rather than a procurement afterthought.