An IDC study commissioned by Lenovo surveyed nearly 3,000 enterprise IT leaders and found that 88% of AI proofs of concept never reach widescale deployment. For every 33 PoCs launched, four graduate. The study’s own gloss is blunt: the gap reflects “low level of organizational readiness in terms of data, processes and IT infrastructure.” S&P Global puts the abandonment rate at 42% of companies scrapping most AI initiatives in 2025, up from 17% the year before. Gartner’s analyst Anushree Verma forecast in June 2025 that over 40% of agentic AI projects specifically will be cancelled by the end of 2027, citing escalating costs and inadequate risk controls.
The models are not the problem. The models that power most enterprise PoCs are genuinely capable. What fails is the infrastructure assumption underneath them: that a system which works elegantly for one agent, one session, one turn will continue working when you point real concurrent load at it.
It will not. And the failure mode is specific enough that you can test for it before you commit to production capacity.
The architectural break nobody tests in a PoC
LLM serving achieves economic efficiency through stateless shared infrastructure. You batch requests, you share GPU time across sessions, you scale horizontally without caring which instance handles which user. Agentic workloads break this model entirely. An agent requires per-session state across multiple inference calls and tool executions. Each turn in a conversation appends to a growing context that must be available, intact, for the next turn.
Consider the arithmetic. A 256,000-token context window requires roughly 22 GB of KV cache for a single session. At 64 concurrent users that becomes approximately 1.4 TB. At 1,000 concurrent users, around 22 TB. A 70B model quantized to FP8 fits on a single 96 GB GPU with headroom to spare, for zero concurrent agent sessions at 128K context. The demo works. The model loads cleanly. Then you point three agents at it in production and the serving framework begins evicting context mid-task, because there is nowhere else for it to go. The agent does not crash; it quietly loses its thread.
The agent that fails in production is not failing because the model is wrong. It is failing because the infrastructure was sized for a demo, not for a conversation that grows with every turn.
Multi-turn conversations compound this. In a five-turn agent workflow with a 200-token system prompt and 150 tokens per turn, the final turn’s prompt contains roughly 950 tokens, of which only about 150 are new. Existing inference frameworks handle this poorly: they recompute the full prefix rather than extending a cached representation. Research benchmarks against vLLM and SGLang on multi-agent tool-calling workloads show a stateful reference implementation running 2.1 times faster per turn on a six-turn agentic workflow, with the advantage widening as conversation depth increases.
Tool-call latency compounds faster than teams expect
Most of an agent request’s total wall-clock time is spent waiting: waiting on model inference, on retrieval, on tool invocations, on memory lookups. The AWS Well-Architected Agentic AI Lens makes this explicit: executing independent operations concurrently, reusing warm connections, and deduplicating repeated lookups cuts total latency without changing models or prompts. But teams that introduce parallelism without respecting downstream rate limits or connection pool capacity discover that concurrent calls throttle or queue, turning the intended latency win into added latency plus cascading failures.
Failure-driven retry loops make this worse. When a tool call fails, the agent appends the error observation and tries again. Each retry grows the context. A long-tail failure that would be invisible in a single-session demo becomes a runaway context-growth pattern under concurrent load, saturating both GPU memory and tool-call queues simultaneously.
Session affinity is the piece most teams discover only after they have already deployed. Without consistent hashing on session or user ID at the load-balancer layer, a follow-up request in a multi-turn conversation can land on a vLLM instance that does not hold that session’s KV cache. The serving framework then performs a full re-prefill of the entire conversation history. This is a correctness issue as much as a latency issue: the agent’s apparent context is correct, but the latency penalty accumulates on every turn after the first, and the KV cache you paid to build is rebuilt from scratch on each hop.
The storage layer your PoC probably did not build
Production agentic infrastructure requires two distinct storage tiers that almost no PoC provisions correctly. Temporary storage holds conversation history during active sessions; Redis is the standard choice because it is fast and supports automatic expiration. Persistent storage maintains long-term memory, tool call history, and evaluation data across sessions and across restarts. These are different systems with different consistency and durability requirements, and conflating them, or omitting persistent storage entirely, is the second most common way a PoC collapses in production after KV cache eviction.
Efficient agentic serving requires jointly managing model execution, persistent context cache, tool interaction, and failure recovery. Optimising any one component in isolation, which is what a PoC almost always does, produces a system that looks production-ready until the second component fails under load.
The checklist that closes the gap
Before signing off on production capacity for an agentic workload, run these tests against your actual PoC environment:
- KV cache pressure test. Simulate your expected peak concurrent sessions at your target context length. Calculate the KV cache requirement (roughly 22 GB per 256K-token session for a large model). Confirm your GPU fleet can hold that cache without eviction. If it cannot, you need either more GPU memory, a KV offloading strategy, or a shorter effective context window.
- Session affinity verification. Send a ten-turn conversation through your load balancer with at least three backend instances. Confirm that every turn lands on the same instance, or that your cache-sharing layer makes the distinction transparent. Measure TTFT on turns two through ten.
- Concurrent tool-call saturation. Run your agent workflow at 50, 100, and 200 concurrent sessions simultaneously. Watch for queue saturation in your inference layer and throttling in your tool backends. Measure P99 TTFT, not mean TTFT: mean latency hides the tail behaviour that causes agents to appear hung.
- Retry loop containment. Deliberately inject tool-call failures at 10% and 25% rates. Confirm that your agent framework bounds retry depth and context growth. An agent that retries indefinitely will exhaust both KV cache and tool-call quota under failure conditions.
- Storage tier separation. Confirm that conversation history lives in a fast ephemeral store with TTL, and that tool call history and long-term memory persist across a full infrastructure restart. Run the restart. Verify the agent resumes correctly.
We run GPU inference workloads out of MUC1, our Munich facility, which serves the finance and industrial automation clusters where agentic AI deployments are moving fastest. The teams we work with on bare-metal GPU provisioning there see the same collapse pattern repeatedly: a PoC that demonstrated cleanly at a single-digit session count hits KV eviction and session affinity failures the first time a real user group is onboarded. The checklist above is what we walk customers through before they size production capacity, because retrofitting GPU memory and a Redis session layer after go-live costs significantly more than testing for it during the PoC phase.
What production readiness actually means for agentic AI
Capgemini’s 2025 research found that only 2% of organisations have deployed agentic AI at full scale. The gap between that number and the volume of PoCs in flight is not a model quality problem. The core failure pattern, as the research consistently describes it, is a capability-deployment verification gap: pilots succeed in controlled conditions and then falter in production because integration, data access, and the infrastructure layer were never stress-tested together.
The PoC that passes the five tests above is not guaranteed to succeed in production. But the PoC that skips them is almost certain to fail, at exactly the moment it matters most.
