November 17, 2026, (Inside AI) — Amazon Web Services has added a routing feature to its SageMaker AI platform that directs incoming requests to the GPU instance already holding the relevant cached prompt data. The capability, called KV cache-aware routing, aims to cut time to first token for large language model workloads that involve long or repeated prompts. It is available now in all AWS Regions where SageMaker AI operates, at no extra charge.
The update targets a specific inefficiency in multi-instance LLM serving. When a model processes a prompt, it computes a key-value (KV) cache that stores the internal state of the prompt's tokens. Serving engines like vLLM keep this cache to avoid recomputing shared system prompts, documents, or conversation history. But across several instances, a request that lands on an instance without the matching cache forces a full recomputation. Under high load, instances also evict cached blocks to free GPU memory, including shared prompts and earlier conversation turns. The new routing strategy addresses both problems by sending each request to the instance that holds the longest matching part of its prompt in cache.
How Live Cache Reporting Changes Request Routing
The mechanism relies on the vLLM container reporting the cache blocks it stores and evicts. SageMaker uses that live view to make routing decisions. If the lookup is slow or finds no match, the endpoint falls back to load-aware routing, so the request is still served. This fallback design matters because cache-aware routing can add latency if the lookup itself becomes a bottleneck. AWS positions the feature as complementary to prefix-aware routing, which suits stable shared prompts, short prompts, or moderate load. KV cache-aware routing is recommended for long prompts, high load, and limited GPU memory.
Customers enable the feature by setting 'RoutingStrategy' to 'KV_AWARE' on a production variant and adding one environment variable so the container reports cache events. They can invoke the endpoint with the same APIs and OpenAI-compatible payloads. The feature supports single-model endpoints and inference components, including LoRA adapters, on GPU instances. It requires the AWS vLLM Deep Learning Container version 0.33.0 or later, with at least two instances or inference component copies. Detailed observability metrics show KV cache hit rate.
Read: Microsoft Oracle AI Cloud Deal
The move reflects a broader shift in LLM serving economics. As models grow and prompts lengthen, the cost of recomputing context has become a major latency and compute burden. Routing based on cache locality is one way to reduce that waste without changing the model itself. It also signals that inference infrastructure is increasingly about orchestration and memory management, not just raw GPU throughput.
Competing cloud providers have introduced similar optimizations. Google Cloud's Vertex AI and Microsoft's Azure AI offer various caching mechanisms, though the specifics of their routing strategies differ. AWS's approach ties routing directly to the serving engine's cache state, which could give it an edge in workloads with highly repetitive prompt structures. However, the feature's effectiveness depends on the vLLM container's reporting accuracy and the overhead of the lookup process. AWS has not disclosed benchmarks showing the actual reduction in time to first token.
Read: Advanced Chip Packaging
The feature is available today. Customers need to update to the latest AWS SDK or CLI to use it. More details are available on the Amazon SageMaker AI inference page and in the API documentation.
If a research paper dropped this week, Mahesh has already read it. He covers AI tools and vibe coding for builders who need to know what is worth their time and what is not.