November 18, 2026, (Inside AI) — Amazon Web Services has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference. The managed add-on, announced today, allows enterprises to deploy large language models that exceed the memory of a single GPU instance across multiple nodes, with disaggregated prefill and decode serving.
The integration targets a growing bottleneck in production AI: models like GLM-5.3, whose FP8 weights alone surpass the 640 GB of GPU memory on a single ml.p5.48xlarge instance, cannot be served on one machine without splitting the workload. HyperPod now automates that split, handling node health, GPU scheduling, and networking through Elastic Fabric Adapter (EFA).
Dynamo, which supports engines such as vLLM, SGLang, and TensorRT-LLM, brings three core capabilities to HyperPod: multi-node deployment, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs Dynamo's custom resource definitions and operators, including the HyperPod Inference Operator, Dynamo operator, Grove for pod grouping, and KAI Scheduler for gang scheduling. AWS Deep Learning Containers provide the runtime.
Customers can configure deployments in two ways. Existing Dynamo users can bring their own DynamoGraphDeployment (DGD) manifest. HyperPod users can use an InferenceEndpointConfig with a dynamoSpec section, which offers defaults for routing and replica counts. Both paths support single-instance and multi-node setups.
Read: Amazon SageMaker AI Adds KV Cache-Aware Routing for Faster LLM Inference
Benchmarks shared by AWS show meaningful gains. For Qwen3-235B-A22B-FP8 on two p5en.48xlarge instances (16 H200 GPUs), disaggregated serving achieved 207 output tokens per second per GPU versus 101 for aggregated serving, a 2.05× improvement, while meeting latency targets. In a multi-node aggregated test with Qwen3.5-397B-A17B-FP8, a two-node configuration retained 99.1% of single-node throughput, indicating that crossing node boundaries does not impose a significant penalty.
The move reflects a broader industry shift toward disaggregated inference architectures, where prompt processing and token generation run on separate pools of GPUs. This separation allows independent scaling and tuning, which is critical for long prompts and high concurrency. Competitors like Google Cloud and Microsoft Azure have also introduced similar capabilities, but AWS's integration with HyperPod's managed infrastructure may lower the operational burden for enterprises already invested in SageMaker.
The integration also addresses a key pain point: KV cache transfers between prefill and decode workers. Dynamo uses EFA for these transfers, and HyperPod monitors EFA health, replacing failed nodes automatically. This level of automation could reduce the undifferentiated heavy lifting that often slows down AI deployments.
However, the benchmarks come from AWS's own testing, and independent validation is pending. The Qwen3.5-397B test, while showing near-linear scaling, used a model that fits on a single instance, so it serves as a controlled baseline rather than a real-world multi-node scenario. Enterprises should conduct their own benchmarks with representative traffic before committing.
Read: Microsoft Winds Down Oracle AI Cloud Deal Over Older Nvidia Chips
For now, the integration is available in supported AWS regions. AWS provides setup instructions and deployment examples in a public repository. The company recommends starting with a matching Dynamo recipe, verifying EFA communication, and benchmarking with tools like AIPerf.
If a research paper dropped this week, Mahesh has already read it. He covers AI tools and vibe coding for builders who need to know what is worth their time and what is not.