NVIDIA Run:ai GPU Fractioning Brings Shared A10G Memory to Amazon EKS

Two workloads, one A10G, and a hard memory ceiling. NVIDIA's open source orchestration stack just proved it can be done inside Amazon EKS.

Last Updated: November 20, 2026 Editorial Process
Editorial Process
See more of Inside AI's trusted news by adding us as a preferred source on Google.
AI neural network visualization
Published on: November 20, 2026

November 20, 2026, (Inside AI) — NVIDIA has opened up its Run:ai orchestration stack, and the first practical consequence is landing on Amazon Elastic Kubernetes Service. A new validated walkthrough shows how the GPU Fractioning Operator can place two workloads on a single NVIDIA A10G GPU inside Amazon EKS, each capped by an enforced 8 GiB memory contract, without relying on Multi-Instance GPU partitioning.

The demonstration matters because it attacks a familiar waste problem. Teams running notebooks, model evaluations, and small inference services often queue for a whole GPU even when most of its memory sits idle. The default NVIDIA device plugin exposes a physical GPU as one indivisible nvidia.com/gpu resource. Two Pods that each request one GPU cannot share a node, even when their combined memory demand would fit comfortably.

The walkthrough, validated on October 22, 2026 and repeated on November 18, 2026, used a single g5.xlarge instance in the US East (Ohio) Region. The A10G carries 23,028 MiB of GPU memory. Two Pods each held an 8 GiB contract and allocated 6 GiB. A third test Pod then attempted a 9 GiB allocation and was rejected by the driver.

Memory Limits Are Not Isolation Guarantees

The distinction between a memory contract and a security boundary is the central caveat. GPU fractioning builds on Multi-Process Service, or MPS, which permits concurrent CUDA processes on one device. NVIDIA documents separate workload error domains in its default mode and a shared error domain in sm-sharing mode. Neither arrangement constitutes a security guarantee, and the walkthrough tested memory limits only, not fault containment or compute performance.

Read: AWS Brings NVIDIA Dynamo to SageMaker HyperPod for Multi-Node LLM Inference

A memory cap also does not reserve proportional compute. Two Pods sharing a device can still contend for streaming multiprocessors and memory bandwidth. The demonstrated density improvement does not establish a throughput gain or a universal cost reduction. That gap between access and performance is where most production evaluations will succeed or fail.

The practical implication is that fractioning widens access to existing capacity without changing the underlying economics of a GPU. Operators gain scheduling control and per-Pod accounting, but they inherit a new monitoring burden. The experiment recorded 6,394 MiB of usage per Pod against a 6,144 MiB allocation, a gap of roughly 250 MiB attributed to CUDA context and runtime overhead. That overhead must be budgeted before contracts are sized.

The operator requires NVIDIA driver branch R615 or newer. At validation time, the EKS-optimized Amazon Linux 2023 NVIDIA image shipped the R580 and R595 branches, so the team built a custom image with Packer. The workflow installed driver 615.71.10 alongside the NVIDIA Container Toolkit and containerd. GPU Operator 26.7.1 supplied the device plugin while preserving the image-owned driver.

NVIDIA announced the open source release of Run:ai on October 13, 2026. The walkthrough uses two of its four modular components: KAI Scheduler and the GPU Fractioning Operator. KAI Resource Management and Workload Map, which address organizational governance and additional workload types, sit outside the tested scope.

The baseline test established the constraint before any sharing was enabled. Two Pods each requested one whole GPU. One ran, the other stayed Pending. After installing KAI Scheduler 0.18.5, which bundles GPU Fractioning Operator 0.1.7, both Pods scheduled onto the same physical GPU. The script verified matching node names and GPU UUIDs, and each Pod reported a bounded device view of 8,192 MiB.

"Fractioning can improve access to existing capacity when workload memory fits," the walkthrough's authors note, adding that production adoption also requires isolation controls and performance testing.

Amazon EKS already documents several sharing routes, including MIG, device-plugin time-slicing, and time-slicing through the NVIDIA DRA driver with Karpenter. Fractioning occupies a specific niche: enforced per-Pod memory limits on GPUs that lack MIG support, paired with fixed-size GPU node groups. Dynamic Resource Allocation offers a newer Kubernetes API for device requests, but this walkthrough relies on NvFractions annotations instead.

Read: Amazon SageMaker AI Adds KV Cache-Aware Routing for Faster LLM Inference

Operational friction remains. The fractioning daemons run as privileged containers with host PID access and NRI and MPS host mounts. Administrators must restrict deployment to trusted operators. The custom image cannot run on EKS Auto Mode-managed nodes, because Auto Mode controls its own AMI selection. Autoscaling is also unvalidated. Fractional Pods request no nvidia.com/gpu resource and target a node group fixed at one node. Karpenter support for secondary schedulers remains an open feature request.

Teams already running Kueue face a governance question before adoption. KAI Scheduler acts as a second scheduler, so organizations must decide which component owns queueing for GPU workloads. The walkthrough also recommends pinning the base AMI, driver, toolkit, device plugin, operator, scheduler, and workload images as one tested combination, then testing rollback before rollout.

The validated configuration is deliberately narrow. Hardware support alone does not establish fractioning compatibility, and the authors confirm only the G5 and A10G pairing. Smaller devices need smaller contracts and adjusted probes. The sample scripts currently require R615 and image-build metadata, so moving to a stock EKS-optimized AMI would demand script changes.

For teams weighing the tradeoff, the decision hinges on workload shape. Interactive notebooks with idle periods, short evaluation jobs, and compatibility tests fit the model well. Regulated or multi-tenant workloads with strict isolation requirements should stay on whole GPUs or MIG where the hardware supports it. The full sample, including the Packer build and cleanup stages, is published in the AWS Samples repository for teams that want to reproduce the results.

More from Inside AI

  • Features, Interviews, Press Releases

    Beyond Transcripts: Modulate Secures $25M to Scale Frontier Audio-Native AI Architecture Against Monolithic LLMs

    September 28, 2026
  • Agentic AI

    Financial Leaders Weigh Agentic AI Opportunities and Fraud Risks

    November 19, 2026
  • AI In Business

    Anthropic Raises $81.3 Billion in Record IPO, But Market Risks Loom

    November 19, 2026
  • AI In Business

    Anthropic Shares Jump 34% in Record IPO Debut

    November 19, 2026
  • AI Hardware & Infrastructure

    Alibaba Trains Next-Gen AI Model on In-House T-Head Chips

    November 19, 2026
  • AI In Business

    Anthropic IPO live: Claude maker goes public on Nasdaq after record share offering

    November 19, 2026
  • AI In Business

    Anthropic Raises $81.3 Billion in Record IPO, Topping SpaceX

    November 19, 2026
  • Artificial Intelligence (AI)

    Trump, Xi, Putin Meet as AI Outage Risks Loom

    November 19, 2026
  • AI In Business

    Anthropic Opens at $1.5 Trillion Valuation After Record $83 IPO

    November 19, 2026

Never Miss a Breakthrough

Join 50,000+ readers who get our daily AI intelligence briefing. No fluff, just what matters.

Join Our Newsletter Community

Subscribe

Inside AI is an independent publication covering artificial intelligence news, machine learning research, and the tools shaping the future of technology. No hype. Just what's happening in the AI world.

Topics

  • Artificial Intelligence
  • Machine Learning
  • Generative AI
  • Agentic AI
  • Vibe Coding
  • Prompt Engineering
  • AI Policy & Regulation
  • AI Hardware & Infrastructure
  • AI Tools
  • AI In Business
  • Robotics
  • Cybersecurity AI
  • AI Safety
  • AI Tools & Reviews (Coming soon)

Company

  • Editorial Standards
  • Privacy Policy
  • Terms of Service
  • Contact
  • About Us

Others

  • Press Releases
  • Features
  • Sponsored Content
  • Advertise with us
  • Newsletter

© 2026 Inside AI. All rights reserved.

Designed by Blue Flare Digital