November 19, 2026, (Inside AI) — Exa, a search infrastructure company building a semantic web index, has raised embedding throughput by roughly 50% and cut performance and cost per million documents embedded by 47% against its initial baseline. The gains come from optimizing inference workloads on AWS Inferentia2 and AWS Trainium2 chips, using a mix of custom kernels and an agentic development workflow. The company says the work directly supports its effort to index tens of billions of documents.
The result matters beyond one startup's infrastructure bill. Exa operates one of the few independent web indexes optimized for semantic search, a category that competes with established search providers and feeds retrieval systems for AI agents. At that scale, small per-document cost changes compound into large annual savings. Exa's disclosure offers a rare public look at how a company tunes alternative AI silicon for production embedding pipelines, a workload that underpins nearly every retrieval-augmented AI system.
Exa's embedding pipeline is asynchronous and high-volume. Raw web data moves from crawlers through parsing, moderation and filtering, then into an embedding stage before entering a vector database. Every document ranked as high enough quality must be embedded before it can be served. The company uses multi-tower semantic models to place web data and queries into the same vector space, allowing natural language queries alongside keyword search.
The optimization effort began with a backfill. Exa moved to a superior document model and selected 30 billion high-priority documents for re-embedding. That migration temporarily doubled embedding load, creating a natural trigger to re-evaluate hardware choices. Exa ran two parallel pipelines sharing crawl and parse stages: one embedding with the live model into the live index, and one re-embedding priority documents with the new model into a new index.
Read: NVIDIA Dynamo on SageMaker
Custom Kernels Beat Generic Libraries
The initial Inferentia2 implementation was a straightforward data-parallel, FP16 representation of the model. From that baseline, Exa pushed throughput higher using the Neuron SDK and support from the AWS Trainium team. The first gains came from weight layout optimization, which eliminated unnecessary transposes by aligning weight layout with the broader model's scheduling. Exa also enabled autocast, letting the compiler downcast matrix multiplications to lower precision while preserving dynamic range. The downcast outputs met a correctness gate of 0.999 cosine similarity against the reference model.
The Neuron Kernel Interface library provided optimized kernels for common layers. A drop-in attention kernel delivered the next performance bump. The next bottleneck was memory spill. Profiles from Neuron Explorer consistently showed large amounts of spill and reload from high-bandwidth memory, keeping the Tensor Engine from being fully utilized. Exa wrote a custom weight-stationary fused SwiGLU MLP kernel that eliminated the HBM stalls. The company used the higher-level NKI language for layout operations, loops and memory operations, dropping into lower-level NKI ISA only for specific tensor operations.
At that point Exa hit a wall. Per-accelerator throughput measured with the other on-chip core idle was significantly higher than what the company saw with both cores running simultaneously. That gap pointed to contention between cores. Profiling showed the workload was not yet saturating the Tensor Engine. Exa rewrote the attention kernel around the exact shapes in its model to cut FLOPS spent on padding. Generic kernels handle any shape, so they cannot exploit the specific shapes in a given architecture. Exa also folded attention operations into single matmuls where the generic kernel used multiple instruction steps, cutting inner-loop operations and HBM round trips.
In total, the final version delivered a 47% improvement in performance and cost per million documents embedded against the initial baseline. Exa describes the throughput gain as roughly 1.5×, or about 50%.
The agentic methodology deserves scrutiny. Exa ran all optimizations inside a three-part harness that verifies correctness, measures performance and profiles the workload. Using Flyte, an ML ops workflow platform, each agent runs its own workflow. The workflow compiles a NEFF Neuron executable and checks a forward pass against a baseline dataset, which must reach 0.999 cosine similarity to the canonical model. It then profiles one forward pass and runs a 30-minute saturation benchmark on real hardware. Lightweight models handle grid searching compiler parameters, reasoning agents test complex hypotheses and find bottlenecks, and the strongest code generators write custom kernels. Matching each agent to its task keeps costs in check, and validation prevents capable agents from drifting. Exa also noted that many agents do not know the latest features and syntax, so Neuron agentic development skills helped.
This agentic kernel-tuning approach is becoming a template across the industry. The economics are straightforward: a reasoning agent that proposes a kernel variant costs cents in inference, while an engineer testing the same hypothesis might spend hours. The correctness gate at 0.999 cosine similarity is the critical control, because it prevents an agent from trading accuracy for speed in ways that would silently degrade retrieval quality. Exa's disclosure suggests that the bottleneck for alternative AI silicon is no longer hardware availability but the software tooling and talent needed to extract performance from it. If agentic workflows lower that barrier, adoption of non-Nvidia accelerators for inference could accelerate.
Read: KV Cache-Aware Routing
Flexible Compute Beats Single-Region Lock-In
Exa has optimized Inferentia2 for the US West (Oregon) Region, but tuning one hardware family in one region has hard limits. Spot and on-demand availability vary by region, and organizations with many consumers compete with themselves. Locking into one region or instance type almost guarantees a compute shortage and possibly degraded service. Exa schedules compute across multiple regions from one Kubernetes control plane. Its scheduler compares performance per dollar across instance types and packs pods onto the cheapest nodes, rescheduling as embedding demand shifts. The Inferentia2 work carried over directly to Trainium2. Shifting hardware profiles lets Exa free up nodes other teams need or absorb their unused reserved compute.
Exa's workload runs on a single NeuronCore, so the company runs Kubernetes pods that pack onto larger instances without leaving cores idle, giving container homogeneity across the embedding fleet. The same process runs on a Trainium2 NeuronCore and scales through data parallelism. Exa says optimizing across hardware pays off enormously for a long-lived model, and that adding Neuron hardware to its pool has proven a major source of long-term cost savings.
The broader context is a market where AI companies are under pressure to reduce inference costs as model sizes and document volumes grow. AWS has positioned Inferentia and Trainium as cost-effective alternatives to Nvidia GPUs for inference, and customer case studies like Exa's serve as proof points. Whether the gains hold at larger scale and across more regions remains an open question. Exa's disclosure does not specify absolute throughput figures or total infrastructure spend, limiting outside verification of the claimed savings.
If a research paper dropped this week, Mahesh has already read it. He covers AI tools and vibe coding for builders who need to know what is worth their time and what is not.