TLDR: At OpenSearchCon Europe 2026, Amazon’s Shubham Chaudhary shared how bypassing Java’s JVM limitations to implement native SIMD architecture achieved a 60% reduction in Lucene vector search latency. Instead of relying on traditional algorithmic changes, his team optimized performance at the hardware level by targeting architecture-specific code for ARM CPUs like AWS Graviton processors. These infrastructure-level improvements run automatically behind the scenes in updated versions of OpenSearch, with ongoing efforts to contribute the work upstream to the broader Lucene ecosystem. |
Based on the OpenSearchCon Europe 2026 talk by Shubham Chaudhary, Amazon | Watch the full session →
When people talk about making vector search faster, the conversation almost always goes to algorithms: better approximate nearest neighbor methods, smarter index structures, more aggressive quantization. What it rarely goes to is the CPU itself—specifically, how modern processors can be coaxed into doing dramatically more work per clock cycle if you meet them where they are.
Shubham Chaudhary’s session at OpenSearchCon Europe 2026 went there. His talk, “Using Native CPU Vectorized Instructions for 60% Speedup on Lucene’s KNN Vector Search,” is a deep dive into a category of optimization that is invisible to most application developers but that has real, measurable impact on the latency and throughput of every vector search query you run.
The bottleneck nobody was talking about
Vector search is computationally intensive at its core because of one operation: the dot product. Computing the similarity between a query vector and every candidate vector in an index requires enormous numbers of floating-point multiplications and additions. For large-scale workloads with high query volumes, the speed of this computation is a direct constraint on how fast your system can respond.
Modern CPUs have a feature specifically designed for this kind of problem: SIMD, or Single Instruction, Multiple Data. A SIMD instruction processes multiple data elements in a single operation—instead of multiplying one pair of floats at a time, the CPU multiplies eight or 16 at once. The potential speedup is substantial, but getting there requires that the software actually uses these instructions rather than falling back to scalar operations.
Why Java made this hard
The natural instinct for a Java-based system like Lucene is to use Java’s native access improvements, specifically, Project Panama, to access SIMD capabilities through the JVM. Chaudhary’s team tried this. It did not work well enough. Project Panama’s vectorization, while improving, introduced overhead and limitations that prevented the team from extracting the full performance available from the hardware.
The decision was to step outside the JVM and implement native SIMD directly, using architecture-specific code targeting ARM CPUs, specifically, the AWS Graviton 2 and Graviton 3 processors. This is not a small engineering decision. It means writing and maintaining code that is tied to specific CPU architectures rather than relying on Java’s portability guarantees. The team made that tradeoff deliberately, because the result, a 60% reduction in vector search latency on ARM-based systems, justified it.
The complexity of “multiple CPU families”
One of the more technically interesting parts of the session was the discussion of what it actually means to support multiple CPU families. ARM and x86 have different SIMD instruction sets. Different generations of the same architecture have different capabilities and optimal instruction sequences. The optimization that works brilliantly on Graviton 3 may not be the right implementation for Graviton 2, and it is a different story again on x86.
Chaudhary walked through the team’s approach to managing this complexity, including how they structure the code to select the right implementation at runtime based on the detected CPU capabilities. This is the kind of engineering that happens in the infrastructure layer and never surfaces in application code, but it is the work that makes the 60% number real.
What this means for OpenSearch users
If you are running OpenSearch on ARM-based infrastructure, which includes AWS Graviton instances, this work directly benefits your vector search workloads. The improvements are not something you need to configure or opt into. They are built into Lucene and flow through to OpenSearch as the Lucene version is updated.
Chaudhary also discussed progress toward contributing this work upstream to Lucene, which would make it available across the broader Java search ecosystem. That contribution, if completed, would be one of the more impactful pieces of infrastructure work to come out of the OpenSearch community in 2026.
For engineers interested in where search performance gains actually come from, below the algorithm level, in the intersection of software and hardware, this session is worth the full 45 minutes.
