TLDR: At OpenSearchCon Europe 2026, Bobby Mohammed (AWS) and Daniel Wrigley (OpenSource Connections) introduced Agentic Relevance Tuning (ART), a framework that uses specialized LLM agents to fully automate the search relevance lifecycle in OpenSearch. By continuously monitoring user behavior data, generating tuning hypotheses, and validating fixes via offline evaluations, ART bridges the gap between identifying search flaws and safely deploying ranking improvements. Operating within human-managed guardrails, the system shifts relevance optimization from a complex, manual staffing challenge into an automated infrastructure capability. |
Based on the OpenSearchCon Europe 2026 talk by Bobby Mohammed, AWS & Daniel Wrigley, OpenSource Connections | Watch the full session →
Search relevance tuning is the job that never ends. You fix the results for one query type, and something else regresses. You ship an improvement to ranking, and three weeks later a new content type arrives that the scoring logic was not designed for. The cycle of detect, analyze, hypothesize, test, and deploy runs perpetually, and it requires people who understand Lucene scoring internals, OpenSearch’s query domain-specific language (DSL), and how to design offline evaluations that actually predict online behavior.
Most organizations running production search either do not have those people, or they have them doing this work manually when they could be doing something more interesting. Bobby Mohammed from AWS and Daniel Wrigley from OpenSource Connections presented a different approach at OpenSearchCon Europe 2026: Agentic Relevance Tuning, or ART.
What ART actually is
ART is an end-to-end framework that uses LLM-powered agents to automate the full relevance lifecycle in OpenSearch. The key word is “automate”—not assist, not surface insights for a human to act on, but orchestrate the complete loop from detecting a problem to deploying a fix, with evaluation gates along the way.
The framework is built around specialized agents, each responsible for a distinct phase of the relevance improvement cycle. One agent continuously monitors relevance signals using data from OpenSearch’s User Behavior Insights (UBI)—the click, conversion, and session data that captures how users are actually responding to search results. When that agent detects a degradation or an opportunity, it hands off to another agent that hypothesizes what changes might improve things. That agent then works with the Search Relevance Workbench to design and run offline evaluations, and if an improvement is validated, it orchestrates the deployment.
Closing the loop between data and improvement
One of the persistent frustrations with relevance improvement tools is that they produce signals but not actions. OpenSearch UBI tells you that users are abandoning searches at a particular query type. The Search Relevance Workbench lets you run A/B comparisons between ranking configurations. But the leap from “I have this data” to “I know what to change and have confidence to deploy it” has historically required a human with deep expertise to sit in the middle.
ART is designed to fill that gap. The LLM-powered agents can reason over the relevance signals, generate hypotheses about what changes would address them, translate those hypotheses into OpenSearch query and ranking configurations, and evaluate whether the proposed changes hold up before they reach users. Mohammed and Wrigley were careful to describe this as augmentation rather than replacement — the agents operate within guardrails, and human oversight remains available — but the vision is a system that can catch and fix the majority of relevance issues without requiring a search engineer to intervene.
Why this is timely
The timing of ART is not accidental. The OpenSearch community has been building the underlying components — UBI, the Search Relevance Workbench, the neural and hybrid ranking capabilities — over the last several years. ART is an attempt to compose those components into something that delivers the full loop, not just individual tools that a skilled practitioner has to wire together manually.
For teams where search quality is important but search engineering expertise is limited, this framework represents a meaningful shift in what is achievable. The gap between “we have OpenSearch running” and “we have OpenSearch running with continuously improving relevance” has been primarily a staffing and expertise problem. ART makes a credible case that it can become, at least partially, an infrastructure problem instead — which is a much more tractable one.
The session is worth watching in full for the live demonstrations of the agent loop in action and the discussion of where the framework currently has limits. Relevance tuning is hard, and automating it is harder. Mohammed and Wrigley do not oversell what ART does today, which makes the honest account of its capabilities and boundaries genuinely useful.
