Is your search evaluation process feeling a bit…manual?
In Taking your first steps towards relevance, we discussed how challenging improving search can be and introduced the Search Relevance Workbench. Then, in Measuring and improving search quality metrics, we introduced an objective approach for measurement and the need for judgments:
A judgment list is a foundational asset for quantifying search quality, as it tells us what good and bad search results look like.
Traditionally, search teams rely on human experts to evaluate search quality. Humans, however, are expensive, get bored easily, and can be hard to recruit for your project. What if there was a way to get consistent, scalable, and intelligent search evaluation without the bottlenecks of manual judging?
OpenSearch 3.5 introduced LLM as a Judge, a powerful new feature that uses large language models to automatically evaluate search result relevance. This capability removes the biggest barrier to getting started with search evaluation: the need for human judgments. LLM as a Judge is having a moment, and now it’s having its moment at OpenSearch!
TL;DR
OpenSearch 3.5 introduces LLM as a Judge, a feature that uses large language models to automatically evaluate search result relevance at scale. Teams connect to an LLM provider, define custom evaluation prompts tailored to their domain, and generate consistent relevance assessments for any query-document pair without relying on expensive human judges. Judgments are cached for cost efficiency and repeatable experimentation within the Search Relevance Workbench.
The search evaluation challenge
Before diving into LLM as a Judge, it helps to understand the landscape. Search evaluation typically relies on one of several approaches to gathering judgments, each with its own tradeoffs.
Subject Matter Experts (SMEs) provide the most accurate, factual assessments. They can definitively determine whether a document matches a query from a technical standpoint. However, SMEs are expensive, rare, and often don’t consider practical user needs. For most organizations, getting consistent access to SMEs for ongoing search evaluation is simply not feasible.
Expert Users bridge the gap between technical accuracy and practical relevance. They understand both the content domain and user goals, making them invaluable for search evaluation. Like SMEs, they are limited by availability and cannot scale to evaluate large volumes of queries.
Implicit judgments from user behavior data capture real user preferences and unconscious biases. While this reflects actual user satisfaction, it requires substantial traffic, can reinforce harmful biases, and provides no evaluation capability for new features or data before they go live.
Each approach has its place, but none solve the fundamental challenge: how do you evaluate search quality at scale, consistently, and cost-effectively?
LLM as a Judge represents a new approach. By connecting OpenSearch to powerful language models such as GPT-4, Claude, or other LLMs, teams can automatically generate relevance assessments for any query-document pair without the limits of human judgments.
The Tomato Test
Consider the Tomato Test to understand when LLM as a Judge fits best.
Question: “Is a tomato a fruit or vegetable?”
- Subject Matter Expert: “Botanically, tomatoes are fruits based on scientific classification.”
- Expert User: “Our users (home gardeners) think of tomatoes as vegetables for cooking purposes.”
- LLM as a Judge: “In everyday usage, tomatoes are commonly considered vegetables, though botanically they’re fruits. For a cooking-focused search, this result belongs in the vegetable category.”
- Implicit Judgments: “User clicks show people search for tomatoes in the vegetable section.”
The LLM synthesizes multiple perspectives and applies contextual reasoning, a key strength when evaluating search results across diverse domains.
Learn more about the different approaches to judgments in the blog post Four horseman of the judging apocalypse.
Why LLM as a Judge works
Consistent evaluation at scale. Unlike human evaluators who may have off days or interpret guidelines differently, LLMs provide consistent evaluation criteria across thousands of assessments.Once an effective prompt is in place, every evaluation follows the same logic. Teams should still run interrater reliability checks like Fleiss Kappa to validate consistency over time.
Multi-factor assessment. Human evaluators typically focus on pure relevance to avoid cognitive overload, LLMs can simultaneously consider multiple factors if provided appropriate instructions. Relevance and recency. Relevance, authority, and user intent. LLMs handle complex, multi-dimensional evaluation criteria in a single assessment.
Always available. No scheduling conflicts, no vacation days, no training sessions. LLM judges are available 24/7, enabling rapid iteration and continuous evaluation.
Cost-effective scaling. Human expert time costs hundreds of dollars per hour. LLM evaluations cost pennies per assessment. This economic shift makes comprehensive evaluation feasible for organizations of all sizes. A massive pool of judgments, even imperfect ones, is typically more valuable than a tiny pool of perfect human judgments that do not cover the query domain broadly.
How LLM as a Judge works in OpenSearch
The Search Relevance Workbench in OpenSearch 3.5 makes LLM evaluation straightforward.
Prerequisites
- OpenSearch 3.5 or later
- Access to an LLM provider (OpenAI, AWS Bedrock, Azure OpenAI, or others)
Process
- Connect to an LLM service. Set up a connector to your preferred LLM provider.
- Define your evaluation criteria. Create custom prompts that specify exactly what constitutes relevant results for the domain.
- Generate judgments. The system automatically retrieves search results and sends them to the LLM for evaluation
- Cache and reuse. Judgments are cached to reduce costs and enable consistent comparisons across experiments
- Run experiments. Use LLM-generated judgments to evaluate different search configurations and measure improvements in the Search Relevance Workbench, just as with human or implicit judgments.
Example
Suppose an e-commerce site and want to evaluate how well your search handles the query “wireless headphones.” Here’s how LLM as a Judge might assess different results:
{
"query": "wireless headphones",
"results": [
{
"title": "Sony WH-1000XM4 Wireless Noise Canceling Headphones",
"llm_rating": 0.95,
"reasoning": "Perfect match - wireless headphones with premium features"
},
{
"title": "Bluetooth Speaker with Wireless Connectivity",
"llm_rating": 0.2,
"reasoning": "Wireless audio device but not headphones"
},
{
"title": "Wired Gaming Headset with Microphone",
"llm_rating": 0.1,
"reasoning": "Headphones but not wireless, doesn't match query intent"
}
]
}
The LLM provides both numerical ratings and explanations, giving you insight into the evaluation logic.
The power of custom prompts
One of LLM as a Judge’s greatest strengths is prompt customization. Teams can tailor evaluation criteria to specific domain and business needs with simple template langauge:
Knowledge base example:
Evaluate how well these {{hits}} answer the user’s question ‘{queryText}’. Consider accuracy, completeness, and recency. Rate 0-1 where 1 means the article fully answers the question with current, accurate information, and 0 for the rest.
E-commerce example:
You can even reference specific fields in your document like brand and instock
Rate the relevance of these search {{hits}} for the query ‘{{queryText}}’ on a scale of 0-1.
Consider: exact product match (0.4), brand reputation (0.3), price competitiveness (0.2), and availability (0.1). Prioritize in-stock items {{instock}} and popular brands {{brand}}.
News search example:
Rate these news {{hits}} for the query ‘{{queryText}}’ considering relevance (0.5), recency (0.3), and source credibility (0.2). Recent articles from established news sources should score higher.
Understanding the trade-offs
Like any evaluation method, LLM as a Judge has limitations to consider alongside its strengths.
Advantages
Consistent at scale. No judgment fatigue, no inconsistency between evaluators, no training drift over time.
Always available. Evaluate new features, run experiments, and iterate quickly without waiting for human availability.
No judgment fatigue. LLMs don’t get tired or lose focus after evaluating hundreds of query-document pairs.
Rapid iteration. Test new search configurations immediately rather than waiting weeks for human evaluation cycles.
Limitations
Domain-dependent performance. LLMs work best in domains similar to their training data. Highly specialized or technical domains can see reduced accuracy. Providing both positive and negative examples in the prompt can help fill in some of the missing domain knowledge.
Prompt engineering required: Effective evaluation requires careful prompt crafting and testing. Poor prompts lead to poor judgments. OpenSearch ships two example prompts, but customizing with domain-specific positive and negative examples of search results produces the best results.
Mixed reliability: While generally consistent, LLMs can produce unexpected or inconsistent evaluations. This is similar to human evaluators and can be addressed with interrater reliability metrics.
Potential for hallucination: LLMs might provide confident-sounding but incorrect assessments. This applies to all use of LLMs, and isn’t specific to this use case.
Cost management. Teams pay to send data to an LLM. Sending data not related to the judgment increases costs without improving judgement quality. OpenSearch filters out data such as vectors from the judgment payload. Judgements are cached based on the exact prompt template, query text, document ID, and index name. Changing any of these parameters invalidates the cache.
Getting started
Check out the comprehensive tutorial on LLM as a Judge and then explore into the documentation for all the settings available to tune the judgment process.
Conclusion
While not a silver bullet, LLM as a Judge in the OpenSearch Search Relevance Workbench offers a powerful new tool that drastically simplifies the challenge of obtaining usable judgments. When used thoughtfully with appropriate prompts and validation, it dramatically accelerates search improvement efforts.
Judgements are the yardstick for measuring your search quality, so having good ones is critical! Get started today:
- Tutorial: Check out our comprehensive tutorial on LLM as a Judge
- Documentation: Explore the Search Relevance Workbench documentation to get started today
- Training: Level up your skills with LLM as Judges for Search Quality training class.
