Introducing the z-score normalization technique for hybrid search

In search engines and machine learning, data normalization is essential for making fair and accurate comparisons between different features or scores.

Hybrid queries use multiple normalization techniques to prepare final results. The two main types are score-based normalization and rank-based combination. In score-based normalization, the default method—min-max normalization—doesn’t handle outliers well. (Outliers are data points that significantly differ from the rest of a dataset. In techniques like min-max scaling and z-score normalization, outliers can greatly impact results.)

In this blog post, we’ll introduce z-score normalization, a technique added in OpenSearch 3.0-beta1. We’ll explore what it is, why it matters, and how OpenSearch uses it.

Z-score normalization, also known as standardization, is a method of scaling data using mean and standard deviation. The formula for calculating the z-score is:

Z = (X – μ) / σ,

where:

• X is the original value.
• μ is the population mean.
• σ is the standard deviation of the population.

Your index’s structure can help you decide which normalization technique to use, as each has its own advantages. If your documents are highly similar and a typical query returns top-k results that are closely clustered together within the index (as shown in the following graph), min-max normalization may be the better choice.

However, z-score normalization is a better choice if the results are more evenly distributed and follow a pattern similar to a normal distribution, as shown in the following graph.

The basic process for choosing between min-max and z-score normalization is outlined in the following diagram.

To use z-score normalization, create a search pipeline and specify z_score as the technique:

PUT /_search/pipeline/z_score-pipeline
{
    "description": "Zscore processor for hybrid search",
    "phase_results_processors": [
        {
            "normalization-processor": {
                "normalization": {
                    "technique": "z_score"
                }
            }
        }
    ]
}

Next, create a hybrid query and apply the pipeline to the query:

POST my_index/_search?search_pipeline=z_score-pipeline
{
   "query": {
     "hybrid": [
         {}, // First Query
         {}, // Second Query
         ... // Other Queries
     ] 
   }
}

Benchmark experiments were conducted using an OpenSearch cluster consisting of a single r6g.8xlarge instance as the coordinator node along with three r6g.8xlarge instances as data nodes with one shard. To assess z-score normalization performance comprehensively, we measured two key metrics across five distinct datasets. For more information about the datasets used, see Datasets.

The following table provides sample queries and passages for each dataset.

Search relevance was quantified using the industry-standard normalized discounted cumulative gain at rank 10 (NDCG@10). We also tracked system performance using search latency measurements. This setup provided a strong foundation for evaluating both search quality and operational efficiency.

The following table compares NDCG@10 scores for hybrid search using min-max normalization and z-score normalization, showing the percent difference in search relevance across five datasets.

The following table presents search latency measurements in milliseconds at different percentiles (p50, p90, and p99) for hybrid search using min-max and z-score normalization. The Percent difference columns indicate the relative performance impact of these methods.

Our benchmark experiments highlight the following advantages and trade-offs of using z-score normalization instead of min-max normalization in hybrid search:

• Search quality (measured using NDCG@10 across four datasets):

  • Z-score normalization shows a modest improvement in search quality, with an average increase of 2.08% in NDCG@10 scores.
  • This suggests that z-score normalization may provide slightly better search result relevance compared to the default min-max normalization.

• Latency impact:

  • Z-score normalization shows a small increase in latency across different percentiles, as shown in the following table.

  Latency percentile	Percent difference
  p50	0.70%
  p90	0.50%
  p99	0.50%

  • The positive percentages indicate that z-score normalization has slightly higher latency compared to min-max normalization, but the differences are minimal (less than 1% on average).

• Trade-offs:

  • There’s a slight trade-off between search quality and latency. Z-score normalization offers an improvement in search relevance (2.08% increase in NDCG@10) at the cost of a marginal increase in latency (0.50% to 0.72% across different percentiles).

• Overall assessment:

  • Z-score normalization provides a modest improvement in search quality with a negligible impact on latency.
  • The choice between z-score and min-max normalization may depend on your use case. If search relevance is a priority, z-score normalization may be the better option, especially if your system can tolerate a small latency increase.

Benchmark experiments were conducted using an OpenSearch cluster consisting of a single r6g.8xlarge instance as the coordinator node along with 3 r6g.8xlarge instances as data nodes with 12 shards.

The following table compares NDCG@mean scores for hybrid search using min-max normalization and z-score normalization, showing the percent difference in search relevance across five datasets.

The following table compares precision at mean (P@mean) scores for hybrid search using min-max normalization and z-score normalization, showing the percent difference in search relevance across four datasets.

Our production experiments highlight the following aspects of z-score normalization compared to min-max normalization in hybrid search approaches:

Search quality:

• Z-score normalization shows a slight decline in search quality, with an average decrease of 0.53% in NDCG@mean scores.
• P@mean results show a similar trend, with a 0.59% average decrease across datasets.
• Only the FIQA dataset demonstrated consistent improvements (+0.48% NDCG@mean, +0.43% P@mean).
• The largest decline was observed in the Touche-2020 dataset (-1.82% NDCG@mean, -1.95% P@mean).

Trade-offs:

• Production metrics indicate consistent but minor degradation in search relevance.
• Dataset size appears to influence performance, with larger datasets showing more pronounced decreases.
• The FIQA dataset demonstrates that z-score normalization can outperform min-max normalization in specific scenarios.
• Performance impact varies significantly across different datasets.

Overall assessment:

• Z-score normalization performs marginally below min-max normalization in production settings.
• Results suggest dataset characteristics significantly influence normalization effectiveness.
• Implementation decisions should prioritize specific use case requirements and dataset characteristics.

These findings indicate that while z-score normalization remains a viable option, min-max normalization demonstrates more reliable performance in production scenarios. The choice between methods should be based on thorough testing with specific datasets and use cases.

We’re continuing to enhance OpenSearch’s hybrid search capabilities beyond z-score normalization by implementing custom normalization functions. These will enable you to define your own normalization logic and allow fine-tuning of search result rankings. This enhancement will provide you more control over search result ranking while ensuring reliable and consistent hybrid search outcomes. For more information, see this issue.

1. Normalization
2. Hybrid Search 2.0: The Pursuit of Better Search
3. [RFC] Z-Score Normalization Technique for Normalization Processor