Improve OpenSearch cluster performance by separating search and indexing workloads

All OpenSearch indexes consist of shards, with each document in an index stored in a shard.

Traditionally, OpenSearch defines two shard types: primary shards and replica shards. The primary shard handles both indexing and search operations, while replica shards maintain a copy of the primary shard’s data to provide redundancy and serve search queries. You configure the number of shards at index creation time, and this number cannot be changed later without reindexing.

With the introduction of segment replication and remote storage in OpenSearch, this model has evolved. In segment replication, only the primary shard node performs indexing operations and writes segment files to a remote object store—such as Amazon Simple Storage Service (Amazon S3), Google Cloud Storage, or Azure Blob Storage. Replica shards then download the segment files from the object store in parallel, eliminating the need to replay indexing operations on each replica.

• Write replicas: Redundant copies of the primary shard. If a primary shard fails, a write replica can be promoted to primary to maintain write availability.
• Search replicas: Serve search queries exclusively and cannot be promoted to primaries.

For hardware separation between indexing and search workloads, we’ve introduced a new search node role. Primary shards and write replicas can be allocated to any node with the data role, while search replicas are allocated only to nodes with the search role. Nodes with the search role act as dedicated search-serving nodes.

The following diagram shows an OpenSearch cluster with two coordinator nodes, two data nodes, and two search nodes, forming an indexing fleet and a search fleet. The index includes one primary shard, one write replica, and two search replicas. The primary shard and write replica are assigned to the data nodes, while the search replicas are assigned to the search nodes.

Each component is configured to perform the following:

• The remote store stores segment files and transaction logs. All replicas—including the write replica and search replicas—download segments from the remote store.
• The primary shard writes both segments and transaction logs to the remote store.
• The write replica downloads segments after the primary completes writing.
• The search replicas continuously poll the remote store for new segments.

• Parallel and isolated processing: Improve throughput and predictability by isolating indexing and search.
• Independent scalability: Scale indexing and search independently by adding data or search nodes.
• Failure resilience: Indexing and search failures are isolated, improving availability.
• Cost efficiency and performance: Use specialized hardware—compute optimized for indexing and memory optimized for search.
• Tuning flexibility: Optimize performance settings like buffers and caches independently for indexing and search.
• Failure isolation: Indexing and search issues are separated, which helps in troubleshooting failures.
• Scalable design: Scale indexing and search independently of each other.

This significantly reduces storage and compute costs while maintaining full search capabilities, providing the following benefits:

• Scale down indexing capacity when writes are no longer needed.
• Free up disk and memory capacity by removing unnecessary write paths.
• Work smoothly with index lifecycle controls through _open/_close operations.
• Keep only search replicas active to reduce resource usage.
• Maintain green cluster health with active search paths.
• Auto-recover search replicas without manual intervention.
• Scale back up at any time to resume indexing.
• Scale search replicas based on search traffic during search-only mode.

OpenSearch maintains cluster health as GREEN when search-only mode is active despite disabling primaries and write replicas. This is because only search replicas are expected to be allocated, and they are all successfully assigned. This ensures operational stability and full search availability.

1. Adds an internal search-only block to the index
2. Scales down all primaries and write replicas
3. Keeps only search replicas active

To resume indexing, run the _scale operation with { "search_only": false }. OpenSearch restores the original index state. Even in search-only mode, cluster health remains GREEN because all expected search replicas are allocated.

The following chart compares indexing throughput across the three cluster configurations.

The following chart shows how separating workloads improves query latency.

The results show that Cluster 3 slightly outperforms the other clusters in simultaneous indexing and search while significantly reducing query latency and lowering costs.

Comparing Cluster 1 and Cluster 3

Comparing Cluster 2 and Cluster 3

Coordinator nodes currently handle both search and indexing requests. For full separation, consider routing indexing and search traffic to distinct coordinator node groups.