Error log: “primary shard is not active” – The indexing block

Error log: You are trying to index a document (a POST or PUT request) and get this 503 Service Unavailable error.

JSON

None
{

  "error": {

    "root_cause": [

      {

        "type": "no_shard_available_action_exception",

        "reason": null,

        "index_uuid": "...",

        "shard": "0",

        "index": "my-new-index"

      }

    ],

    "type": "unavailable_shards_exception",

    "reason": "[my-new-index][0] primary shard is not active, 

                wait operation timed out",

    "index_uuid": "...",

    "shard": "0",

    "index": "my-new-index"

  },

  "status": 503

}

Why… is this happening? OpenSearch requires that the primary shard for a document be active and available before it can accept a write request. This error means that when OpenSearch tried to route your document, it found that the primary shard it needed to write to was not available.

Common causes:

1. Cluster is Red (most common): Your cluster has unassigned primary shards. This means no node in the cluster is holding the primary copy of that data, so all writes to it are blocked. This is why your cluster status is red.

2. Transient state: This error can happen in a few transient (temporary) situations:

• Index just created: You just sent the CREATE INDEX command, and your application tried to write to it in the same millisecond. The primary shards haven’t finished allocating to a node yet.
• Node just failed: The node holding the primary shard just failed. A replica on another node is about to be promoted to be the new primary, but there’s a small window where the primary is “not active.”

3. Allocation problems: The cluster is yellow or red because it can’t allocate the primary shard (see Blog #1), perhaps due to low disk space or allocation rules.

Best Practice:

1. Check cluster health: Your first command should always be GET /_cluster/health. If the status is red, you have unassigned primary shards. This is your root problem.

2. Use allocation explain API: If the cluster is red, find out why the shard is unassigned. Use the GET /_cluster/allocation/explain API. It will tell you the exact reason (e.g., “node has no disk space”).

3. Use wait_for_active_shards: When creating a new index, add this parameter to your request. This tells the CREATE INDEX command to not return “success” until the primary shards are ready.

Bash

None
PUT /my-new-index?wait_for_active_shards=1

4. Implement client-side retries: If this error is transient (due to a node failure or index creation), your client should be configured to retry the request with an exponential backoff.

What else can I do? If your cluster is stuck in a red state and you can’t figure out why, the _cluster/allocation/explain API is your best friend. Post the output of that API to the OpenSearch community forums, or contact us in The OpenSearch Slack Channel in #General.