Error log: ClusterBlockException – The “index read-only” Error

Error log: You’ll see this error when trying to write data (index, update) to your cluster:

org.opensearch.cluster.block.ClusterBlockException: 
  blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

You may also see this in your cluster health response or logs:

[WARN ][o.o.c.r.a.DiskThresholdMonitor] 
  flood stage disk watermark [95%] exceeded on [node-id] [...] 
  all indices on this node will be marked read-only

Why… is this happening? This is a critical, self-preservation mechanism. The ClusterBlockException with a FORBIDDEN/12 code almost always means one or more of your nodes has run out of disk space.

OpenSearch has disk “watermarks”:

• Low Watermark (default 85%): OpenSearch stops allocating new shards to this node.
• High Watermark (default 90%): OpenSearch will try to relocate shards off this node.
• Flood Stage Watermark (default 95%): This is the danger zone. To prevent data corruption from a completely full disk, OpenSearch blocks all write operations and marks the indices on that node as read-only.

Best practice:

• Check disk usage immediately: Run GET _cat/allocation?v&s=disk.percent:desc to see the disk usage on all your nodes, sorted by the fullest.
• Free up space (immediate fix): The fastest way to resolve this is to delete old, unneeded indices.
• Clear the block (required step): After you have freed up disk space, OpenSearch does not automatically remove the read-only block. You must clear it manually:
  Bash

PUT _cluster/settings
{
  "persistent": {
    "cluster.blocks.read_only_allow_delete": null
  }
}

Prevent it (Long-Term Fix):

• Set up Index State Management (ISM) policies to automatically delete or roll over indices after a certain time (e.g., delete logs older than 30 days).
• Add more data nodes to your cluster.
• Increase the disk size on your existing nodes.

What else can I do? Can’t free up disk space or struggling to clear the read-only block? The OpenSearch community can help you explore options. For direct support on cluster sizing and ISM policies, contact us at help@opensearchsoftwarefoundation.org.