OpenSearch Kubernetes Operator 3.0 is now generally available

In February, the 3.0 alpha of the OpenSearch Kubernetes Operator was announced and the community was asked to try it in lower environments. Many did. Today, 3.0 is generally available and is the recommended version for production use. For anyone still running 2.8 or earlier, this post covers what has changed, how the release was tested, and how to upgrade.

Some background

The value of running OpenSearch on Kubernetes is that Kubernetes manages the operational overhead. It does, on day one. A StatefulSet, a couple of Services, a Secret with certificates, and there is a running cluster.

The trouble starts on day two. Kubernetes has no awareness of shards, quorum, or certificate lifecycles. A rolling upgrade has to preserve cluster manager quorum. A scale-down has to move shards off a node before the node goes away. An expired certificate has to be reissued, and nodes have to load it. A pod stuck in a not-ready state has to be detected and recovered. Bridging that gap is the purpose of an operator, and it is where the 2.x releases fell short: upgrades, restarts, scaling, and certificate rotation.

At a certain point, patching stopped making sense. The maintainers decided to rework the core reconcilers and, just as importantly, to build a test framework that could tell us whether the rework was actually correct. Since v2.8.0, the OpenSearch Project has merged over 200 pull requests (PRs) from more than 20 contributors. More than 100 of those PRs came after the alpha, and almost all of them are fixes for bugs found either by the new test harness or by people running pre-release builds.

How 3.0 was tested

The alpha blog post described a data integrity testing framework. That framework has since become a proper release gate, and it is the primary basis for the production-readiness designation in this release.

The harness builds the operator from source, installs it on a real Kubernetes cluster, and drives it through scenarios that simulate what happens to a cluster over a year of its life. Every step runs with an observer attached that samples cluster health, pod readiness, and per-index document counts every few seconds. The step fails if the cluster ever goes red, if more pods are down at once than a rolling operation is allowed to take, or if an index ever loses a single document. When the operation is done, the harness checks the operator’s own promises as well: the custom resource is back to RUNNING, no component status is left dangling, and shard allocation and node exclusions have been restored to their original state. The harness never cleans up after the operator. If the operator left allocation disabled, that’s a failed run.

There are close to 60 playbooks at the moment. They cover upgrades under live indexing, scaling during in-flight operations, chaos scenarios (pod kills, disk loss, operator restarts mid-operation), migrations from Operator 2.3 and 2.8, and a growing set of regression playbooks written from specific GitHub issues.

The testing workflow made heavy use of AI-assisted tooling. Turning a bug report into a playbook, reading 40 minutes of operator logs to determine why a restart stalled, checking whether a failure matches something already on the tracker, and writing up findings with a reproduction and a suggested fix are all tasks now performed with an AI coding assistant in the loop. Findings are checked against known issues and filed upstream. What never changes is the assertions. If a run fails, the operator gets fixed, or occasionally the harness does, but the bar does not move.

Between alpha and GA, this found close to 50 separate bugs. The following are among the most significant bugs identified before GA:

• The scale-down drain check failed open. An API error, or a shard that was still relocating, could get a node removed while it still had data on it.
• emptyDir recovery would tear down every StatefulSet on a transient readiness blip. On an emptyDir cluster that means the operator deleted all of your data.
• Generated TLS certificates expired after a year. Rotation was off by default, an expired certificate was never reissued, and when one was renewed the nodes didn’t load it.
• A rolling restart could delete a second pod while the pod the scaler had just removed was still terminating. Two members down at once.
• One sub-reconciler failing would freeze all maintenance on the cluster, and requeue signals were being dropped on the floor.
• Reverting an aborted upgrade left shard allocation set to primaries indefinitely, with the CR cheerfully reporting RUNNING.

All of these existed in 2.8, but have since been corrected with the launch of 3.0.

The harness ran in a lab, and a lab is not production. In parallel, several community members and commercial users ran pre-release 3.0 builds on their clusters, staging first and then production, including clusters migrated in place from Operator 2.x. They encountered issues the harness could not, and a significant portion of the post-alpha work came from their feedback.

What changed since the alpha

The major features were already in the alpha, which are documented in the alpha announcement: quorum-safe rolling restarts, SmartScaler on by default, multiple namespaces, TLS hot reload, PVC-backed bootstrap pods, init containers and sidecars, gRPC, and the move to the opensearch.org API group. The months since have been about making those features hold up when things go wrong.

Upgrades and rolling restarts received the most attention. The upgrade state machine now handles aborts, custom images, and crash loops. If a pod keeps getting killed while running but never ready, the operator detects the condition, flags it, and recovers the pod rather than waiting indefinitely. Restarts and upgrades proceed on a cluster that is permanently yellow. With drain mode on, they will refuse to take down a node that holds the only copy of a shard. Reverting the version is sufficient to recover from an upgrade to an image that cannot be pulled, and reverting restores shard allocation. Version changes are validated at admission time, so an unsupported jump or a downgrade is rejected before anything gets rendered into a StatefulSet.

Scaling is now built around never removing data. Drain checks fail closed. The scaler waits with timed requeues rather than falling into exponential backoff, and it coordinates with restarts and upgrades so that at most one member of the cluster is down at any time. Tiered data roles count as data nodes. Removing a node pool drains it one node at a time until it is gone.

Generated certificates are renewed a configurable number of days before expiry, expired or CA-mismatched ones are reissued, and nodes either hot-reload the new certificate or get rolled to pick it up. The operator communicates with OpenSearch using an mTLS client certificate rather than the admin password, its HTTP client has timeouts, and it verifies TLS by default.

Migration from 2.x has been rehearsed thoroughly. The migration controller was tested against clusters created by Operator 2.3.2 and 2.8.0 in a variety of configurations. That process surfaced a long list of fixes: finalizers, adopting the securityconfig job so that users and roles created through the REST API are not wiped, certificate secret ownership, PVC labels, and status on migrated resources.

There is also a lot of general hardening. One failing sub-reconciler no longer freezes everything else. StatefulSets use parallel pod management, so a new pool comes up all at once while restarts still go one pod at a time. Reconciler concurrency is configurable. The validation webhook now rejects unknown node pool roles, clusters without a cluster manager pool, ambiguous TLS configurations and storage class changes. And the operator emits warning events for a number of things it used to do silently, like removing a node without draining it, or a custom roles mapping that leaves the Dashboards user unmapped.

New capabilities that landed after the alpha: shard allocation awareness driven by Kubernetes node labels, per-node-pool image overrides, a PVC retention policy, running with the security plugin disabled, host network and hostPath support, per-node-pool ingress in the cluster chart, an OLM bundle with Red Hat OpenShift certification, and a long tail of dependency and CVE updates.

About 2.8

With 3.0 out, 2.x is end of life. There will be no more fixes to that line.

This matters because many clusters are still running 2.8 with the assumption that it is fine as long as nobody touches it. The bugs listed above are not exotic. They are what happens to a normal cluster over a normal year, and 2.8 responds to several of them by losing data or by getting stuck in a state that requires manual intervention to resolve. Any environment still running 2.x in production should plan the upgrade.

The upgrade is in-place: install the 3.0 operator over the old one, and the migration controller adopts each opensearch.opster.io/v1 resource and create its opensearch.org/v1 twin once the old one is healthy. Expect one rolling restart of the OpenSearch pods during adoption, since the StatefulSet selector changes. The Migration Guide covers the steps, how to verify the migration, and how to disable the legacy API once nothing depends on it.

Breaking changes

These are the same as in the alpha. Nothing new for GA, but they are worth reviewing before upgrading:

• SmartScaler is on by default, even if you don’t have a confMgmt block at all. Nodes are drained before they are removed. kube-rbac-proxy is gone. Authentication and authorization for the metrics endpoint now come from controller-runtime. setVMMaxMapCount defaults to true.
• Validation webhooks are active and will reject invalid manifests. The operator chart requires cert-manager for the webhook certificate, or operators can provide one themselves.
• TLS is on by default for transport and HTTP.
• There is no fixed default admin password. Operators must set one, or use the generated one.
• The API group is opensearch.org/v1. The old group is still served for the deprecation period, but new resources can only be created with the new one.

Where this fits

Everything running on top of OpenSearch on Kubernetes assumes a healthy cluster: OpenTelemetry Collectors, Data Prepper, Dashboards. The operator keeps that assumption true through upgrades, node failures, certificate rotation, and scaling. With that layer in place, multi-AZ, autoscaled, continuously upgraded clusters become the standard operating model.

Getting started

helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/
helm repo update
helm upgrade --install opensearch-operator opensearch-operator/opensearch-operator

cert-manager needs to be in the cluster first; the user guide explains why and what the alternatives are. The operator supports OpenSearch 2.19.2 through the latest 3.x.

For environments on 2.x, start with a non-production cluster, follow the migration guide, and move up from there. For new deployments, write manifests against opensearch.org/v1 from day one to avoid the migration path entirely.

The project welcomes continued feedback on GitHub, in the #k8s-operator channel on the OpenSearch Slack, and on the forum. Thank you to everyone who filed issues, sent pull requests, and ran the pre-releases.

Contributors

The 3.0 work was driven by the team at BigData Boutique, an OpenSearch Software Foundation member company and OpenSearch Accredited Long-Term Support provider. Two of the operator’s three active maintainers, Jose Barato and Itamar Syn-Hershko, are from BigData Boutique. Ryan Patterson and Lior Friedler contributed significantly to the reconciler rework, the test harness, and the validation work described above. Prudhvi Godithi also contributed as a co-maintainer throughout the release cycle.