OpenSearch 2.0 is now available!

Thu, May 26, 2022 · Charlotte Henkle, Nick Knize, James McIntyre, Mohammad Qureshi, Sean Zheng

OpenSearch 2.0 is now generally available! This release incorporates user feedback and contributions from across the OpenSearch community to deliver a wealth of new capabilities and performance enhancements. We’re grateful for the collaborative effort of the community to build a distributed search and analytics toolset with the features, usability, and open-source flexibility that developers can rely on to create their most innovative solutions yet.

Here’s a look at some of the new features and enhancements you can benefit from with OpenSearch 2.0. You’ll find the full release notes here.

Lucene 9.1

With the 2.0 release, OpenSearch has upgraded to Lucene 9.1 (Lucene 9.1 documentation). The move to the latest version of Lucene affords a number of exciting advancements for this release and will continue to offer more value in future releases. For 2.0, this upgrade enables the following enhancements:

Performance optimizations delivered with Lucene 9.1 include 10–15% faster indexing of multi-dimensional points and sorting on fields indexed with points that is now several times faster. Lucene 9’s ConcurrentMergeScheduler now assumes fast I/O, which likely improves indexing speed in cases where heuristics would incorrectly detect whether or not the system had modern I/O. Lucene 9 also changed all file formats from big-endian order to little-endian order, speeding up decoding of postings lists.
Java Jigsaw module support means that with version 9.1, Lucene JARs are now proper Java modules, with module descriptors and dependency information. This aligns with the continuing evolution of OpenSearch to make the toolset more modular and extensible.

Document-level Alerting

Document-level alerting (see GitHub issue) allows users to create monitors that can generate alerts per document. Commonly used in security detection, these monitors use a similar approach as other types of alerting monitors available in OpenSearch: query level and bucket level. While those alerts use a summarized view of the data, document-level alerting can issue alerts on each document in the index, spotlighting which specific documents or records are triggering an alert in the monitor and avoiding monitoring gaps or data overlaps based on timestamps.

Image: Document-level alerting

Notifications

A new Notifications plugin (see GitHub issue) adds a unified notifications system to OpenSearch. Users no longer need to configure and manage notification channels for each plugin independently; with version 2.0, the Notifications plugin provides a centralized location to set up and manage notifications for relevant OpenSearch plugins. For example, in addition to managing notifications for the Alerting plugin, this plugin can generate a notification when a scheduled action is completed in the Index State Management (ISM) plugin.

Image: Notifications plugin

ML Commons Upgrades

Introduced with version 1.3, the ML Commons plugin (see GitHub repo gains two new algorithms in version 2.0 to extend OpenSearch’s machine learning (ML) functionality to additional workloads, reduce the effort required to build ML features, and centralize computation, resource management, and security for ML processes. New algorithms for linear regression and localization join existing algorithms for kmeans and Random Cut Forests to provide a comprehensive foundation for building and training ML models. The addition of linear regression aims to simplify development of ML models for predictive analysis; with localization, users can get a head start on developing ML approaches that reveal the key contributors to anomalies or any events that are detected, facilitating analyses and visualizations for root cause analysis and other use cases.

Replacing Non-inclusive Terminology

This release replaces non-inclusive terminology (such as master, blacklist) throughout OpenSearch with inclusive terminology (such as cluster manager, allowlist). (See issue in GitHub).

RPM Package Manager

Version 2.0 follows version 1.3.2 to include the availability of RPM Package Manager distribution (see GitHub issue). This simplifies installation of the OpenSearch distribution for Red Hat Linux-based operating systems. You can view compatible Linux versions here.

Breaking Changes and Continuing Support

OpenSearch follows Semantic Versioning, or SemVer, so breaking changes are only included in major version releases, like this one. For version 2.0, the list of incompatible changes includes updates like the Lucene upgrades and inclusive terminology mentioned above as well as breaking changes in the Destination API that follow from the addition of the Notifications plugin, along with others. For a comprehensive list of breaking changes in version 2.0, refer to the documentation.

Please note that we are continuing to support the OpenSearch 1.x release line. For more information about deprecation support, see section 3.28 of our FAQ.

Contributing to the Project

Your thoughts and contributions make a real impact on the OpenSearch project! From Lucene upgrades to new plugins to ML advancements, this release includes many valuable contributions from the OpenSearch community. We extend our deepest thanks to everyone who contributed to OpenSearch 2.0.

If you’re interested in learning more, have a specific question, or just want to offer your feedback, please visit OpenSearch.org, open an issue on GitHub for OpenSearch or OpenSearch Dashboards, or post in the forums. There are also regular Community Meetings that include updates and time for Q&A in every session.

For almost any type of contribution, opening an issue is the first step. If you’re eager to jump in, check out issues with the “help wanted” label.