OpenSearch 2.13 is ready for download!

OpenSearch 2.13 is live and available for download! The latest version of OpenSearch arrives with a range of machine learning (ML) tools to help you build AI-powered applications, along with upgrades that make it easier to access and analyze your operational data and new ways to enhance the resiliency of your OpenSearch clusters. To explore the release for yourself, visit OpenSearch Playground. For a fuller view of what’s new, check out the release notes.

Automate configurations for AI-powered search applications

Introduced as experimental in the 2.12 release, the flow framework is now generally available and ready for production. You can use this functionality to automate the configuration of search and ingest pipeline resources required by advanced search features like semantic, multimodal, and conversational search. This adds to existing capabilities for automating ml-commons resource setup, allowing you to package OpenSearch AI solutions into portable templates. Additionally, we’ve added predefined templates to automate setup for models that are integrated through our connectors to APIs like OpenAI, Amazon Bedrock, and Cohere. We’ve also included templates to automate a variety of configurations for semantic, neural sparse, hybrid, and multimodal search. These templates simplify the multi-step configuration process by prescribing automation workflows that can be executed through a single API call using a higher-level declarative interface.

Build AI-powered workloads with updated connectors

This release includes enhancements to OpenSearch’s AI connectors, which let you integrate OpenSearch with AI/ML services like OpenAI, Cohere, and Amazon SageMaker to build AI-powered workloads like semantic search. We’ve added new settings to allow users to configure timeouts and automatically deploy models, making it easier and more convenient to manage your AI integrations.

Reduce memory footprint for FAISS engine indexes

With the 2.13 release, you’re now able to index quantized vectors with FAISS-engine-based k-NN indexes. Instead of storing vectors that require 4 bytes per dimension, you can compress the dimensions down to 2 bytes (half-precision floating-point). This enables builders to reduce the index footprint of their k-NN indexes, offering the potential to reduce memory footprint by as much as 50 percent with minimal impact to accuracy and latency. Previously, vector quantization was only supported on Lucene-based k-NN indexes; now, you can also realize these benefits when using high-performance and scalable FAISS engine indexes.

Build user experiences powered by generative AI

The OpenSearch Assistant Toolkit is now generally available after being introduced as an experimental feature in OpenSearch 2.12. This toolkit equips developers to build interactive, AI-powered assistant experiences in OpenSearch that let users query their operational and security log data using natural language. When paired with a large language model (LLM), these AI assistants let users apply generative AI to identify operational issues quickly, without needing to be familiar with the OpenSearch Piped Processing Language (PPL) or their data schema and fields. For new or less experienced team members, this eliminates the hurdle of learning a new query language, while more experienced team members can save time by eliminating the need to look up the data schema when building queries. Instead, users can simply ask a question about their data, such as “How many 503 errors occurred in the payment service in the last hour?”, and OpenSearch automatically generates and runs the corresponding PPL query.

Add guardrails for LLM output

OpenSearch provides the ability to integrate with LLMs to power generative AI use cases. LLMs can potentially generate harmful content, so for this release, the OpenSearch agent framework adds support for user-defined regex rules or word lists to filter inappropriate text generation that could be produced by integrated LLMs.

Apply aggregations to hybrid search results

Hybrid search combines results from lexical and neural search, providing more relevant results than either one separately. OpenSearch now offers the ability to post-filter the combined results and to apply aggregations to them to support use cases such as faceting.

Query and manage external data sources more efficiently

OpenSearch community members looking to optimize costs can find themselves storing infrequently queried data outside of OpenSearch in object stores. In the 2.9 release, we introduced data sources, which allow you to create a new Apache Spark data source type to directly query object stores. For those who wanted to increase query performance, we released skipping indexes (in the 2.9 release) to speed up direct queries as well as materialized views and covering indexes (both in the 2.11 release) to ingest data directly into OpenSearch indexes. In the 2.13 release, we’ve added a new skipping index type, Bloom filter, which is more efficient for data types like IP addresses and hostnames that have many different values that can be stored. In addition, we’ve made improvements to the data sources experience, where community members can now manage tables and accelerations visually instead of using SQL statements in Query Workbench. In upcoming releases, we will improve the querying experience as we merge observability logs functionality into Discover.

Contributors to this feature are eager for community feedback. Learn more and share your input in opensearch-spark, sql-plugin, or opensearch-dashboards.

Support resiliency with I/O-based admission control

OpenSearch incorporates features like search backpressure and shard-based indexing to help prevent a cluster from becoming overwhelmed by incoming requests. In addition to these reactive approaches to supporting cluster resiliency, OpenSearch 2.13 adds a proactive mechanism to protect a cluster from spikes or increases in capacity. With the addition of I/O-based admission control, if a cluster’s I/O usage breaches a defined threshold, OpenSearch will begin rejecting requests until I/O utilization falls below the threshold. This adds another layer of resiliency to the cluster, separate from JVM- and CPU-based admission control.

Scale alerting across clusters

OpenSearch community members who have large amounts of data may choose to store that data across clusters and nodes to meet requirements for scaling, compliance, or disaster recovery. Community members who store data across clusters can find it frustrating that they have to manage alerts for each cluster. In 2.13, we’ve introduced new cross-cluster monitors in the Alerting plugin. Cross-cluster monitors leverage OpenSearch’s cross-cluster search to execute alerting queries on remote clusters, making it easier to centrally manage your alerting infrastructure. If you want to scale alerting independently of indexing and querying, you can now set up a purpose-based alerting cluster that manages all alerting tasks. For more information on cross-cluster monitors, please review the alerting documentation, or create a GitHub issue if you would like to propose additional functionality.

Get started with OpenSearch 2.13

The latest version of OpenSearch is available for download here in a range of distributions. You can find out more about these capabilities and many other new features in the release notes, documentation release notes, and documentation, and OpenSearch Playground is kept current with the latest version so that you can explore the new OpenSearch Dashboards visualization tools. Please feel free to share your feedback on this release on our community forum!

OpenSearchCon Europe is coming soon!

The OpenSearch Project is excited to host our first-ever user conference in Europe on May 6 and 7 in Berlin. You’re invited to join us as community members from across the region come together for two days of learning and community building. Organizers have received an overwhelming response to the call for presentations, and we’re looking forward to sharing the conference program soon. Go here to reserve your seat and take advantage of early-bird pricing through April 8.