Open Distro for Elasticsearch 1.4.0 with K-nearest neighbor (k-NN) search support is now available

Tue, Feb 11, 2020 · Pavani Baddepudi
This post was imported from the Open Distro For Elasticsearch blog, a predecessor project of OpenSearch. Information reflected in this post may not be current or accurate.

We are happy to announce the release of Open Distro for Elasticsearch 1.4.0. Version 1.4.0 includes the upstream open source versions of Elasticsearch 7.4.2, Kibana 7.4.2 and the latest updates for the alerting, SQL, security, performance analyzer and Kibana plugins. We are also pleased to announce the general availability of Open Distro for Elasticsearch k-NN plugin, and installer for Windows deployments. We would like to thank the community for their contributions and support in testing out the new features. Here are the full release notes.

Download the latest packages

You can find Docker Hub images Open Distro for Elasticsearch 1.4.0 and Open Distro for Elasticsearch Kibana 1.4.0 on Docker Hub. Make sure your compose file specifies 1.4.0 or uses the latest tag. See our documentation on how to install Open Distro for Elasticsearch with RPMs and install Open Distro for Elasticsearch with Debian packages. With Windows exe supporting version 1.4.0, users can now easily install Elasticsearch and Kibana on Windows and run it as a service. You can find our Open Distro for Elasticsearch security plugin artifacts on Maven Central. You can download the latest versions of Open Distro for Elasticsearch’s PerfTop client, and Open Distro for Elasticsearch’s SQL JDBC driver.

Note: k-NN plugin is only available as part of docker image in this release.

New Features

k-NN plugin

Our new k-NN search plugin enables high scale, low latency nearest neighbor search on billions of documents across thousands of dimensions with the same ease as running any regular Elasticsearch query. Built using the Non-Metric Space Library (NMSLIB), this plugin can power use cases such as product recommendations, fraud detection, and image, video, and related document search. We have extended the Apache Lucene codec to introduce a new file format to store vector data. k-NN Search uses the standard Elasticsearch mapping and query syntax —to designate a field as a k-NN vector you can simply map it to the new k-NN field type provided by the k-NN plugin. k-NN functionality integrates seamlessly with other Elasticsearch features. This provides users the flexibility to use Elasticsearch’s extensive search features such as aggregations and filtering with k-NN to further improve the search results. Learn more at k-NN

Packages and features under development

Anomaly Detection

Machine learning based anomaly detection has been released as an alpha plugin and is under active development. The anomaly detection feature can handle large volumes of high dimensional data and detect outliers in real-time. The anomaly detection feature depends on the underlying Random Cut Forest (RCF) library, a proven algorithm for streaming use cases. RCF is also now open source and provides better visibility into the anomaly detection decision framework.

Performance Analyzer Root Cause Analysis (RCA) framework

The Performance Analyzer RCA is a framework that builds on the Performance Analyzer engine to support Root Cause Analysis (RCA) of performance and reliability problems in Elasticsearch clusters. This framework executes real time root cause analyses using Performance Analyzer metrics. You can now weigh in your feedback on the design proposal released recently.

Questions?

Please feel free to ask questions on the Open Distro for Elasticsearch community discussion forum.

Report a bug or request a feature

Please feel free to file a bug or an issue, request a feature, or propose new ideas to enhance Open Distro for Elasticsearch. To file bugs, raise a PR or propose a feature for a particular plugin, navigate to the specific plugin repository and add an issue. This will help us manage the content better.

Getting Started

Feel free to select an issue tagged as a “Good First Issue” in the plugin repos to start contributing to Open Distro for Elasticsearch. Read the Open Distro technical documentation on the project website to help you get started. We look forward to sharing new ideas, seek feedback and development Open Distro collaboratively with you!