Real-time Anomaly Detection is now available in Open Distro for Elasticsearch 1.7.0

We are excited to announce the general availability of real-time anomaly detection for streaming applications in this release. We would like to thank the community for their feedback on the preview release of the feature. The anomaly detection feature is built on RCF (Random Cut Forest), an unsupervised algorithm, that detects anomalies on live data and identifies issues as they evolve in real time. RCF is a proven algorithm built on years of academic and industry research. We are glad to announce the general availability of the open source RCF libraries for the greater benefit of our data science community.

One of our key considerations was to design the anomaly detection to be lightweight so there is no overhead on the system resources processing application workloads. The computation of anomaly models are distributed across the nodes in Elasticsearch cluster, which makes the implementation highly scalable, and not requiring dedicated machine learning nodes. For more deep dive into anomaly detection system design and RCF algorithm, we recommend these previously released blogs: Real-time Anomaly Detection in Open Distro for Elasticsearch and Random Cut Forests.

Our new Kibana user interface for anomaly detection makes it easy for users with no prior machine learning knowledge to take advantage of the feature. The rich visualizations make it easy for users to detect the data points that contributed to an anomaly. The plugin is integrated with Open Distro for Elasticsearch Alerting to notify users through various supported channels as the anomalies are detected.

Figure 1: The new Anomaly Detection dashboard

Figure 2: The Anomaly Detection results view