You're viewing version 2.13 of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated GitHub issue.
While the OpenSearch Stats APIs offer insight into the inner workings of each node and an OpenSearch cluster as a whole, the statistics lack certain details, such as percentiles, and do not provide the semantics of richer metric types, like histograms. Consequently, identifying outliers within cluster statistics becomes challenging when using only the Stats API.
The metrics framework feature adds comprehensive metrics support to effectively monitor an OpenSearch cluster. Using the Metrics Framework APIs, plugin, and extension, developers can add new monitoring metrics. In addition, the OpenSearch distribution bundles the telemetry-otel
plugin, which provides the implementation for metrics instrumentation based on the OpenTelemetry Java SDK.
Getting started
The metrics framework feature is experimental as of OpenSearch 2.11. To begin using the metrics framework feature, you need to first enable the telemetry feature
by using the opensearch.experimental.feature.telemetry.enabled
feature flag and subsequently by using the metrics framework feature flag.
Enabling this feature can consume system resources. Before enabling the metrics framework feature, determine whether you have sufficient cluster resources to allocate.
Enabling the feature flag on a node using tarball
The enable
flag is toggled using a Java Virtual Machine (JVM) parameter that is set either in OPENSEARCH_JAVA_OPTS
or in config/jvm.options
.
Option 1: Enable the experimental feature flag in the opensearch.yml
file
- Navigate to your OpenSearch directory using the following command:
cd \path\to\opensearch
- Open your
opensearch.yaml
file. - Add the following setting to
opensearch.yaml
:
opensearch.experimental.feature.telemetry.enabled=true
- Save your changes and close the file.
Option 2: Modify jvm.options
To enable the metrics framework feature using jvm
, add the following line to config/jvm.options
before starting OpenSearch:
-Dopensearch.experimental.feature.telemetry.enabled=true
Option 3: Enable from an environment variable
You can enable the metrics framework feature with a single command by adding the metrics framework environment variable to the OPENSEARCH_JAVA_OPTS
command, as shown in the following example:
OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.telemetry.enabled=true" ./opensearch-2.9.0/bin/opensearch
You can also define the environment variable separately before running OpenSearch by running the following command:
export OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.telemetry.enabled=true"
./bin/opensearch
Enable with Docker
If you’re running OpenSearch using Docker, add the following line to docker-compose.yml
under environment
:
OPENSEARCH_JAVA_OPTS="-Dopensearch.experimental.feature.telemetry.enabled=true"
Enable the metrics framework feature
Once you’ve enabled the feature flag, you can enable the metrics framework feature by using the following setting, which enables metrics in the opensearch.yaml
file:
telemetry.feature.metrics.enabled=true
The metrics framework feature supports various telemetry solutions through plugins. Use the following instructions to enable the telemetry-otel
plugin:
- Publish interval: The metrics framework feature can locally aggregate metrics with unique information about the configured publish interval and then export those metrics. By default, the interval is 1 minute. However, you can change the interval using the
telemetry.otel.metrics.publish.interval
cluster setting. - Exporters: Exporters are responsible for persisting the data. OpenTelemetry provides several out-of-the-box exporters. OpenSearch supports the following exporters:
LoggingMetricExporter
: Exports metrics to a log file, generating a separate file in the logs directory_otel_metrics.log
. Default istelemetry.otel.metrics.exporter.class=io.opentelemetry.exporter.logging.LoggingMetricExporter
.OtlpGrpcMetricExporter
: Exports spans through gRPC. To use this exporter, you need to install theotel-collector
on the node. By default, it writes to the http://localhost:4317/ endpoint. To use this exporter, set the following static setting:telemetry.otel.metrics.exporter.class=io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter
.
Supported metric types
The metrics framework feature supports the following metric types:
- Counters: Counters are continuous and synchronous meters used to track the frequency of events over time. Counters can only be incremented with positive values, making them ideal for measuring the number of monitoring occurrences such as errors, processed or received bytes, and total requests.
- UpDown counters: UpDown counters can be incremented with positive values or decremented with negative values. UpDown counters are well suited for tracking metrics like open connections, active requests, and other fluctuating quantities.
- Histograms: Histograms are valuable tools for visualizing the distribution of continuous data. Histograms offer insight into the central tendency, spread, skewness, and potential outliers that might exist in your metrics. Patterns such as normal distribution, skewed distribution, or bimodal distribution can be readily identified, making histograms ideal for analyzing latency metrics and assessing percentiles.
- Asynchronous Gauges: Asynchronous gauges capture the current value at the moment a metric is read. These metrics are non-additive and are commonly used to measure CPU utilization on a per-minute basis, memory utilization, and other real-time values.