Deriving metrics from traces
You can use Data Prepper to derive metrics from OpenTelemetry traces. The following example pipeline receives incoming traces and extracts a metric called durationInNanos
, aggregated over a tumbling window of 30 seconds. It then derives a histogram from the incoming traces.
The pipeline contains the following pipelines:
-
entry-pipeline
– Receives trace data from the OpenTelemetry collector and forwards it to thetrace_to_metrics_pipeline
pipeline. -
trace-to-metrics-pipeline
- Receives the trace data from theentry-pipeline
pipeline, aggregates it, and derives a histogram ofdurationInNanos
from the traces based on the value of theserviceName
field. It then sends the derived metrics to the OpenSearch index calledmetrics_for_traces
.
entry-pipeline:
source:
otel_trace_source:
# Provide the path for ingestion. ${pipelineName} will be replaced with pipeline name.
# In this case it would be "/entry-pipeline/v1/traces". This will be endpoint URI path in OpenTelemetry Exporter configuration.
path: "/${pipelineName}/v1/traces"
sink:
- pipeline:
name: "trace-to-metrics-pipeline"
trace-to-metrics-pipeline:
source:
pipeline:
name: "entry-pipeline"
processor:
- aggregate:
# Pick the required identification keys
identification_keys: ["serviceName"]
action:
histogram:
# Pick the appropriate values for each of the following fields
key: "durationInNanos"
record_minmax: true
units: "seconds"
buckets: [0, 10000000, 50000000, 100000000]
# Specify an aggregation period
group_duration: "30s"
sink:
- opensearch:
...
index: "metrics_for_traces"