trace peer forwarder
trace_peer_forwarder processor is used with peer forwarder to reduce by half the number of events forwarded in a Trace Analytics pipeline. In Trace Analytics, each event is typically duplicated when it is sent from
service-map-pipeline. When pipelines forward events, this causes the core peer forwarder to send multiple HTTP requests for the same event. You can use
trace peer forwarder to forward an event once through the
otel-trace-pipeline instead of
service-map-pipeline, which prevents unnecessary HTTP requests.
You should use
trace_peer_forwarder for Trace Analytics pipelines when you have multiple nodes.
To get started with
trace_peer_forwarder, first configure peer forwarder. Then create a
pipeline.yaml file and specify
trace peer forwarder as the processor. You can configure
peer forwarder in your
data-prepper-config.yaml file. For more detailed information, see Configuring Data Prepper.
See the following example
otel-trace-pipeline: delay: "100" source: otel_trace_source: processor: - trace_peer_forwarder: sink: - pipeline: name: "raw-pipeline" - pipeline: name: "service-map-pipeline" raw-pipeline: source: pipeline: name: "entry-pipeline" processor: - otel_trace_raw: sink: - opensearch: service-map-pipeline: delay: "100" source: pipeline: name: "entry-pipeline" processor: - service_map_stateful: sink: - opensearch:
In the preceding
pipeline.yaml file, events are forwarded in the
otel-trace-pipeline to the target peer, and no forwarding is performed in
service-map-pipeline. This process helps improve network performance by forwarding events (as HTTP requests) once instead of twice.