Data Prepper pipelines support dead-letter queues (DLQs) for offloading failed events and making them accessible for analysis.
As of Data Prepper 2.3, only the
s3 source supports DLQs.
Configure a DLQ writer
To configure a DLQ writer for the
s3 source, add the following to your pipeline.yaml file:
sink: opensearch: dlq: s3: bucket: "my-dlq-bucket" key_path_prefix: "dlq-files/" region: "us-west-2" sts_role_arn: "arn:aws:iam::123456789012:role/dlq-role"
The resulting DLQ file outputs as a JSON array of DLQ objects. Any file written to the S3 DLQ contains the following name pattern:
The following information is replaced in the name pattern:
version: The Data Prepper version.
pipelineName: The pipeline name indicated in pipeline.yaml.
pluginId: The ID of the plugin associated with the DLQ event.
DLQ supports the following configuration options.
|bucket||Yes||String||The name of the bucket into which the DLQ outputs failed records.|
|region||No||String||The AWS Region of the S3 bucket. Defaults to |
|sts_role_arn||No||String||The STS role the DLQ assumes in order to write to an AWS S3 bucket. Default is |
When using DLQ with an OpenSearch sink, you can configure the max_retries option to send failed data to the DLQ when the sink reaches the maximum number of retries.
DLQ supports the following metrics.
dlqS3RecordsSuccess: Measures the number of successful records sent to S3.
dlqS3RecordsFailed: Measures the number of records that failed to be sent to S3.
dlqS3RequestSuccess: Measures the number of successful S3 requests.
dlqS3RequestFailed: Measures the number of failed S3 requests.
dlqS3RequestSizeBytes: Measures the distribution of the S3 request’s payload size in bytes.
dlqS3RequestLatency: Measures latency when sending each S3 request, including retries.
DLQ supports the following DLQ objects:
pluginId: The ID of the plugin that originated the event sent to the DLQ.
pluginName: The name of the plugin.
failedData: An object that contains the failed object and its options. This object is unique to each plugin.
pipelineName: The name of the Data Prepper pipeline in which the event failed.
timestamp: The timestamp of the failures in an