Dead-letter queues
Data Prepper pipelines support dead-letter queues (DLQs) for offloading failed events and making them accessible for analysis.
As of Data Prepper 2.3, only the s3
source supports DLQs.
Configure a DLQ writer
To configure a DLQ writer for the s3
source, add the following to your pipeline.yaml file:
sink:
opensearch:
dlq:
s3:
bucket: "my-dlq-bucket"
key_path_prefix: "dlq-files/"
region: "us-west-2"
sts_role_arn: "arn:aws:iam::123456789012:role/dlq-role"
The resulting DLQ file outputs as a JSON array of DLQ objects. Any file written to the S3 DLQ contains the following name pattern:
dlq-v${version}-${pipelineName}-${pluginId}-${timestampIso8601}-${uniqueId}
The following information is replaced in the name pattern:
version
: The Data Prepper version.pipelineName
: The pipeline name indicated in pipeline.yaml.pluginId
: The ID of the plugin associated with the DLQ event.
Configuration
DLQ supports the following configuration options.
Option | Required | Type | Description |
---|---|---|---|
bucket | Yes | String | The name of the bucket into which the DLQ outputs failed records. |
key_path_prefix | No | String | The key_prefix used in the S3 bucket. Defaults to "" . Supports time value pattern variables, such as /%{yyyy}/%{MM}/%{dd} , including any variables listed in the Java DateTimeFormatter. For example, when using the /%{yyyy}/%{MM}/%{dd} pattern, you can set key_prefix as /2023/01/24 . |
region | No | String | The AWS Region of the S3 bucket. Defaults to us-east-1 . |
sts_role_arn | No | String | The STS role the DLQ assumes in order to write to an AWS S3 bucket. Default is null , which uses the standard SDK behavior for credentials. To use this option, the S3 bucket must have the S3:PutObject permission configured. |
When using DLQ with an OpenSearch sink, you can configure the max_retries option to send failed data to the DLQ when the sink reaches the maximum number of retries.
Metrics
DLQ supports the following metrics.
Counter
dlqS3RecordsSuccess
: Measures the number of successful records sent to S3.dlqS3RecordsFailed
: Measures the number of records that failed to be sent to S3.dlqS3RequestSuccess
: Measures the number of successful S3 requests.dlqS3RequestFailed
: Measures the number of failed S3 requests.
Distribution summary
dlqS3RequestSizeBytes
: Measures the distribution of the S3 request’s payload size in bytes.
Timer
dlqS3RequestLatency
: Measures latency when sending each S3 request, including retries.
DLQ objects
DLQ supports the following DLQ objects:
pluginId
: The ID of the plugin that originated the event sent to the DLQ.pluginName
: The name of the plugin.failedData
: An object that contains the failed object and its options. This object is unique to each plugin.pipelineName
: The name of the Data Prepper pipeline in which the event failed.timestamp
: The timestamp of the failures in anISO8601
format.