Configuring Data Prepper

You can customize your Data Prepper configuration by editing the data-prepper-config.yaml file in your Data Prepper installation. The following configuration options are independent from pipeline configuration options.

Data Prepper configuration

Use the following options to customize your Data Prepper configuration.

Peer forwarder options

The following section details various configuration options for peer forwarder.

General options for peer forwarding

TLS/SSL options for peer forwarder

Authentication options for peer forwarder

Circuit breakers

Data Prepper provides a circuit breaker to help prevent exhausting Java memory. And is useful when pipelines have stateful processors as these can retain memory usage outside of the buffers.

When a circuit breaker is tripped, Data Prepper rejects incoming data routing into buffers.

Heap circuit breaker

Configures Data Prepper to trip a circuit breaker when JVM heap reaches a specified usage threshold.

Extension plugins

Data Prepper provides support for user-configurable extension plugins. Extension plugins are common configurations shared across pipeline plugins, such as sources, buffers, processors, and sinks.

AWS extension plugins

To use the AWS extension plugin, add the following setting to your data-prepper-config.yaml under aws.

AWS secrets extension plugin

The AWS secrets extension plugin configures the AWS Secrets Manager to be referenced in pipeline plugin configurations, as shown in the following example:

extensions:
  aws:
    secrets:
      <YOUR_SECRET_CONFIG_ID_1>:
        secret_id: <YOUR_SECRET_ID_1>
        region: <YOUR_REGION_1>
        sts_role_arn: <YOUR_STS_ROLE_ARN_1>
        refresh_interval: <YOUR_REFRESH_INTERVAL>
        disable_refresh: false
      <YOUR_SECRET_CONFIG_ID_2>:
        ...

To use the secrets extension plugin, add the following setting to your pipeline.yaml under extensions > aws.

Secrets

Use the following settings under the secrets extension setting.

Reference secrets

ß In pipelines.yaml, secret values can be referenced within the pipeline plugins using the following formats:

• plaintext: ${{aws_secrets:<YOUR_SECRET_CONFIG_ID>}}.
• JSON (key-value pairs): ${{aws_secrets:<YOUR_SECRET_CONFIG_ID>:<YOUR_KEY>}}

Replace <YOUR_SECRET_CONFIG_ID> with the corresponding secret config ID under /extensions/aws/secrets. Replace <YOUR_KEY> with the desired key in the secret JSON value. The secret value reference string format can be interpreted for the following plugin setting data types:

• String
• Number
• Long
• Short
• Integer
• Double
• Float
• Boolean
• Character

The following example section of data-prepper-config.yaml names two secret config IDs, host-secret-config and credential-secret-config:

extensions:
  aws:
    secrets:
      host-secret-config:
        secret_id: <YOUR_SECRET_ID_1>
        region: <YOUR_REGION_1>
        sts_role_arn: <YOUR_STS_ROLE_ARN_1>
        refresh_interval: <YOUR_REFRESH_INTERVAL_1>
      credential-secret-config:
        secret_id: <YOUR_SECRET_ID_2>
        region: <YOUR_REGION_2>
        sts_role_arn: <YOUR_STS_ROLE_ARN_2>
        refresh_interval: <YOUR_REFRESH_INTERVAL_2>

After <YOUR_SECRET_CONFIG_ID> is configured, you can reference the IDs in your pipelines.yaml:

sink:
    - opensearch:
        hosts: [ "${{aws_secrets:host-secret-config}}" ]
        username: "${{aws_secrets:credential-secret-config:username}}"
        password: "${{aws_secrets:credential-secret-config:password}}"
        index: "test-migration"

Automatically refreshing secrets

For each individual secret configuration, the latest secret value is polled on a regular interval to support refreshing secrets in AWS Secrets Manager. The refreshed secret values are utilized by certain pipeline plugins to refresh their components, such as connection and authentication to the backend service.

For multiple secret configurations, jitter within 60s will be applied across all configurations during the initial secrets polling.