Announcing OpenSearch Data Prepper 2.11: Supercharged OpenTelemetry and even more sources

source:
  otel_logs_source:
      output_format: otel
sink:
  - opensearch:
        hosts: [ "https://..." ]
        aws:
            region: "<<region>>"
            sts_role_arn: "<<role-arn>>"
        index_type: "log-analytics-plain"

- add_entries:
       entries:
         - key: "severity/number"
           format: "${/severityNumber}"
         - key: "severity/text"
           format: "${/severityText}"
- delete_entries:
    with_keys: ["severityNumber", "severityText"]

version: "2"
extension:
    aws:
      secrets:
        jira-account-credentials:
          secret_id: <<secret-arn>>
          region: <<secrets-region>>
          sts_role_arn: <<role-to-access-secret>>
jira-pipeline:
  source:
    jira:
      hosts: ["<<Atlassian-host-url>>"]
      authentication: 
        basic:
          username: ${{aws_secrets:jira-account-credentials:jiraId}}
          password: ${{aws_secrets:jira-account-credentials:jiraCredential}}
      filter:
        project:
          key:
            include:
              - "<<project-key>>"
            exclude:
               - "<<project-key>>"
        issue_type:
          include: 
            - "Story"
            - "Epic"
            - "Task"
          exclude:
            - "Bug"
        status:
          include: 
            - "To Do"
            - "In Progress"
            - "Done"
          exclude:
            - "Closed"
  sink:
    - opensearch:

version: "2"
extension:
    aws:
      secrets:
        confluence-account-credentials:
          secret_id: <<secret-arn>>
          region: <<secrets-region>>
          sts_role_arn: <<role-to-access-secret>>
confluence-pipeline:
  source:
    confluence:
      hosts: ["<<Atlassian-host-url>>"]
      authentication: 
        basic:
          username: ${{aws_secrets:confluence-account-credentials:confluenceId}}
          password: ${{aws_secrets:confluence-account-credentials:confluenceCredential}}
      filter:
        space:
          key:
            include:
              - "<<space-key>>"
            exclude:
               - "<<space-key>>"
        page_type:
          include: 
            - "page"
            - "blogpost"
            - "comment"
          exclude:
            - "attachment"
  sink:
    - opensearch:

aurora-mysql-pipeline:
  source:
    rds:
      db_identifier: "my-aurora-cluster"
      engine: "aurora-mysql"
      database: "hr_db"
      tables:
        include:
          - "employees"
          - "departments"
s3_bucket: "my-s3-bucket"
s3_prefix: "pipeline-data"
export:
  kms_key_id: "1234abcd-1234-abcd-1234-123456abcdef"
  export_role_arn: "arn:aws:iam::123456789012:role/ExportRole"
stream: true
aws:
  sts_role_arn: "arn:aws:iam::123456789012:role/PipelineRole"
  region: "us-east-1"
authentication:
  username: ${{aws_secrets:secret:username}}
  password: ${{aws_secrets:secret:password}}

Data Prepper now supports a new Amazon Simple Queue Service (Amazon SQS) source for reading events from SQS queues. Amazon SQS is a fully managed message queue. The new sqs source in Data Prepper efficiently receives messages from Amazon SQS to create events that you can route to sinks.

Data Prepper receives SQS messages from an SQS queue in batches and then creates Data Prepper events from those SQS messages. By default, Data Prepper will create a single Data Prepper event per SQS message. Data Prepper provides a robust collection of processors that you can use to parse or grok, depending on the format of your message.

To help reduce your Amazon SQS costs, you can instead design your SQS data and configure Data Prepper to support multiple events per message. Using this approach reduces your Amazon SQS costs by combining data into SQS messages, allowing for fewer SQS sends and receives. To take this approach, you must design your sending application to send SQS messages to Amazon SQS in a format available as a Data Prepper codec. Then you can configure your Data Prepper pipeline to use that codec to parse the message into multiple events.

This release also includes several additional features to help you create pipelines:

• The rename_keys processor can now rename keys with variable names using a regex pattern.
• The opensearch sink now supports new index types for OTel logs and metrics.
• Data Prepper expressions now support slashes (/) in the key by allowing you to escape the / character.

Work is continuing on Data Prepper, with a few new features upcoming. One exciting feature is the integration of ML Commons and Amazon Bedrock into Data Prepper pipelines. This feature is provided through a new machine learning (ML) inference processor and the Amazon Simple Storage Service (Amazon S3) source and sink. The feature is currently in development and is available for experimental use in version 2.11.

See the Data Prepper Project Roadmap for more information on upcoming features.

Thanks to the following community members who contributed to this release!

• akshay0709 — Akshay Pawar
• chenqi0805 — Qi Chen
• dinujoh — Dinu John
• divbok — Divyansh Bokadia
• dlvenable — David Venable
• FedericoBrignola
• Galactus22625 — Maxwell Brown
• graytaylor0 — Taylor Gray
• janhoy — Jan Høydahl
• jmsusanto — Jeremy Michael
• juergen-walter — Jürgen Walter
• KarstenSchnitter — Karsten Schnitter
• kkondaka — Krishna Kondaka
• MohammedAghil — Mohammed Aghil Puthiyottil
• oeyh — Hai Yan
• RashmiRam — Rashmi
• Rishikesh1159 — Rishikesh
• saketh-pallempati — Saketh Pallempati
• san81 — Santhosh Gandhe
• sb2k16 — Souvik Bose
• seschis — Shane Schisler
• shenkw1 — Katherine Shen
• srikanthjg — Srikanth Govindarajan
• TomasLongo — Tomas
• Zhangxunmt — Xun Zhang