Persistent and performant ingestion using Data Prepper

Data Prepper is a last-mile data collector for ingestion into OpenSearch. It has a number of capabilities to help users reliably move their data into OpenSearch. We will discuss the Kafka buffer. This buffer uses Apache Kafka to persist data before writing to OpenSearch. You can use it to simplify your pipeline creation. We will also share how to use dead-letter queues to ensure your data is saved when OpenSearch has a transient error preventing ingestion. We will also discuss how you can use end-to-end acknowledgements for pull-based sources. Finally, we will share some of our future plans for Data Prepper and how they will improve on these solutions.

Details

Monday, May 6 2:25pm-2:45pm in Moskau

Track: Analytics, Observability, and Security

Speakers

David Venable photograph

David Venable

Senior Software Development Engineer at AWS - OpenSearch