Data Prepper is a last-mile data collector for ingestion into OpenSearch. It has a number of capabilities to help users reliably move their data into OpenSearch. We will discuss the Kafka buffer. This buffer uses Apache Kafka to persist data before writing to OpenSearch. You can use it to simplify your pipeline creation. We will also share how to use dead-letter queues to ensure your data is saved when OpenSearch has a transient error preventing ingestion. We will also discuss how you can use end-to-end acknowledgements for pull-based sources. Finally, we will share some of our future plans for Data Prepper and how they will improve on these solutions.
