http
The http
plugin accepts HTTP requests from clients. The following table describes options you can use to configure the http
source.
Option | Required | Type | Description |
---|---|---|---|
port | No | Integer | The port that the source is running on. Default value is 2021 . Valid options are between 0 and 65535 . |
health_check_service | No | Boolean | Enables the health check service on the /health endpoint on the defined port. Default value is false . |
unauthenticated_health_check | No | Boolean | Determines whether or not authentication is required on the health check endpoint. Data Prepper ignores this option if no authentication is defined. Default value is false . |
request_timeout | No | Integer | The request timeout, in milliseconds. Default value is 10000 . |
thread_count | No | Integer | The number of threads to keep in the ScheduledThreadPool. Default value is 200 . |
max_connection_count | No | Integer | The maximum allowed number of open connections. Default value is 500 . |
max_pending_requests | No | Integer | The maximum allowed number of tasks in the ScheduledThreadPool work queue. Default value is 1024 . |
max_request_length | No | ByteCount | The maximum number of bytes allowed in the payload of a single HTTP request. Default value is 10mb . |
authentication | No | Object | An authentication configuration. By default, this creates an unauthenticated server for the pipeline. This uses pluggable authentication for HTTPS. To use basic authentication define the http_basic plugin with a username and password . To provide customer authentication, use or create a plugin that implements ArmeriaHttpAuthenticationProvider. |
ssl | No | Boolean | Enables TLS/SSL. Default value is false. |
ssl_certificate_file | Conditionally | String | SSL certificate chain file path or Amazon Simple Storage Service (Amazon S3) path. Amazon S3 path example s3://<bucketName>/<path> . Required if ssl is set to true and use_acm_certificate_for_ssl is set to false. |
ssl_key_file | Conditionally | String | SSL key file path or Amazon S3 path. Amazon S3 path example s3://<bucketName>/<path> . Required if ssl is set to true and use_acm_certificate_for_ssl is set to false. |
use_acm_certificate_for_ssl | No | Boolean | Enables a TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default value is false. |
acm_certificate_arn | Conditionally | String | The ACM certificate Amazon Resource Name (ARN). The ACM certificate takes preference over Amazon S3 or a local file system certificate. Required if use_acm_certificate_for_ssl is set to true. |
acm_private_key_password | No | String | ACM private key password that decrypts the private key. If not provided, Data Prepper generates a random password. |
acm_certificate_timeout_millis | No | Integer | Timeout, in milliseconds, that ACM takes to get certificates. Default value is 120000. |
aws_region | Conditionally | String | AWS region used by ACM or Amazon S3. Required if use_acm_certificate_for_ssl is set to true or ssl_certificate_file and ssl_key_file is the Amazon S3 path. |
Ingestion
Clients should send HTTP POST
requests to the endpoint /log/ingest
.
The http
protocol only supports the JSON UTF-8 codec for incoming requests, for example, [{"key1": "value1"}, {"key2": "value2"}]
.
Example: Ingest data with cURL
The following cURL command can be used to ingest data:
curl "http://localhost:2021/log/ingest" --data '[{"key1": "value1"}, {"key2": "value2"}]'
Metrics
The http
source includes the following metrics.
Counters
requestsReceived
: Measures the total number of requests received by the/log/ingest
endpoint.requestsRejected
: Measures the total number of requests rejected (429 response status code) by the HTTP Source plugin.successRequests
: Measures the total number of requests successfully processed (200 response status code) the by HTTP Source plugin.badRequests
: Measures the total number of requests with either an invalid content type or format processed by the HTTP Source plugin (400 response status code).requestTimeouts
: Measures the total number of requests that time out in the HTTP source server (415 response status code).requestsTooLarge
: Measures the total number of requests where the size of the event is larger than the buffer capacity (413 response status code).internalServerError
: Measures the total number of requests processed by the HTTP Source with a custom exception type (500 response status code).
Timers
requestProcessDuration
: Measures the latency of requests processed by the HTTP Source plugin in seconds.
Distribution summaries
payloadSize
: Measures the incoming request payload size in bytes.