Date index name processor
The date_index_name
processor is used to point documents to the correct time-based index based on the date or timestamp field within the document. The processor sets the _index
metadata field to a date math index name expression. Then the processor fetches the date or timestamp from the field
field in the document being processed and formats it into a date math index name expression. The extracted date, index_name_prefix
value, and date_rounding
value are then combined to create the date math index expression. For example, if the field
field contains the value 2023-10-30T12:43:29.000Z
and index_name_prefix
is set to week_index-
and date_rounding
is set to w
, then the date math index name expression is week_index-2023-10-30
. You can use the date_formats
field to specify how the date in the date math index expression should be formatted.
The following is the syntax for the date_index_name
processor:
{
"date_index_name": {
"field": "your_date_field or your_timestamp_field",
"date_rounding": "rounding_value"
}
}
Configuration parameters
The following table lists the required and optional parameters for the date_index_name
processor.
Parameter | Required/Optional | Description | |
---|---|---|---|
field | Required | The date or timestamp field in the incoming document. Supports template snippets. | |
date_rounding | Required | The rounded date format within the index name . Valid values are y (year), M (month), w (week), d (day), h (hour), m (minute), and s (second). | |
date_formats | Optional | An array of date formats used to parse the date or timestamp field. Valid options include a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. Default is yyyy-MM-dd'T'HH:mm:ss.SSSXX . | |
index_name_format | Optional | The date format. Default is yyyy-MM-dd . Supports template snippets. | |
index_name_prefix | Optional | The index name prefix to append before the date. Supports template snippets. | |
description | Optional | A brief description of the processor. | |
if | Optional | A condition for running this processor. | |
ignore_failure | Optional | If set to true , failures are ignored. Default is false . | |
locale | locale | Optional | The locale to use when parsing the month name and week day of the date. Default is ENGLISH . Supports template snippets. |
on_failure | Optional | A list of processors to run if the processor fails. | |
tag | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | |
timezone | Optional | The time zone to use when parsing the date. Default is UTC . |
Using the processor
Follow these steps to use the processor in a pipeline.
Step 1: Create a pipeline.
The following query creates a pipeline, named date-index-name1
, that uses the date_index_name
processor to index logs into monthly indexes:
PUT /_ingest/pipeline/date-index-name1
{
"description": "Create weekly index pipeline",
"processors": [
{
"date_index_name": {
"field": "date_field",
"index_name_prefix": "week_index-",
"date_rounding": "w",
"date_formats": ["YYYY-MM-DD"]
}
}
]
}
Step 2 (Optional): Test the pipeline.
It is recommended that you test your pipeline before you ingest documents.
To test the pipeline, run the following query:
POST _ingest/pipeline/date-index-name1/_simulate
{
"docs": [
{
"_index": "testindex1",
"_id": "1",
"_source": {
"date_field": "2023-10-30"
}
}
]
}
Response
The following example response confirms that the pipeline is working as expected:
{
"docs": [
{
"doc": {
"_index": "<week_index-{2023-10-01||/w{yyyy-MM-dd|UTC}}>",
"_id": "1",
"_source": {
"date_field": "2023-10-30"
},
"_ingest": {
"timestamp": "2023-11-13T18:23:10.408593092Z"
}
}
}
]
}
Step 3: Ingest a document.
The following query ingests a document into an index named testindex1
:
PUT testindex1/_doc/1?pipeline=date-index-name1
{
"date_field": "2023-10-30"
}
Response
The request indexes the document into the index week_index-2023-10-23
and will index all documents with a timestamp within that week into the same index because the pipeline rounds by week.
{
"_index": "week_index-2023-10-30",
"_id": "1",
"_version": 4,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 3,
"_primary_term": 1
}
Step 4 (Optional): Retrieve the document.
To retrieve the document, run the following query:
GET week_index-2023-10-30/_doc/1