You're viewing version 2.18 of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
Lowercase tokenizer
The lowercase tokenizer breaks text into terms at white space and then lowercases all the terms. Functionally, this is identical to configuring a letter tokenizer with a lowercase token filter. However, using a lowercase tokenizer is more efficient because the tokenizer actions are performed in a single step.
Example usage
The following example request creates a new index named my-lowercase-index and configures an analyzer with a lowercase tokenizer:
PUT /my-lowercase-index
{
"settings": {
"analysis": {
"tokenizer": {
"my_lowercase_tokenizer": {
"type": "lowercase"
}
},
"analyzer": {
"my_lowercase_analyzer": {
"type": "custom",
"tokenizer": "my_lowercase_tokenizer"
}
}
}
}
}
Generated tokens
Use the following request to examine the tokens generated using the analyzer:
POST /my-lowercase-index/_analyze
{
"analyzer": "my_lowercase_analyzer",
"text": "This is a Test. OpenSearch 123!"
}
The response contains the generated tokens:
{
"tokens": [
{
"token": "this",
"start_offset": 0,
"end_offset": 4,
"type": "word",
"position": 0
},
{
"token": "is",
"start_offset": 5,
"end_offset": 7,
"type": "word",
"position": 1
},
{
"token": "a",
"start_offset": 8,
"end_offset": 9,
"type": "word",
"position": 2
},
{
"token": "test",
"start_offset": 10,
"end_offset": 14,
"type": "word",
"position": 3
},
{
"token": "opensearch",
"start_offset": 16,
"end_offset": 26,
"type": "word",
"position": 4
}
]
}