translate
The translate
processor transforms values in events into preconfigured values.
Basic usage
To use the translate
processor, create the following pipeline.yaml
file:
translate-pipeline:
source:
file:
path: "/full/path/to/logs_json.log"
record_type: "event"
format: "json"
processor:
- translate:
mappings:
- source: "status"
targets:
- target: "translated_result"
map:
404: "Not Found"
sink:
- stdout:
Then create the following file named logs_json.log
and replace the path
in the file source of your pipeline.yaml
file with the path of a file containing the following JSON data:
{ "status": "404" }
The translate
processor configuration in pipeline.yaml
retrieves the source
value from the event data and compares it against the keys specified under the targets
. When a match is found, the processor places the corresponding mapped value into the target
key provided in the configuration.
When you run Data Prepper with the previous pipeline.yaml
file, you should receive the following output:
{
"status": "404",
"translated_result": "Not Found"
}
Advanced options
The following example shows a more involved mapping with additional configurations for the translate
processor:
processor:
- translate:
mappings:
- source: "status"
targets:
- target: "translated_result"
map:
404: "Not Found"
default: "default"
type: "string"
translate_when: "/response != null"
- target: "another_translated_result"
regex:
exact: false
patterns:
"2[0-9]{2}" : "Success" # Matches ranges from 200-299
"5[0-9]{2}": "Error" # Matches ranges form 500-599
file:
name: "path/to/file.yaml"
aws:
bucket: my_bucket
region: us-east-1
sts_role_arn: arn:aws:iam::123456789012:role/MyS3Role
On the top level, specify mappings
for inline mapping configurations, or file
pull mapping configurations from a file. Both mappings
and file
options can be specified together, and the processor considers the mappings from both sources for translations. In instances where the pipeline configuration and file mappings share duplicate source
and target
pairs, the mappings specified within the pipeline configuration take precedence.
Configuration
You can use the following options to configure the translate
processor.
Parameter | Required | Type | Description |
---|---|---|---|
mappings | No | List | Defines inline mappings. For more information, see mappings. |
file | No | Map | Points to the file that contains mapping configurations. For more information, see file. |
mappings
Each item in the mappings
configuration contains the following options.
Parameter | Required | Type | Description |
---|---|---|---|
source | Yes | String or list | The source field to translate. Can be a string or a list of strings. |
targets | Yes | List | A list of target field configurations, such as the target field key or translation maps. |
Each item in the targets
configuration contains the following options.
Parameter | Required | Type | Description |
---|---|---|---|
target | Yes | String | The key that specifies the field in the output in which the translated value will be placed. |
map | No | Map | A list of key-value pairs that define the translations. Each key represents a possible value in the source field, and the corresponding value represents what it should be translated to. For examples, see map option. At least one of map and regex should be configured. |
regex | No | Map | A map of keys that defines the translation map. For more options, see regex option. At least one of map and regex should be configured. |
default | No | String | The default value to use when no match is found during translation. |
type | No | String | Specifies the data type for the target value. |
translate_when | No | String | Uses a Data Prepper expression to specify a condition for performing the translation. When specified, the expression will only translate when the condition is met. |
map option
You can use the following key types when using the map option:
- Individual keys
map: ok : "Success" 120: "Found"
- Number ranges
map: "100-200": "Success" "400-499": "Error"
- Comma-delimited keys
map: "key1,key2,key3": "value1" "100-200,key4": "value2"
When configuring the keys inside the map
option, do not use any overlapping number ranges or duplicate keys.
regex option
You can use the following options with the regex
option.
Parameter | Required | Type | Description |
---|---|---|---|
patterns | Yes | Map | A map of key-value pairs defining the regex patterns of keys and the value to translate to for each pattern. |
exact | No | Boolean | Whether to use full string match or partial string match on the regex pattern. If true , the pattern is considered a match only when the entire key matches the pattern. Otherwise, the pattern is considered a match when a sub-string of the key matches the pattern. |
file
The file
option in the translate
processor takes a local YAML file or an Amazon Simple Storage Service (Amazon S3) object containing translation mappings. The file’s contents should be in the following format:
mappings:
- source: "status"
targets:
- target: "result"
map:
"foo": "bar"
# Other configurations
You can use the following options in the file
configuration.
Parameter | Required | Type | Description |
---|---|---|---|
name | Yes | String | The full path to a local file or key name for an S3 object. |
aws | No | Map | The AWS configuration when the file is an S3 object. See the following table for more information. |
You can use the following options with the aws
configuration.
Parameter | Required | Type | Description |
---|---|---|---|
bucket | Yes | String | The Amazon S3 bucket name. |
region | Yes | String | The AWS Region to use for credentials. |
sts_role_arn | Yes | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon S3. |