Link Search Menu Expand Document Documentation Menu
Documentation

translate

The translate processor transforms values in events into preconfigured values.

Basic usage

To use the translate processor, create the following pipeline.yaml file:

translate-pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - translate:
        mappings:
          - source: "status"
            targets:
              - target: "translated_result"
                map:
                  404: "Not Found"
  sink:
    - stdout:

Then create the following file named logs_json.log and replace the path in the file source of your pipeline.yaml file with the path of a file containing the following JSON data:

{ "status": "404" }

The translate processor configuration in pipeline.yaml retrieves the source value from the event data and compares it against the keys specified under the targets. When a match is found, the processor places the corresponding mapped value into the target key provided in the configuration.

When you run Data Prepper with the previous pipeline.yaml file, you should receive the following output:

{
  "status": "404",
  "translated_result": "Not Found"
}

Advanced options

The following example shows a more involved mapping with additional configurations for the translate processor:

processor:
  - translate:
      mappings:
        - source: "status"
          targets:
            - target: "translated_result"
              map:
                404: "Not Found"
              default: "default"
              type: "string"
              translate_when: "/response != null"
            - target: "another_translated_result"
              regex:
                exact: false
                patterns:
                  "2[0-9]{2}" : "Success" # Matches ranges from 200-299
                  "5[0-9]{2}": "Error"    # Matches ranges form 500-599
      file: 
        name: "path/to/file.yaml"
        aws:
          bucket: my_bucket
          region: us-east-1
          sts_role_arn: arn:aws:iam::123456789012:role/MyS3Role

On the top level, specify mappings for inline mapping configurations, or file pull mapping configurations from a file. Both mappings and file options can be specified together, and the processor considers the mappings from both sources for translations. In instances where the pipeline configuration and file mappings share duplicate source and target pairs, the mappings specified within the pipeline configuration take precedence.

Configuration

You can use the following options to configure the translate processor.

Parameter Required Type Description
mappings No List Defines inline mappings. For more information, see mappings.
file No Map Points to the file that contains mapping configurations. For more information, see file.

mappings

Each item in the mappings configuration contains the following options.

Parameter Required Type Description
source Yes String or list The source field to translate. Can be a string or a list of strings.
targets Yes List A list of target field configurations, such as the target field key or translation maps.

Each item in the targets configuration contains the following options.

Parameter Required Type Description
target Yes String The key that specifies the field in the output in which the translated value will be placed.
map No Map A list of key-value pairs that define the translations. Each key represents a possible value in the source field, and the corresponding value represents what it should be translated to. For examples, see map option. At least one of map and regex should be configured.
regex No Map A map of keys that defines the translation map. For more options, see regex option. At least one of map and regex should be configured.
default No String The default value to use when no match is found during translation.
type No String Specifies the data type for the target value.
translate_when No String Uses a Data Prepper expression to specify a condition for performing the translation. When specified, the expression will only translate when the condition is met.

map option

You can use the following key types when using the map option:

  • Individual keys 
      map:
    ok : "Success"
    120: "Found"
  • Number ranges 
      map:
    "100-200": "Success"
    "400-499": "Error"
  • Comma-delimited keys 
      map:
    "key1,key2,key3": "value1"
    "100-200,key4": "value2"

When configuring the keys inside the map option, do not use any overlapping number ranges or duplicate keys.

regex option

You can use the following options with the regex option.

Parameter Required Type Description
patterns Yes Map A map of key-value pairs defining the regex patterns of keys and the value to translate to for each pattern.
exact No Boolean Whether to use full string match or partial string match on the regex pattern. If true, the pattern is considered a match only when the entire key matches the pattern. Otherwise, the pattern is considered a match when a sub-string of the key matches the pattern.

file

The file option in the translate processor takes a local YAML file or an Amazon Simple Storage Service (Amazon S3) object containing translation mappings. The file’s contents should be in the following format:

mappings:
  - source: "status"
    targets:
      - target: "result"
        map:
          "foo": "bar"
        # Other configurations

You can use the following options in the file configuration.

Parameter Required Type Description
name Yes String The full path to a local file or key name for an S3 object.
aws No Map The AWS configuration when the file is an S3 object. See the following table for more information.

You can use the following options with the aws configuration.

Parameter Required Type Description
bucket Yes String The Amazon S3 bucket name.
region Yes String The AWS Region to use for credentials.
sts_role_arn Yes String The AWS Security Token Service (AWS STS) role to assume for requests to Amazon S3.
350 characters left

Have a question? .

Want to contribute? or .