Link Search Menu Expand Document Documentation Menu
Documentation

Mutate event processors

Mutate event processors allow you to modify events in Data Prepper. The following processors are available:

  • AddEntries allows you to add entries to an event.
  • CopyValues allows you to copy values within an event.
  • DeleteEntry allows you to delete entries from an event.
  • RenameKey allows you to rename keys in an event.
  • ConvertEntry allows you to convert value types in an event.

AddEntries

The AddEntries processor adds entries to an event.

Configuration

You can configure the AddEntries processor with the following options.

Option Required Description
entries Yes A list of entries to add to an event.
key Yes The key of the new entry to be added. Some examples of keys include my_key, myKey, and object/sub_Key.
value Yes The value of the new entry to be added. You can use the following data types: strings, Booleans, numbers, null, nested objects, and arrays.
overwrite_if_key_exists No When set to true, the existing value is overwritten if key already exists in the event. The default value is false.

Usage

To get started, create the following pipeline.yaml file:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - add_entries:
        entries:
        - key: "newMessage"
          value: 3
          overwrite_if_key_exists: true
  sink:
    - stdout:

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring Data Prepper.

For example, before you run the AddEntries processor, if the logs_json.log file contains the following event record:

{"message": "hello"}

Then when you run the AddEntries processor using the previous configuration, it adds a new entry {"newMessage": 3} to the existing event {"message": "hello"} so that the new event contains two entries in the final output:

{"message": "hello", "newMessage": 3}

If newMessage already exists, its existing value is overwritten with a value of 3.

CopyValues

The CopyValues processor copies the values of an existing key within an event to another key.

Configuration

You can configure the CopyValues processor with the following options.

Option Required Description
entries Yes A list of entries to be copied in an event.
from_key Yes The key of the entry to be copied.
to_key Yes The key of the new entry to be added.
overwrite_if_key_exists No When set to true, the existing value is overwritten if key already exists in the event. The default value is false.

Usage

To get started, create the following pipeline.yaml file:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - copy_values:
        entries:
        - from_key: "message"
          to_key: "newMessage"
          overwrite_if_to_key_exists: true
  sink:
    - stdout:

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring Data Prepper.

For example, before you run the CopyValues processor, if the logs_json.log file contains the following event record:

{"message": "hello"}

When you run this processor, it parses the message into the following output:

{"message": "hello", "newMessage": "hello"}

If newMessage already exists, its existing value is overwritten with value.

DeleteEntry

The DeleteEntry processor deletes entries, such as key-value pairs, from an event. You can define the keys you want to delete in the with-keys field following delete_entries in the YAML configuration file. Those keys and their values are deleted.

Configuration

You can configure the DeleteEntry processor with the following options.

Option Required Description
with_keys Yes An array of keys for the entries to be deleted.

Usage

To get started, create the following pipeline.yaml file:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - delete_entries:
        with_keys: ["message"]
  sink:
    - stdout:

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring Data Prepper.

For example, before you run the DeleteEntry processor, if the logs_json.log file contains the following event record:

{"message": "hello", "message2": "goodbye"}

When you run the DeleteEntry processor, it parses the message into the following output:

{"message2": "goodbye"}

If message does not exist in the event, then no action occurs.

RenameKey

The RenameKey processor renames keys in an event.

Configuration

You can configure the RenameKey processor with the following options.

Option Required Description  
  entries Yes A list of event entries to rename.
  from_key Yes The key of the entry to be renamed.
  to_key Yes The new key of the entry.
  overwrite_if_to_key_exists No When set to true, the existing value is overwritten if key already exists in the event. The default value is false.

Usage

To get started, create the following pipeline.yaml file:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - rename_keys:
        entries:
        - from_key: "message"
          to_key: "newMessage"
          overwrite_if_to_key_exists: true
  sink:
    - stdout:

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring Data Prepper.

For example, before you run the RenameKey processor, if the logs_json.log file contains the following event record:

{"message": "hello"}

When you run the RenameKey processor, it parses the message into the following “newMessage” output:

{"newMessage": "hello"}

If newMessage already exists, its existing value is overwritten with value.

Special considerations

Renaming operations occur in the order that the key-value pair entries are listed in the pipeline.yaml file. This means that chaining (where key-value pairs are renamed in sequence) is implicit in the RenameKey processor. See the following example pipline.yaml file:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - rename_key:
        entries:
        - from_key: "message"
          to_key: "message2"
        - from_key: "message2"
          to_key: "message3"
  sink:
    - stdout:

Add the following contents to the logs_json.log file:

{"message": "hello"}

After the RenameKey processor runs, the following output appears:

{"message3": "hello"}

ConvertEntry

The ConvertEntry processor converts a value type associated with the specified key in a event to the specified type. It is a casting processor that changes the types of some fields in events. Some data must be converted to a different type, such as an integer to a double, or a string to an integer, so that it will pass the events through condition-based processors or perform conditional routing.

Configuration

You can configure the ConvertEntry processor with the following options.

Option Required Description
key Yes Keys whose value needs to be converted to a different type.
type No Target type for the key-value pair. Possible values are integer, double, string, and Boolean. Default value is integer.

Usage

To get started, create the following pipeline.yaml file:

type-conv-pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - convert_entry_type:
        key: "response_status"
        type: "integer"
  sink:
    - stdout:

Next, create a log file named logs_json.log and replace the path in the file source of your pipeline.yaml file with that filepath. For more information, see Configuring Data Prepper.

For example, before you run the ConvertEntry processor, if the logs_json.log file contains the following event record:

{"message": "value", "response_status":"200"}

The ConvertEntry processor converts the output received to the following output, where the type of response_status value changes from a string to an integer:

{"message":"value","response_status":200}
350 characters left

Have a question? .

Want to contribute? or .