Link Search Menu Expand Document Documentation Menu
Documentation

list_to_map

The list_to_map processor converts a list of objects from an event, where each object contains a key field, into a map of target keys.

Configuration

The following table describes the configuration options used to generate target keys for the mappings.

Option Required Type Description
source Yes String The list of objects with key fields to be converted into keys for the generated map.
target No String The target for the generated map. When not specified, the generated map will be placed in the root node.
key Conditionally String The key of the fields to be extracted as keys in the generated mappings. Must be specified if use_source_key is false.
use_source_key No Boolean When true, keys in the generated map will use original keys from the source. Default is false.
value_key No String When specified, values given a value_key in objects contained in the source list will be extracted and converted into the value specified by this option based on the generated map. When not specified, objects contained in the source list retain their original value when mapped.
extract_value No Boolean When true, object values from the source list will be extracted and added to the generated map. When false, object values from the source list are added to the generated map as they appear in the source list. Default is false
flatten No Boolean When true, values in the generated map output flatten into single items based on the flattened_element. Otherwise, objects mapped to values from the generated map appear as lists.
flattened_element Conditionally String The element to keep, either first or last, when flatten is set to true.

Usage

The following example shows how to test the usage of the list_to_map processor before using the processor on your own source.

Create a source file named logs_json.log. Because the file source reads each line in the .log file as an event, the object list appears as one line even though it contains multiple objects:

{"mylist":[{"name":"a","value":"val-a"},{"name":"b","value":"val-b1"},{"name":"b",  "value":"val-b2"},{"name":"c","value":"val-c"}]}

Next, create a pipeline.yaml file that uses the logs_json.log file as the source by pointing to the .log file’s correct path:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        value_key: "value"
        flatten: true
  sink:
    - stdout:

Run the pipeline. If successful, the processor returns the generated map with objects mapped according to their value_key. Similar to the original source, which contains one line and therefore one event, the processor returns the following JSON as one line. For readability, the following example and all subsequent JSON examples have been adjusted to span multiple lines:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": "val-a",
  "b": "val-b1",
  "c": "val-c"
}

Example: Maps set to target

The following example pipeline.yaml file shows the list_to_map processor when set to a specified target, mymap:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        target: "mymap"
        value_key: "value"
        flatten: true
  sink:
    - stdout:

The generated map appears under the target key:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "mymap": {
    "a": "val-a",
    "b": "val-b1",
    "c": "val-c"
  }
}

Example: No value_key specified

The follow example pipeline.yaml file shows the list_to_map processor with no value_key specified. Because key is set to name, the processor extracts the object names to use as keys in the map.

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        flatten: true
  sink:
    - stdout:

The values from the generated map appear as original objects from the .log source, as shown in the following example response:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": {
    "name": "a",
    "value": "val-a"
  },
  "b": {
    "name": "b",
    "value": "val-b1"
  },
  "c": {
    "name": "c",
    "value": "val-c"
  }
}

Example: flattened_element set to last

The following example pipeline.yaml file sets the flattened_element to last, therefore flattening the processor output based on each value’s last element:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        target: "mymap"
        value_key: "value"
        flatten: true
        flattened_element: "last"
  sink:
    - stdout:

The processor maps object b to value val-b2 because val-b2 is the last element in object b, as shown in the following output:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": "val-a",
  "b": "val-b2",
  "c": "val-c"
}

Example: flatten set to false

The following example pipeline.yaml file sets flatten to false, causing the processor to output values from the generated map as a list:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        key: "name"
        source: "mylist"
        target: "mymap"
        value_key: "value"
        flatten: false
  sink:
    - stdout:

Some objects in the response may have more than one element in their values, as shown in the following response:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "a": [
    "val-a"
  ],
  "b": [
    "val-b1",
    "val-b2"
  ],
  "c": [
    "val-c"
  ]
}

Example: use_source_key and extract_value set to true

The following example pipeline.yaml file sets flatten to false, causing the processor to output values from the generated map as a list:

pipeline:
  source:
    file:
      path: "/full/path/to/logs_json.log"
      record_type: "event"
      format: "json"
  processor:
    - list_to_map:
        source: "mylist"
        use_source_key: true
        extract_value: true
  sink:
    - stdout:

Object values from mylist are extracted and added to fields with the source keys name and value, as shown in the following response:

{
  "mylist": [
    {
      "name": "a",
      "value": "val-a"
    },
    {
      "name": "b",
      "value": "val-b1"
    },
    {
      "name": "b",
      "value": "val-b2"
    },
    {
      "name": "c",
      "value": "val-c"
    }
  ],
  "name": ["a", "b", "b", "c"],
  "value": ["val-a", "val-b1", "val-b2", "val-c"]
}
350 characters left

Have a question? .

Want to contribute? or .