Managing type mapping deprecation

This guide provides solutions for managing the deprecation of the type mapping functionality when migrating from Elasticsearch 6.x or earlier to OpenSearch.

Understanding type mapping deprecation

Elasticsearch indexes created prior to version 6.x may contain multiple types per index. This feature has been deprecated and removed in newer versions of Elasticsearch and OpenSearch. During migration, you may need to specify how to handle items that used type mappings. For more information, see Elasticsearch’s official documentation on the removal of mapping types.

Using the type mapping transformer

To address type mapping deprecation, use the TypeMappingsSanitizationTransformer. This transformer can modify data, including metadata, documents, and requests, so that the previously mapped data can be used in OpenSearch. To use the mapping transformer:

1. Navigate to the bootstrap box and open the cdk.context.json file with Vim.
2. Add or update the key reindexFromSnapshotExtraArgs to include --doc-transformer-config-file /shared-logs-output/transformation.json.
3. Add or update the key trafficReplayerExtraArgs to include --transformer-config-file /shared-logs-output/transformation.json.
4. Deploy Migration Assistant.
5. Navigate to the Migration Assistant console.
6. Create a file named /shared-logs-output/transformation.json.
7. Add your transformation configuration to the file. For configuration options, see Configuration options.
8. When running the metadata migration, run the configuration with the transformer using the command console metadata migrate --transformer-config-file /shared-logs-output/transformation.json.

Whenever the transformation configuration is updated, the backfill and replayer tools need to be stopped and restarted in order to apply the changes. Any previously migrated data and metadata may need to be cleared in order to avoid an inconsistent state.

Configuration options

The TypeMappingsSanitizationTransformer supports several strategies for managing type mappings:

1. Route different types to separate indexes: Split different types into their own indexes.
2. Merge all types into one index: Combine multiple types into a single index.
3. Drop specific types: Selectively migrate only specific types.
4. Keep the original structure: Maintain the same index name while conforming to the new type standards.

Type mapping transformer configuration schema

The type mapping transformer uses the following configuration options.

The following example JSON configuration provides a transformation schema:

Example configurations

The following example configurations show you how to use the transformer for different mapping type scenarios.

Route different types to separate indexes

If you have an index activity with types user and post that you want to split into separate indexes, use the following configuration:

[
  {
    "TypeMappingSanitizationTransformerProvider": {
      "staticMappings": {
        "activity": {
          "user": "new_users",
          "post": "new_posts"
        }
      },
      "sourceProperties": {
        "version": {
          "major": 6,
          "minor": 8
        }
      }
    }
  }
]
<label hidden class="copy-label">copy</label>

This transformer will perform the following:

• Route documents with type user to the new_users index.
• Route documents with type post to the new_posts index.

Merge all types into one index

To merge all types into one index, use the following configuration:

[
  {
    "TypeMappingSanitizationTransformerProvider": {
      "staticMappings": {
        "activity": {
          "user": "activity",
          "post": "activity"
        }
      },
      "sourceProperties": {
        "version": {
          "major": 6,
          "minor": 8
        }
      }
    }
  }
]

Drop specific types

To migrate only the user type within the activity index and drop all documents/requests with types not directly specified, use the following configuration:

[
  {
    "TypeMappingSanitizationTransformerProvider": {
      "staticMappings": {
        "activity": {
          "user": "users_only"
        }
      },
      "sourceProperties": {
        "version": {
          "major": 6,
          "minor": 8
        }
      }
    }
  }
]

This configuration only migrates documents of type user and ignores other document types in the activity index.

Keep the original structure

To migrate only specific types and keep the original structure, use the following configuration:

[
  {
    "TypeMappingSanitizationTransformerProvider": {
      "regexMappings": [
        {
          "sourceIndexPattern": "(.*)",
          "sourceTypePattern": ".*",
          "targetIndexPattern": "$1"
        }
      ],
      "sourceProperties": {
        "version": {
          "major": 6,
          "minor": 8
        }
      }
    }
  }
]

This is equivalent to the strategy of merging all types into one index but also uses a pattern-based routing strategy.

Combining multiple strategies

You can combine both static and regex-based mappings to manage different indexes or patterns in a single migration. For example, you might have one index that must use staticMappings and another that uses regexMappings to route all types by pattern.

For each document, request, or metadata item (processed individually for bulk requests), the following steps are performed:

1. The index is checked to determine whether it matches an entry in the static mappings.
   • If matched, the type is checked against the index component of the static mappings entry.
     • If the type matches, the mapping is applied, and the resulting index includes the value of the type key.
     • If the type doesn’t match, the request/document/metadata is dropped and not migrated.
2. If the index is not matched in the static mappings, the index-type combination is checked against each item in the regex mappings list, in order from first to last. If a match is found, the mapping is applied, the resulting index includes the value of the type key, and no further regex matching is performed.
3. Any request, document, or metadata that doesn’t match the preceding cases is dropped, and the documents they contain are not migrated.

The following example demonstrates how to combine static and regex-based mappings for different indexes:

[
  {
    "TypeMappingSanitizationTransformerProvider": {
      "staticMappings": {
        "activity": {
          "user": "users_activity",
          "post": "posts_activity"
        },
        "logs": {
          "error": "logs_error",
          "info": "logs_info"
        }
      },
      "regexMappings": [
        {
          "sourceIndexPattern": "orders.*",
          "sourceTypePattern": ".*",
          "targetIndexPattern": "all_orders"
        }
      ],
      "sourceProperties": {
        "version": {
          "major": 6,
          "minor": 8
        }
      }
    }
  }
]

Defaults

When the regexMappings key is missing from the transformation configuration, regexMappings will default to the following:

{
  "regexMappings": [
    {
      "sourceIndexPattern": "(.+)",
      "sourceTypePattern": "_doc",
      "targetIndexPattern": "$1"
    },
    {
      "sourceIndexPattern": "(.+)",
      "sourceTypePattern": "(.+)",
      "targetIndexPattern": "$1_$2"
    }
  ]
}

This has the effect of retaining the index name for indexes created in Elasticsearch 6.x or later while combining the type and index name for indexes created in Elasticsearch 5.x. If you want to retain the index name for indexes created in Elasticsearch 5.x, use the staticMappings option or override the type mappings using the regexMappings option.

Limitations

When using the transformer, remember the following limitations. When using the transformer, remember the following limitations.

Traffic Replayer

For the Traffic Replayer, only a subset of requests that include types is supported. These requests are listed in the following table.

Reindex-From_Shapshot

For Reindex-From-Snapshot, indexes created in Elasticsearch 6.x or later will use _doc as the type for all documents, even if a different type was specified in Elasticsearch 6.