Link Search Menu Expand Document Documentation Menu

test_procedures

If your workload only defines one benchmarking scenario, specify the schedule at the top level. Use the test-procedures element to specify additional properties, such as a name or description. A test procedure is like a benchmarking scenario. If you have multiple test procedures, you can define a variety of challenges.

The following table lists test procedures for the benchmarking scenarios in this dataset. A test procedure can reference all operations that are defined in the operations section.

Parameter Required Type Description
name Yes String The name of the test procedure. When naming the test procedure, do not use spaces; this ensures that the name can be easily entered on the command line.
description No String Describes the test procedure in a human-readable format.
user-info No String Outputs a message at the start of the test to notify you about important test-related information, for example, deprecations.
default No Boolean When set to true, selects the default test procedure if you did not specify a test procedure on the command line. If the workload only defines one test procedure, it is implicitly selected as the default. Otherwise, you must define "default": true on exactly one challenge.
schedule Yes Array Defines the order in which workload tasks are run.

schedule

The schedule element contains a list of a tasks, which are operations supported by OpenSearch Benchmark, that are run by the workload during the benchmark test.

Usage

The schedule element defines tasks using the methods described in this section.

Using the operations element

The following example defines a force-merge and match-all query task using the operations element. The force-merge operation does not use any parameters, so only the name and operation-type are needed. The match-all-query parameter requires a query body and operation-type.

Operations defined in the operations element can be reused in the schedule more than once:

{
  "operations": [
    {
      "name": "force-merge",
      "operation-type": "force-merge"
    },
    {
      "name": "match-all-query",
      "operation-type": "search",
      "body": {
        "query": {
          "match_all": {}
        }
      }
    }
  ],
  "schedule": [
    {
      "operation": "force-merge",
      "clients": 1
    },
    {
      "operation": "match-all-query",
      "clients": 4,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

Defining operations inline

If you don’t want to reuse an operation in the schedule, you can define operations inside the schedule element, as shown in the following example:

{
  "schedule": [
    {
      "operation": {
        "name": "force-merge",
        "operation-type": "force-merge"
      },
      "clients": 1
    },
    {
      "operation": {
        "name": "match-all-query",
        "operation-type": "search",
        "body": {
          "query": {
            "match_all": {}
          }
        }
      },
      "clients": 4,
      "warmup-iterations": 1000,
      "iterations": 1000,
      "target-throughput": 100
    }
  ]
}

Task options

Each task contains the following options.

Parameter Required Type Description
operation Yes List Either refers to the name of an operation, defined in the operations element, or includes the entire operation inline.
name No String Specifies a unique name for the task when multiple tasks use the same operation.
tags No String Unique identifiers that can be used to filter between tasks.clients or the number of clients that should execute a task concurrently. Default is 1.
clients No Integer Specifies the number of clients that will run the task concurrently. Default is 1.

Target options

OpenSearch Benchmark requires one of the following options when running a task:

target-throughput No Integer Defines the benchmark mode. When not defined, OpenSearch Benchmark assumes that it is a throughput benchmark and runs the task as fast as possible. This is useful for batch operations, where achieving better throughput is preferred over better latency. When defined, the target specifies the number of requests per second across all clients. For example, if you specify target-throughput: 1000 with 8 clients, each client issues 125 (= 1000 / 8) requests per second.
target-interval No Interval Defines an interval of 1 divided by the target-throughput (in seconds) when the target-throughput is less than 1 operation per second. Define either target-throughput or target-interval but not both, otherwise OpenSearch Benchmark raises an error.
ignore-response-error-level No Boolean Controls whether to ignore errors encountered during the task when a benchmark is run with the on-error=abort command flag.

Iteration-based options

Iteration-based options determine the number of times that an operation should run. It can also define the number of iterative runs when tasks are run in parallel. To configure an iteration-based schedule, use the following options.

Parameter Required Type Description
iterations No Integer Specifies the number of times that a client should execute an operation. All iterations are included in the measured results. Default is 1.
warmup-iterations No Integer Specifies the number of times that a client should execute an operation in order to warm up the benchmark candidate. The warmup-iterations do not appear in the measurement results. Default is 0.

Parallel tasks

The parallel element concurrently runs tasks wrapped inside the element.

When running tasks in parallel, each task requires the client option in order to ensure that clients inside your benchmark are reserved for that task. Otherwise, when the client option is specified inside the parallel element without a connection to the task, the benchmark uses that number of clients for all tasks.

Usage

In the following example, parallel-task-1 and parallel-task-2 execute a bulk operation concurrently:

{
  "name": "parallel-any",
  "description": "Workload completed-by property",
  "schedule": [
    {
      "parallel": {
        "tasks": [
          {
            "name": "parellel-task-1",
            "operation": {
              "operation-type": "bulk",
              "bulk-size": 1000
            },
            "clients": 8
          },
          {
            "name": "parellel-task-2",
            "operation": {
              "operation-type": "bulk",
              "bulk-size": 500
            },
            "clients": 8
          }
        ]
      }
    }
  ]
}

Options

The parallel element supports all schedule parameters, in addition to the following options.

tasks Yes Array Defines a list of tasks that should be executed concurrently.
completed-by No String Allows you to define the name of one task in the task list or the value any. If completed-by is set to the name of one task in the list, the parallel-task structure is considered to be complete once that specific task has been completed. If completed-by is set to any, the parallel-task structure is considered to be complete when any one of the tasks in the list has been completed. If completed-by is not explicitly defined, the parallel-task structure is considered to be complete as soon as all of the tasks in the list have been completed.

Time-based options

Time-based options determine the duration of time, in seconds, for which operations should run. This is ideal for batch-style operations, which may require an additional warmup period.

To configure a time-based schedule, use the following options.

Parameter Required Type Description
time-period No Integer Specifies the time period, in seconds, that OpenSearch Benchmark considers for measurement. This is not required for bulk indexing because OpenSearch Benchmark bulk indexes all documents and naturally measures all samples after the specified warmup-time-period.
ramp-up-time-period No Integer Specifies the time period, in seconds, during which OpenSearch Benchmark gradually adds clients and reaches the total number of clients specified for the operation.
warmup-time-period No Integer Specifies the amount of time, in seconds, to warm up the benchmark candidate. None of the response data captured during the warmup period appears in the measurement results.
350 characters left

Have a question? .

Want to contribute? or .