Link Search Menu Expand Document Documentation Menu
Documentation

Running distributed loads

OpenSearch Benchmark loads always run on the same machine on which the benchmark was started. However, you can use multiple load drivers to generate additional benchmark testing loads, particularly for large clusters on multiple machines. This tutorial describes how to distribute benchmark loads across multiple machines in a single cluster.

System architecture

The following tutorial uses a three-node architecture; each node is generated in Amazon Elastic Compute Cloud (Amazon EC2):

  • Node 1: Node 1 acts as the coordinator node and enables distribution and communication between the other two nodes.
  • Node 2 and Node 3: The remaining nodes in the cluster are used to generate the load for the benchmark test.

OpenSearch Benchmark must be installed on all nodes. For installation instructions, see Installing OpenSearch Benchmark.

Make note of each node’s IP address. This tutorial uses the following IP addresses:

  • Node 1 – Coordinator node: 192.0.1.0
  • Node 2 – Worker node: 198.52.100.0
  • Node 3 – Worker node: 198.53.100.0

Step 1: Enable node communication

Make sure to enable communication for each node. In the AWS Management Console:

  1. Go to the EC2 host for the node.
  2. Select Security, and then select the security group associated with the node.
  3. Use Add inbound rules to open traffic to the node, based on the port range and traffic type of your cluster.

Step 2: Run daemon processes on each node

Start OpenSearch Benchmark on each node, using --node-ip to initialize OpenSearch Benchmark on the node itself and then --coordinator-ip to connect each node to the coordinator node.

For Node 1, the following command identifies the node as the coordinator node:

opensearch-benchmarkd start --node-ip=192.0.1.0 --coordinator-ip=192.0.1.0

The following commands enable Node 2 and Node 3 to listen to the coordinator node for load generation instructions.

Node 2

opensearch-benchmarkd start --node-ip=198.52.100.0 --coordinator-ip=192.0.1.0

Node 3

opensearch-benchmarkd start --node-ip=198.53.100.0 --coordinator-ip=192.0.1.0

With OpenSearch Benchmark running on all three nodes and the worker nodes set to listen to the coordinator node, you can now run the benchmark test.

Step 3: Run the benchmark test

On Node 1, run a benchmark test with the worker-ips set to the IP addresses for your worker nodes, as shown in the following example:

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=eventdata --worker-ips=198.52.100.0,198.53.100.0 --target-hosts=<DOMAIN_ENDPOINT> --client-options=<STANDARD_CLIENT_OPTIONS> --kill-running-processes

After the test completes, the logs generated by the test appear on your worker nodes.

350 characters left

Have a question? .

Want to contribute? or .