Link Search Menu Expand Document Documentation Menu

Metrics reference

Performance Analyzer provides a number of metrics to help you evaluate performance. The following tables describe the available metrics, grouped by the dimensions that are most relevant for that metric. All metrics support the avg, sum, min, and max aggregations, although for certain metrics, the measured value is the same regardless of aggregation type.

For information about each of the dimensions, see dimensions reference later in this topic.

This list is extensive. We recommend using Ctrl/Cmd + F to find what you’re looking for.

Relevant dimensions: ShardID, IndexName, Operation, ShardRole

Metric Description
CPU_Utilization CPU usage ratio. CPU time (in milliseconds) used by the associated thread(s) in the past five seconds, divided by 5000 milliseconds.
Paging_MajfltRate The number of major faults per second in the past five seconds. A major fault requires the process to load a memory page from disk.
Paging_MinfltRate The number of minor faults per second in the past five seconds. A minor fault does not requires the process to load a memory page from disk.
Paging_RSS The number of pages the process has in real memory---the pages that count towards text, data, or stack space. This number does not include pages that have not been demand-loaded in or swapped out.
Sched_Runtime Time (seconds) spent executing on the CPU per context switch.
Sched_Waittime Time (seconds) spent waiting on a run queue per context switch.
Sched_CtxRate Number of times run on the CPU per second in the past five seconds.
Heap_AllocRate An approximation, in bytes, of the heap memory allocated per second in the last 5 seconds.
IO_ReadThroughput Number of bytes read per second in the last five seconds.
IO_WriteThroughput Number of bytes written per second in the last five seconds.
IO_TotThroughput Number of bytes read or written per second in the last five seconds.
IO_ReadSyscallRate Read system calls per second in the last five seconds.
IO_WriteSyscallRate Write system calls per second in the last five seconds.
IO_TotalSyscallRate Read and write system calls per second in the last five seconds.
Thread_Blocked_Time The average amount of time, in seconds, that the associated thread has been blocked from entering or reentering a monitor.
Thread_Blocked_Event The total number of times that the associated thread has been blocked from entering or reentering a monitor (that is, the number of times a thread has been in the `blocked` state).
Thread_Waited_Time The average amount of time, in seconds, that the associated thread has waited to enter or reenter a monitor (that is, the amount of time a thread has been in the `WAITING` or `TIMED_WAITING` state)".
Thread_Waited_Event The total number of times that the associated thread has waited to enter or reenter a monitor (that is, the number of times a thread has been in the WAITING or TIMED_WAITING state).
ShardEvents The total number of events executed on a shard in the past five seconds.
ShardBulkDocs The total number of documents indexed in the past five seconds.

Relevant dimensions: ShardID, IndexName

Metric Description
Indexing_ThrottleTime Time (milliseconds) that the index has been under merge throttling control in the past five seconds.
Cache_Query_Hit The number of successful lookups in the query cache in the past five seconds.
Cache_Query_Miss The number of lookups in the query cache that failed to retrieve a `DocIdSet` in the past five seconds. `DocIdSet` is a set of document IDs in Lucene.
Cache_Query_Size Query cache memory size in bytes.
Cache_FieldData_Eviction The number of times OpenSearch has evicted data from the fielddata heap space (occurs when the heap space is full) in the past five seconds.
Cache_FieldData_Size Fielddata memory size in bytes.
Cache_Request_Hit The number of successful lookups in the shard request cache in the past five seconds.
Cache_Request_Miss The number of lookups in the request cache that failed to retrieve the results of search requests in the past five seconds.
Cache_Request_Eviction The number of times OpenSearch evicts data from shard request cache (occurs when the request cache is full) in the past five seconds.
Cache_Request_Size Shard request cache memory size in bytes.

Relevant dimensions: ShardID, IndexName, IndexingStage

Metric Description
Indexing_Pressure_Current_Limits The total heap size, in bytes, that is available for use by an index shard in a particular indexing stage (Coordinating, Primary, or Replica).
Indexing_Pressure_Current_Bytes The total heap size, in bytes, occupied by an index shard in a particular indexing stage (Coordinating, Primary, or Replica).
Indexing_Pressure_Last_Successful_Timestamp The timestamp of a successful request for an index shard in a particular indexing stage (Coordinating, Primary, or Replica).
Indexing_Pressure_Rejection_Count The total number of rejections performed by OpenSearch for an index shard in a particular indexing stage (Coordinating, Primary, or Replica).
Indexing_Pressure_Average_Window_Throughput The average throughput of the last n requests (The value of n is determined by the `shard_indexing_pressure.secondary_parameter.throughput.request_size_window` setting) for an index shard in a particular indexing stage (Coordinating, Primary, or Replica).

Relevant dimensions: Operation, Exception, Indices, HTTPRespCode, ShardID, IndexName, ShardRole

Metric Description
Latency Latency (milliseconds) of a request.

Relevant dimension: MemType

Metric Description
GC_Collection_Event The number of garbage collections that have occurred in the past five seconds.
GC_Collection_Time The approximate accumulated time (milliseconds) of all garbage collections that have occurred in the past five seconds.
Heap_Committed The amount of memory (bytes) that is committed for the JVM to use.
Heap_Init The amount of memory (bytes) that the JVM initially requests from the operating system for memory management.
Heap_Max The maximum amount of memory (bytes) that can be used for memory management.
Heap_Used The amount of used memory in bytes.

Relevant dimension: DiskName

Metric Description
Disk_Utilization Disk utilization rate: percentage of disk time spent reading and writing by the OpenSearch process in the past five seconds.
Disk_WaitTime Average duration (milliseconds) of read and write operations in the past five seconds.
Disk_ServiceRate Service rate: MB read or written per second in the past five seconds. This metric assumes that each disk sector stores 512 bytes.

Relevant dimension: DestAddr

Metric Description
Net_TCP_NumFlows The number of samples collected. Performance Analyzer collects 1 sample every 5 seconds.
Net_TCP_TxQ The average number of TCP packets in the send buffer.
Net_TCP_RxQ The average number of TCP packets in the receive buffer.
Net_TCP_Lost The average number of unrecovered recurring timeouts. This number is reset when the recovery finishes or `SND.UNA` is advanced. `SND.UNA` is the sequence number of the first byte of data that has been sent but not yet acknowledged.
Net_TCP_SendCWND The average size, in bytes, of the sending congestion window.
Net_TCP_SSThresh The average size, in bytes, of the slow start size threshold.

Relevant dimension: Direction

Metric Description
Net_PacketRate4 The total number of IPv4 datagrams transmitted/received from/by interfaces per second, including those transmitted or received in error.
Net_PacketDropRate4 The total number of IPv4 datagrams transmitted or received in error per second.
Net_PacketRate6 The total number of IPv6 datagrams transmitted or received from or by interfaces per second, including those transmitted or received in error.
Net_PacketDropRate6 The total number of IPv6 datagrams transmitted or received in error per second.
Net_Throughput The number of bits transmitted or received per second by all network interfaces.

Relevant dimension: ThreadPoolType

Metric Description
ThreadPool_QueueSize The size of the task queue.
ThreadPool_RejectedReqs The number of rejected executions.
ThreadPool_TotalThreads The current number of threads in the pool.
ThreadPool_ActiveThreads The approximate number of threads that are actively executing tasks.
ThreadPool_QueueLatency The latency of the task queue.
ThreadPool_QueueCapacity The current capacity of the task queue.

Relevant dimension: ClusterManager_PendingTaskType

Metric Description
ClusterManager_PendingQueueSize The current number of pending tasks in the cluster state update thread. Each node has a cluster state update thread that submits cluster state update tasks, such as create index, update mapping, allocate shard, and fail shard.

Relevant dimensions: Operation, Exception, Indices, HTTPRespCode

Metric Description
HTTP_RequestDocs The number of items in the request (only for the `_bulk` request type).
HTTP_TotalRequests The number of requests completed in the last 5 seconds.

Relevant dimension: CBType

Metric Description
CB_EstimatedSize The current number of estimated bytes.
CB_TrippedEvents The number of times that the circuit breaker has tripped.
CB_ConfiguredSize The limit, in bytes, of the amount of memory operations can use.

Relevant dimensions: ClusterManagerTaskInsertOrder, ClusterManagerTaskPriority, ClusterManagerTaskType, ClusterManagerTaskMetadata

Metric Description
ClusterManager_Task_Queue_Time The amount of time, in milliseconds, that a cluster manager task spent in the queue.
ClusterManager_Task_Run_Time The amount of time, in milliseconds, that a cluster manager task has been running.

Relevant dimension: CacheType

Metric Description
Cache_MaxSize The maximum size of the cache, in bytes.

Relevant dimension: ControllerName

Metric Description
AdmissionControl_RejectionCount The total number of rejections performed by a Controller of Admission Control.
AdmissionControl_CurrentValue The current value for Controller of Admission Control.
AdmissionControl_ThresholdValue The threshold value for Controller of Admission Control.

Relevant dimension: NodeID

Metric Description
Data_RetryingPendingTasksCount The number of throttled pending tasks on which the data node is actively performing retries. It is an absolute metric at that point in time.
ClusterManager_ThrottledPendingTasksCount The sum of the total pending tasks that were throttled by the cluster manager node. This is a cumulative metric, so make sure to check the max aggregation.

Relevant dimensions: N/A

The following metrics are relevant to the cluster as a whole and do not require specific dimensions.

Metric Description
Election_Term A number that increases monotonically with every cluster manager election.
PublishClusterState_Latency The amount of time taken by the quorum of nodes to publish the new cluster state. This metric is available for the current cluster manager.
PublishClusterState_Failure The number of times the new cluster state failed to publish on the cluster manager node.
ClusterApplierService_Latency The amount of time taken by each node for the apply cluster state sent by the cluster manager.
ClusterApplierService_Failure The number of times that the apply cluster state action failed on each node.

Relevant dimensions: IndexName, NodeName, ShardType, ShardID

Metric Description
Shard_State The state of each shard, for example, `STARTED`, `UNASSIGNED`, or `RELOCATING`.

Relevant dimensions: NodeID, searchbp_mode

Metric Description
SearchBP_Shard_Stats_CancellationCount The number of tasks marked for cancellation at the shard task level.
SearchBP_Shard_Stats_LimitReachedCount The number of times that the cancellable task total exceeded the set cancellation threshold at the shard task level.
SearchBP_Shard_Stats_Resource_Heap_Usage_CancellationCount The number of tasks marked for cancellation because of excessive heap usage since the node last restarted at the shard task level.
SearchBP_Shard_Stats_Resource_Heap_Usage_CurrentMax The maximum heap usage for tasks currently running at the shard task level.
SearchBP_Shard_Stats_Resource_Heap_Usage_RollingAvg The rolling average heap usage for the _n_ most recent tasks at the shard task level. The default value for _n_ is `100`.
SearchBP_Shard_Stats_Resource_CPU_Usage_CancellationCount The number of tasks marked for cancellation because of excessive CPU usage since the node last restarted at the shard task level.
SearchBP_Shard_Stats_Resource_CPU_Usage_CurrentMax The maximum CPU time for all tasks currently running on the node at the shard task level.
SearchBP_Shard_Stats_Resource_CPU_Usage_CurrentAvg The average CPU time for all tasks currently running on the node at the shard task level.
SearchBP_Shard_Stats_Resource_ElaspedTime_Usage_CancellationCount The number of tasks marked for cancellation because of excessive time elapsed since the node last restarted at the shard task level.
SearchBP_Shard_Stats_Resource_ElaspedTime_Usage_CurrentMax The maximum time elapsed for all tasks currently running on the node at the shard task level.
SearchBP_Shard_Stats_Resource_ElaspedTime_Usage_CurrentAvg The average time elapsed for all tasks currently running on the node at the shard task level.
Searchbp_Task_Stats_CancellationCount The number of tasks marked for cancellation at the search task level.
SearchBP_Task_Stats_LimitReachedCount The number of times that the cancellable task total exceeded the set cancellation threshold at the search task level.
SearchBP_Task_Stats_Resource_Heap_Usage_CancellationCount The number of tasks marked for cancellation because of excessive heap usage since the node last restarted at the search task level.
SearchBP_Task_Stats_Resource_Heap_Usage_CurrentMax The maximum heap usage for tasks currently running at the search task level.
SearchBP_Task_Stats_Resource_Heap_Usage_RollingAvg The rolling average heap usage for the _n_ most recent tasks at the search task level. The default value for _n_ is `10`.
SearchBP_Task_Stats_Resource_CPU_Usage_CancellationCount The number of tasks marked for cancellation because of excessive CPU usage since the node last restarted at the search task level.
SearchBP_Task_Stats_Resource_CPU_Usage_CurrentMax The maximum CPU time for all tasks currently running on the node at the search task level.
SearchBP_Task_Stats_Resource_CPU_Usage_CurrentAvg The average CPU time for all tasks currently running on the node at the search task level.
SearchBP_Task_Stats_Resource_ElaspedTime_Usage_CancellationCount The number of tasks marked for cancellation because of excessive time elapsed since the node last restarted at the search task level.
SearchBP_Task_Stats_Resource_ElaspedTime_Usage_CurrentMax The maximum time elapsed for all tasks currently running on the node at the search task level.
SearchBP_Task_Stats_Resource_ElaspedTime_Usage_CurrentAvg The average time elapsed for all tasks currently running on the node at the search task level.

Dimensions reference

Dimension Return values
ShardID The ID of the shard, for example, 1.
IndexName The name of the index, for example, my-index.
Operation The type of operation, for example, shardbulk.
ShardRole The shard role, for example, primary or replica.
Exception OpenSearch exceptions, for example, org.opensearch.index_not_found_exception.
Indices The list of indexes in the request URL.
HTTPRespCode The OpenSearch response code, for example, 200.
MemType The memory type, for example, totYoungGC, totFullGC, Survivor, PermGen, OldGen, Eden, NonHeap, or Heap.
DiskName The name of the disk, for example, sda1.
DestAddr The destination address, for example, 010015AC.
Direction The direction, for example, in or out.
ThreadPoolType The OpenSearch thread pools, for example, index, search, or snapshot.
CBType The circuit breaker type, for example, accounting, fielddata, in_flight_requests, parent, or request.
ClusterManagerTaskInsertOrder The order in which the task was inserted, for example, 3691.
ClusterManagerTaskPriority The priority of the task, for example, URGENT. OpenSearch executes higher-priority tasks before lower-priority ones, regardless of insert_order.
ClusterManagerTaskType The task type, for example, shard-started, create-index, delete-index, refresh-mapping, put-mapping, CleanupSnapshotRestoreState, or Update snapshot state.
ClusterManagerTaskMetadata The metadata for the task (if any).
CacheType The cache type, for example, Field_Data_Cache, Shard_Request_Cache, or Node_Query_Cache.
NodeID The ID of the node.
Searchbp_mode The search backpressure mode, for example, monitor_only (default), enforced, or disabled.