Link Search Menu Expand Document Documentation Menu

Nodes hot threads

The nodes hot threads endpoint provides information about busy JVM threads for selected cluster nodes. It provides a unique view of the of activity each node.

Example

GET /_nodes/hot_threads

Path and HTTP methods

GET /_nodes/hot_threads
GET /_nodes/{nodeId}/hot_threads

URL parameters

You can include the following URL parameters in your request. All parameters are optional.

Parameter Type Description
nodeId String A comma-separated list of node IDs to filter results. Supports node filters. Defaults to _all.
snapshots Integer Number of samples of thread stacktraces. Defaults to 10.
interval TimeValue Interval between consecutive samples. Defaults to 500ms.
threads Integer A number of top busiest threads to return information about. Defaults to 3.
ignore_idle_threads Boolean Don’t show threads that are in known-idle states, such as waiting on a socket select or pulling from an empty task queue. Defaults to true.
type String Supported thread types are cpu, wait, or block. Defaults to cpu.
timeout TimeValue A request timeout. Defaults to 30s.

Response

Unlike the majority of OpenSearch API responses, this response is in a text format.

It consists of one section per each cluster node included in the response.

Each section starts with a single line containing the following segments:

Line segment Description
:::  Line start (a distinct visual symbol).
{global-eu-35} Node name.
{uFPbKLDOTlOmdnwUlKW8sw} NodeId.
{OAM8OT5CQAyasWuIDeVyUA} EphemeralId.
{global-eu-35.local} Host name.
{[gdv2:a284:2acv:5fa6:0:3a2:7260:74cf]:9300} Host address.
{dimr} Node roles (d=data, i=ingest, m=cluster manager, r=remote cluster client).
{zone=west-a2, shard_indexing_pressure_enabled=true} Node attributes.

Then follows information about threads of selected type.

::: {global-eu-35}{uFPbKLDOTlOmdnwUlKW8sw}{OAM8OT5CQAyasWuIDeVyUA}{global-eu-35.local}{[gdv2:a284:2acv:5fa6:0:3a2:7260:74cf]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
   Hot threads at 2022-04-01T15:15:27.658Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
    0.1% (645micros out of 500ms) cpu usage by thread 'opensearch[global-eu-35][transport_worker][T#7]'
     4/10 snapshots sharing following 3 elements
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
::: {global-eu-62}{4knOxAdERlOB19zLQIT1bQ}{HJuZs2HiQ_-8Elj0Fvi_1g}{global-eu-62.local}{[gdv2:a284:2acv:5fa6:0:3a2:bba6:fe3f]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
   Hot threads at 2022-04-01T15:15:27.659Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
      
   18.7% (93.4ms out of 500ms) cpu usage by thread 'opensearch[global-eu-62][transport_worker][T#3]'
     6/10 snapshots sharing following 3 elements
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
::: {global-eu-44}{8WW3hrkcTwGvgah_L8D_jw}{Sok7spHISFyol0jFV6i0kw}{global-eu-44.local}{[gdv2:a284:2acv:5fa6:0:3a2:9120:e79e]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
   Hot threads at 2022-04-01T15:15:27.659Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   42.6% (212.7ms out of 500ms) cpu usage by thread 'opensearch[global-eu-44][write][T#5]'
     2/10 snapshots sharing following 43 elements
       java.base@11.0.14.1/sun.nio.ch.IOUtil.write1(Native Method)
       java.base@11.0.14.1/sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:254)
       io.netty.channel.nio.NioEventLoop.wakeup(NioEventLoop.java:787)
       io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:846)
       io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
       io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
       io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:796)
       io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
       io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
       io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
       org.opensearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:159)
       app//org.opensearch.transport.OutboundHan...

Required permissions

If you use the security plugin, make sure you set the following permissions: cluster:monitor/nodes/hot_threads.