You're viewing version 2.16 of the OpenSearch documentation. This version is no longer maintained. For the latest version, see the current documentation. For information about OpenSearch version maintenance, see Release Schedule and Maintenance Policy.
Remote Store Stats API
Introduced 2.8
Use the Remote Store Stats API to monitor shard-level remote-backed storage performance.
Metrics returned from this API only relate to indexes stored on remote-backed nodes. For an aggregated output on an index at the node or cluster level, use the Index Stats, Nodes Stats, or Cluster Stats API.
Path and HTTP methods
GET _remotestore/stats/<index_name>
GET _remotestore/stats/<index_name>/<shard_id>
Path parameters
The following table lists the available path parameters. All path parameters are optional.
| Parameter | Type | Description |
|---|---|---|
index_name | String | The index name or index pattern. |
shard_id | String | The shard ID. |
Remote store stats for an index
Use the following API to get remote store statistics for all index shards.
Example request
GET _remotestore/stats/<index_name>
Example response
Response
{
"_shards": {
"total": 4,
"successful": 4,
"failed": 0
},
"indices": {
"remote-index": {
"shards": {
"0": [{
"routing": {
"state": "STARTED",
"primary": true,
"node": "q1VxWZnCTICrfRc2bRW3nw"
},
"segment": {
"download": {},
"upload": {
"local_refresh_timestamp_in_millis": 1694171634102,
"remote_refresh_timestamp_in_millis": 1694171634102,
"refresh_time_lag_in_millis": 0,
"refresh_lag": 0,
"bytes_lag": 0,
"backpressure_rejection_count": 0,
"consecutive_failure_count": 0,
"total_uploads": {
"started": 5,
"succeeded": 5,
"failed": 0
},
"total_upload_size": {
"started_bytes": 15342,
"succeeded_bytes": 15342,
"failed_bytes": 0
},
"remote_refresh_size_in_bytes": {
"last_successful": 0,
"moving_avg": 3068.4
},
"upload_speed_in_bytes_per_sec": {
"moving_avg": 99988.2
},
"remote_refresh_latency_in_millis": {
"moving_avg": 44.0
}
}
},
"translog": {
"upload": {
"last_successful_upload_timestamp": 1694171633644,
"total_uploads": {
"started": 6,
"failed": 0,
"succeeded": 6
},
"total_upload_size": {
"started_bytes": 1932,
"failed_bytes": 0,
"succeeded_bytes": 1932
},
"total_upload_time_in_millis": 21478,
"upload_size_in_bytes": {
"moving_avg": 322.0
},
"upload_speed_in_bytes_per_sec": {
"moving_avg": 2073.8333333333335
},
"upload_time_in_millis": {
"moving_avg": 3579.6666666666665
}
},
"download": {}
}
},
{
"routing": {
"state": "STARTED",
"primary": false,
"node": "EZuen5Y5Sv-eDCLwh9gv-Q"
},
"segment": {
"download": {
"last_sync_timestamp": 1694171634148,
"total_download_size": {
"started_bytes": 15112,
"succeeded_bytes": 15112,
"failed_bytes": 0
},
"download_size_in_bytes": {
"last_successful": 2910,
"moving_avg": 1259.3333333333333
},
"download_speed_in_bytes_per_sec": {
"moving_avg": 382387.3333333333
}
},
"upload": {}
},
"translog": {
"upload": {},
"download": {}
}
}
],
"1": [{
"routing": {
"state": "STARTED",
"primary": false,
"node": "q1VxWZnCTICrfRc2bRW3nw"
},
"segment": {
"download": {
"last_sync_timestamp": 1694171633181,
"total_download_size": {
"started_bytes": 18978,
"succeeded_bytes": 18978,
"failed_bytes": 0
},
"download_size_in_bytes": {
"last_successful": 325,
"moving_avg": 1265.2
},
"download_speed_in_bytes_per_sec": {
"moving_avg": 456047.6666666667
}
},
"upload": {}
},
"translog": {
"upload": {},
"download": {}
}
},
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "EZuen5Y5Sv-eDCLwh9gv-Q"
},
"segment": {
"download": {},
"upload": {
"local_refresh_timestamp_in_millis": 1694171633122,
"remote_refresh_timestamp_in_millis": 1694171633122,
"refresh_time_lag_in_millis": 0,
"refresh_lag": 0,
"bytes_lag": 0,
"backpressure_rejection_count": 0,
"consecutive_failure_count": 0,
"total_uploads": {
"started": 6,
"succeeded": 6,
"failed": 0
},
"total_upload_size": {
"started_bytes": 19208,
"succeeded_bytes": 19208,
"failed_bytes": 0
},
"remote_refresh_size_in_bytes": {
"last_successful": 0,
"moving_avg": 3201.3333333333335
},
"upload_speed_in_bytes_per_sec": {
"moving_avg": 109612.0
},
"remote_refresh_latency_in_millis": {
"moving_avg": 25.333333333333332
}
}
},
"translog": {
"upload": {
"last_successful_upload_timestamp": 1694171633106,
"total_uploads": {
"started": 7,
"failed": 0,
"succeeded": 7
},
"total_upload_size": {
"started_bytes": 2405,
"failed_bytes": 0,
"succeeded_bytes": 2405
},
"total_upload_time_in_millis": 27748,
"upload_size_in_bytes": {
"moving_avg": 343.57142857142856
},
"upload_speed_in_bytes_per_sec": {
"moving_avg": 1445.857142857143
},
"upload_time_in_millis": {
"moving_avg": 3964.0
}
},
"download": {}
}
}
]
}
}
}
}
Response fields
The response body of the Remote Store Stats API is split into three categories:
routing: Contains information related to the shard’s routingsegment: Contains statistics related to segment transfers from remote-backed storagetranslog: Contains statistics related to translog transfers from remote-backed storage
routing
The routing object contains the following fields.
| Field | Description |
|---|---|
primary | Denotes whether the shard copy is a primary shard. |
node | The name of the node to which the shard is assigned. |
segment
The segment.upload object contains the following fields.
| Field | Description |
|---|---|
local_refresh_timestamp_in_millis | The last successful local refresh timestamp, in milliseconds. |
remote_refresh_timestamp_in_millis | The last successful remote refresh timestamp, in milliseconds. |
refresh_time_lag_in_millis | The amount of time, in milliseconds, that the remote refresh is behind the local refresh. |
refresh_lag | The number of refreshes by which the remote store is lagging behind the local store. |
bytes_lag | The lag, in bytes, between the remote and local stores. |
backpressure_rejection_count | The total number of write rejections issued due to backpressure in the remote store. |
consecutive_failure_count | The number of consecutive remote refresh failures since the last successful refresh. |
total_remote_refresh | The total number of remote refreshes. |
total_uploads_in_bytes | The total number of bytes in all uploads to the remote store. |
remote_refresh_size_in_bytes.last_successful | The size of the data uploaded during the last successful refresh. |
remote_refresh_size_in_bytes.moving_avg | The average size of the data, in bytes, uploaded in the last N refreshes. N is defined in the remote_store.moving_average_window_size setting. For more information, see Remote segment backpressure. |
upload_latency_in_bytes_per_sec.moving_avg | The average speed of remote segment uploads, in bytes per second, for the last N uploads. N is defined in the remote_store.moving_average_window_size setting. For more information, see Remote segment backpressure. |
remote_refresh_latency_in_millis.moving_avg | The average amount of time, in milliseconds, taken by a single remote refresh during the last N remote refreshes. N is defined in the remote_store.moving_average_window_size setting. For more information, see Remote segment backpressure. |
The segment.download object contains the following fields.
| Field | Description |
|---|---|
last_sync_timestamp | The timestamp, in milliseconds, since the last successful segment file download from remote-backed storage. |
total_download_size.started_bytes | The total number of bytes of segment files actively being downloaded from remote-backed storage. |
total_download_size.succeeded_bytes | The total number of bytes of segment files successfully downloaded from remote-backed storage. |
total_download_size.failed_bytes | The total number of bytes of segment files that failed to download from remote-back storage. |
download_size_in_bytes.last_successful | The size, in bytes, of the last segment file successfully downloaded from remote-backed storage. |
download_size_in_bytes.moving_avg | The average size of segment data, in bytes, downloaded in the last 20 downloads. |
download_speed_in_bytes_per_sec.moving_avg | The average download speed, in bytes per second, of the last 20 downloads. |
translog
The translog.upload object contains the following fields.
| Field | Description |
|---|---|
last_successful_upload_timestamp | The timestamp, in milliseconds, since the last translog file successfully uploaded to remote-backed storage. |
total_uploads.started | The total number of attempted translog upload syncs to remote-backed storage. |
total_uploads.failed | The total number of failed translog upload syncs to remote-backed storage. |
total_uploads.succeeded | The total number of successful translog upload syncs to remote-backed storage. |
total_upload_size.started_bytes | The total number of bytes of translog files actively being downloaded from remote-backed storage. |
total_upload_size.succeeded_bytes | The total number of bytes of translog files successfully uploaded to remote-backed storage. |
total_upload_size.failed_bytes | The total number of bytes of translog files that failed to upload to remote-backed storage. |
total_upload_time_in_millis | The total amount of time spent, in milliseconds, uploading translog files to remote-backed storage. |
upload_size_in_bytes.moving_avg | The average size of translog data, in bytes, uploaded in the last N downloads. N is defined in the remote_store.moving_average_window_size setting. |
upload_speed_in_bytes_per_sec.moving_avg | The average speed of translog uploads, in bytes per second, for the last N uploads. N is defined in the remote_store.moving_average_window_size setting. |
upload_time_in_millis.moving_avg | The average amount of time taken by a single translog upload, in milliseconds, since the last N uploads. N is defined in the remote_store.moving_average_window_size setting. |
The translog.download object contains the following fields.
| Field | Description |
|---|---|
last_successful_download_timestamp | The timestamp, in milliseconds, since the last translog file successfully uploaded to remote-backed storage. |
total_downloads.succeeded | The total number of successful translog download syncs from remote-backed storage. |
total_download_size.succeeded_bytes | The total number of bytes of translog files successfully uploaded from remote-backed storage. |
total_download_time_in_millis | The total amount of time spent, in milliseconds, downloading translog files from remote-backed storage. |
download_size_in_bytes.moving_avg | The average size of translog data, in bytes, downloaded in the last N downloads. N is defined in the remote_store.moving_average_window_size setting. |
download_speed_in_bytes_per_sec.moving_avg | The average speed of translog downloads, in bytes per second, for the last N uploads. N is defined in the remote_store.moving_average_window_size setting. |
download_time_in_millis.moving_avg | The average amount of time taken by a single translog download, in milliseconds, since the last N uploads. N is defined in the remote_store.moving_average_window_size setting. |
Remote store stats for a single shard
Use the following API to get remote store statistics for a single shard.
Example request
GET _remotestore/stats/<index_name>/<shard_id>
Example response
Response
{
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"indices": {
"remote-index": {
"shards": {
"0": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "q1VxWZnCTICrfRc2bRW3nw"
},
"segment": {
"download": {},
"upload": {
"local_refresh_timestamp_in_millis": 1694171634102,
"remote_refresh_timestamp_in_millis": 1694171634102,
"refresh_time_lag_in_millis": 0,
"refresh_lag": 0,
"bytes_lag": 0,
"backpressure_rejection_count": 0,
"consecutive_failure_count": 0,
"total_uploads": {
"started": 5,
"succeeded": 5,
"failed": 0
},
"total_upload_size": {
"started_bytes": 15342,
"succeeded_bytes": 15342,
"failed_bytes": 0
},
"remote_refresh_size_in_bytes": {
"last_successful": 0,
"moving_avg": 3068.4
},
"upload_speed_in_bytes_per_sec": {
"moving_avg": 99988.2
},
"remote_refresh_latency_in_millis": {
"moving_avg": 44.0
}
}
},
"translog": {
"upload": {
"last_successful_upload_timestamp": 1694171633644,
"total_uploads": {
"started": 6,
"failed": 0,
"succeeded": 6
},
"total_upload_size": {
"started_bytes": 1932,
"failed_bytes": 0,
"succeeded_bytes": 1932
},
"total_upload_time_in_millis": 21478,
"upload_size_in_bytes": {
"moving_avg": 322.0
},
"upload_speed_in_bytes_per_sec": {
"moving_avg": 2073.8333333333335
},
"upload_time_in_millis": {
"moving_avg": 3579.6666666666665
}
},
"download": {}
}
},
{
"routing": {
"state": "STARTED",
"primary": false,
"node": "EZuen5Y5Sv-eDCLwh9gv-Q"
},
"segment": {
"download": {
"last_sync_timestamp": 1694171634148,
"total_download_size": {
"started_bytes": 15112,
"succeeded_bytes": 15112,
"failed_bytes": 0
},
"download_size_in_bytes": {
"last_successful": 2910,
"moving_avg": 1259.3333333333333
},
"download_speed_in_bytes_per_sec": {
"moving_avg": 382387.3333333333
}
},
"upload": {}
},
"translog": {
"upload": {},
"download": {}
}
}
]
}
}
}
}
Remote store stats for a local shard
If you want to fetch only shards present on the node serving a Remote Store Stats API request, set the local query parameter to true, as shown in the following example request:
GET _remotestore/stats/<index_name>?local=true