OpenSearch Administration and Tuning Guide

Administrator procedures for monitoring OpenSearch cluster health and increasing the shard count of an existing index on the AgileSec Platform. This guide applies to both On-Prem Linux and Kubernetes installations.

Overview

This guide documents administrator procedures for maintaining and tuning OpenSearch on the AgileSec platform.

There are two procedures to follow for administering and tuning OpenSearch:

1. Cluster Monitoring and Health Checks. Use these checks to catch oversized shards, unhealthy indices, and incorrect templates before they become an outage. Catching a shard-size problem in Cluster Monitoring and Health Checks means you should add capacity or schedule a re-sharding maintenance window.

2. Update the Shard Count of an Index. Use the re-sharding procedure when an existing index already needs more primary shards, without losing data or changing the index name. OpenSearch does not allow index shard counts to be updated in-place, so this procedure will require downtime.

Cluster Monitoring and Health Checks

Run these checks on a regular cadence, including after a large ingest, after creating a new index, and as part of routine cluster review. Each subsection lists the query, what to look for in the response, and follow-up action.

Cluster health status

1. Get cluster-wide health:

   GET _cluster/health
   

2. Read the following fields:

Template correctness

Most deployments this guide targets should be initially created with 4 primary shards. If an index is on track to exceed roughly 50 GB per primary shard, review Shard fill and disk headroom section to plan a larger shard count at index creation or create a re-sharding plan.

1. Inspect composable index templates:

   GET _index_template
   

2. In each template applying to AgileSec data indices, confirm:

1. 1. template.settings.index.number_of_shards is 4 (or the value your organization has standardized on during installation).

   2. template.settings.index.number_of_replicas matches the intended replica count for the cluster (typically 1 when you have at least two data nodes).

   3. Index patterns still match how new indices are named (organization domain with dots replaced by underscores).

Shard fill and disk headroom

Each primary shard should stay at or below 50 GB.

1. Check disk per node, then size per shard:

   GET _cat/allocation?v&h=node,shards,disk.indices,disk.used,disk.avail,disk.total,disk.percent
   GET _cat/shards/agilesec.*.v3.event-*,agilesec.*.v3.alert-*?v&h=index,shard,prirep,state,store,node
   

2. What to look for:

   1. Shard store approaching 50 GB: Schedule a re-sharding window using Update the Shard Count of an Index. Do not wait until shards exceed this size, as large shards degrade cluster performance and recovery times.

   2. Disk percent high on any node: Add disk capacity before a watermark forces the index to read-only. Note an index split requires roughly 2× the primary store size free because the original and temporary index exist together until cleanup.

3. Confirm watermark thresholds. Defaults vary by version; high watermark stops allocating to the node, flood stage marks indices as read-only:

   GET _cluster/settings?include_defaults=true&filter_path=**.cluster.routing.allocation.disk.*
   

4. If an index is unexpectedly read-only, check cluster.routing.allocation.disk.watermark.flood_stage and free space on every data node before clearing index.blocks.read_only_allow_delete.

System and node health

1. List node-level CPU, memory, heap, and disk:

GET _cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent

2. Observe if heap.percent stays high (for example above 75–80%) and note if cpu / load_1m does not come down after ingest. Obtain JVM heap and GC pressure with the following command:

   GET _nodes/stats/jvm?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_used_percent,nodes.*.jvm.mem.heap_used_in_bytes,nodes.*.jvm.gc
   

3. Write and search thread pools. Rejections mean the cluster is dropping work:

   GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected&s=rejected:desc
   

4. Focus on write, search, and search_throttled results. A growing rejected count needs fewer concurrent heavy queries, more nodes, or a pause in ingest, not a template change.

Index health

1. List only unhealthy indices:

   GET _cat/indices?v&health=yellow
   GET _cat/indices?v&health=red
   

   An empty table means no indices are in that state.

2. When a row appears, note index, pri, rep, docs.count, and store.size, then collect per-index stats:

   GET <index name>/_stats
   

3. Use _stats to confirm document count, store size, and indexing/search rates for each index.

4. If the index is yellow or red because of allocation, run allocation explain to determine why the shards are unassigned:

   GET /_cluster/allocation/explain
   

5. If the index is green but shards are larger than 50 GB, plan capacity or re-sharding. Do not try to “fix” size at this point. The proper re-sharding process must be followed to avoid data corruption, data loss, or other consequences.

Additional checks

These are not shard-sizing checks, but they catch cluster problems indicating “OpenSearch is slow” during ingest or recovery.

1. Run the following checks:

   GET _cluster/pending_tasks
   GET _cluster/allocation/explain
   GET _cat/recovery?v&active_only=true
   GET _cluster/stats?filter_path=indices.count,indices.shards,indices.store,nodes.count,status
   

2. Note the following from the check results:

1. 1. _cluster/pending_tasks should normally be empty. A long queue means cluster-manager work is backed up, often mapping updates or reroutes.

   2. _cluster/allocation/explain with no body explains the first unassigned shard. Use it whenever health is yellow or red.

   3. _cat/recovery?active_only=true shows in-flight recoveries. Progressing bytes_percent is healthy; a frozen percentage is not.

Update the Shard Count of an Index (Re-sharding)

OpenSearch does not support changing index.number_of_shards on an existing index in place. To increase shard count while keeping the original name, split the index into a temporary index with the target shard count, then clone the temporary index back to the original name.

Prerequisites

The following prerequisite steps confirm which indexes to re-shard and determine the target shard counts. Ensure all prerequisites are complete before proceeding with updating shard counts.

Confirm cluster health

1. Check overall cluster health:

   GET _cluster/health
   

   This is cluster-wide health, not the health of a single index.

2. Confirm status is green and unassigned_shards is 0. Do not start updating shard counts on a yellow or red cluster.

Confirm disk headroom

3. The split step briefly requires enough free space to hold both the original index and the new temporary index at the same time (the original is not deleted until after the split completes and is verified). Check available capacity per node before proceeding:

   GET _cat/allocation?v&h=node,shards,disk.indices,disk.used,disk.avail,disk.total,disk.percent
   

4. Add disk space if any node is close to capacity. You need about 2 × primary store size. (Use the shard query in the next step to get current primary shard count sizes.)

Identify candidate indexes

5. List indexes for your organization, sorted by primary shard count:

   GET _cat/indices/agilesec.*.v3.event-*,agilesec.*.v3.alert-*?v&h=index,pri,rep,docs.count,store.size&s=pri
   

6. Review the index list generated in the previous step and identify any index where the average shard size (store.size divided by pri) is approaching or exceeds 50 GB.

7. These large indexes are candidates for re-sharding. The remaining prerequisites and procedure steps apply to the results when the list command is run.

Determine the target shard count

8. Each shard should hold no more than 50 GB. Use the following formula as a starting point, then round up to a convenient valid count:

   Target shard count = (store × 2) / 50
   

9. store is the size of the primary index. Get current shard sizes:

   GET _cat/shards/<index name>?v&h=index,shard,prirep,state,store,node
   

Review target shard count limitations

10. The target shard count must also be reachable for this index. Check shard index limits:

     GET <index name>/_settings?include_defaults=true
    

11. Review index.number_of_routing_shards. This value is fixed when the index is created and limits how far the index can ever be split, so a target satisfying the multiple rule above can still be rejected.

12. Confirm your target is reachable before planning the maintenance window.

Backup candidate indexes

13. Important: back up candidate indexes before proceeding with the next procedure.

Procedure: Update Index’s Shard Count / Re-sharding

Repeat the following steps for each index identified in the prerequisites. Substitute the actual index name for <index name> and the value from the sizing formula for <target shard count>.

Step 1: Block writes on the source index

1. Run the following command to block writes on the source index:

   PUT <index name>/_settings
   {
     "settings": {
       "index.blocks.write": true
     }
   }
   

Step 2: Resolve shard size mismatches (only if needed)

1. Run forcemerge to compress segments and reduce the index store size:

   POST <index name>/_forcemerge?only_expunge_deletes=false&max_num_segments=1&wait_for_completion=false
   

2. forcemerge can take a while on large shards. Run forcemerge asynchronously and poll for completion for better efficiency:

   GET _tasks?actions=*forcemerge*&detailed=true
   

3. An empty result (no matching tasks) means the merge has finished. Re-check shard sizes with a shard-layout query. All shards should be roughly equal once complete.

   GET _cat/shards/<index name>?v
   

Step 3: Split into a temporary index with the target shard count

1. Split the target index into a temporary index with the target shard count with the split API:

   POST <index name>/_split/reshard-<index name>-tmp
   {
     "settings": {
       "index.number_of_shards": <target shard count>,
       "index.number_of_replicas": 0
     }
   }
   

Step 4: Monitor split progress

1. Poll recovery status until every shard shows stage: done:

   GET _cat/recovery/reshard-<index name>-tmp?v&h=index,shard,stage,files_percent,bytes_percent,time
   

2. Confirm cluster health for the temporary index is green:

   GET _cluster/health/reshard-<index name>-tmp?wait_for_status=green&timeout=60s
   

Step 5: Verify document counts match

1. Verify the target and temporary document counts match:

   GET <index name>/_count
   GET reshard-<index name>-tmp/_count
   

2. Counts must match exactly before you continue.

Step 6: Verify shard layout and distribution

1. Verify shard layout and distribution:

   GET _cat/shards/reshard-<index name>-tmp?v
   

2. Confirm the expected number of primary and replica shards are all in STARTED state.

3. While reviewing shard layout, also check for size mismatches between shards that hold similar document counts. Size mismatches usually indicates unmerged segments rather than data loss.

GET _cat/shards/reshard-<index name>-tmp?v&h=index,shard,prirep,state,store,node

POST _cluster/reroute
{
  "commands": [
    {
      "cancel": {
        "index": "reshard-<index name>-tmp",
        "shard": <n>,
        "node": "<current primary node>",
        "allow_primary": true
      }
    }
  ]
}

POST _cluster/reroute
{
  "commands": [
    {
      "move": {
        "index": "reshard-<index name>-tmp",
        "shard": <n>,
        "from_node": "<current primary node>",
        "to_node": "<free node>"
      }
    }
  ]
}

Step 7: Delete the original index

1. Delete the original index for replacement:

   DELETE <index name>
   

Step 8: Clone the temporary index back to the original name

1. Clone the temporary index to the original index’s name:

   POST reshard-<index name>-tmp/_clone/<index name>
   

Step 9: Confirm the final index is healthy with the correct shard count

1. Confirm the final, cloned index is healthy and get the shard count:

GET _cluster/health/<index name>?wait_for_status=green&timeout=60s
GET <index name>/_settings?pretty

2. Confirm index.number_of_shards equals the target count.

Step 10: Verify document count one more time

1. Re-confirm the document count:

GET <index name>/_count

Step 11: Clean up / delete the temporary index

1. Clean up the temporary index with a delete command:

   DELETE reshard-<index name>-tmp
   

Step 12: Clear write block

1. Clear the write block carried over from cloning:

   PUT <index name>/_settings
   {
     "settings": {
       "index.blocks.write": false
     }
   }
   

2. Confirm if the write block is cleared. :

   GET <index name>/_settings?pretty
   

3. index.blocks.write should now be false.

Troubleshooting

Use the following steps for troubleshooting common errors.

Shards stuck in UNASSIGNED or INITIALIZING

1. Get the allocation decision for a specific shard (this is the most direct way to find the blocker):

   GET _cluster/allocation/explain
   {
     "index": "reshard-<index name>-tmp",
     "shard": <n>,
     "primary": false
   }
   

2. The response reason and deciders fields explain why the shard has not been allocated. The most common cause during re-sharding is recovery throttling, not a hard error:

   1. cluster.routing.allocation.node_concurrent_recoveries limits how many shards can recover at once on a given node. Extra shards queue as UNASSIGNED (INDEX_CREATED) until a slot frees up.

   2. cluster.routing.allocation.node_initial_replicas_recoveries limits concurrent outgoing replica recoveries from the node holding the primaries.

3. Both node_concurrent_recoveries and node_initial_replicas_recoveries resolve as in-flight recoveries finish. To speed things up temporarily, increase node_initial_replicas_recoveries:

   PUT _cluster/settings
   {
     "transient": {
       "cluster.routing.allocation.node_initial_replicas_recoveries": 8
     }
   }
   

4. Revert node_initial_replicas_recoveries once the split finishes. This setting is cluster-wide and affects later recoveries and rebalances:

   PUT _cluster/settings
   {
     "transient": {
       "cluster.routing.allocation.node_initial_replicas_recoveries": null
     }
   }
   

Recovery looks stalled

1. Check byte-level progress rather than relying on shard state alone. A shard reported as INITIALIZING is still progressing as long as its recovery percentage is climbing:

   GET reshard-<index name>-tmp/_recovery?detailed=true
   

2. If the percentage is frozen across repeated polls (not just slow), check recovery bandwidth throttling and node-level I/O:

   GET _nodes/stats/indices/recovery?filter_path=**.total.throttle_time_in_millis
   GET _cluster/settings?include_defaults=true&filter_path=**.indices.recovery.*
   

Appendix: Diagnostic query reference

Quick reference for the diagnostic queries used in this guide, plus a few additional ones useful for general cluster health checks.

Disk and capacity

Shard state and recovery

Merging and tasks

Health and templates