cluster-operations

description: Cluster health, shard allocation, capacity planning, rolling upgrades, and snapshot/restore operations

OpenSearch/Elasticsearch Cluster Operations

> **Scope**: Cluster health management, shard allocation debugging, node roles, JVM heap tuning, rolling upgrades, and snapshot configuration. OpenSearch 2.x and Elasticsearch 8.x. > **Version range**: OpenSearch 2.0+ / Elasticsearch 8.0+ > **Generated**: 2026-04-08

---

Overview

Cluster operations have asymmetric consequences: misconfigured heap or shard counts are silent until load spikes. Yellow cluster status is tolerable; red is data loss risk. The most dangerous operations — DELETE index, update live mapping, shrink shards — are irreversible without snapshots. Every cluster configuration change requires a before/after snapshot.

---

Pattern Table

| Pattern | Version | Use When | Prefer Another Pattern When | |---------|---------|----------|------------| | `cluster.routing.allocation.enable: all` | All versions | After maintenance window | Never set to `none` and forget | | `indices.recovery.max_bytes_per_sec` | All versions | Limiting recovery bandwidth | Default is unlimited (saturates network) | | Hot-warm-cold node roles | OS 2.0+ / ES 7.0+ | Mixed workload (active + archive data) | Single-tier small clusters | | Cross-cluster replication | OS 1.1+ / ES 6.5+ | DR, geographic distribution | Simple single-cluster HA | | Snapshot before destructive ops | Always | Before DELETE, reindex, mapping update | Never skip |

---

Correct Patterns

Diagnosing Yellow/Red Cluster Status

Start with allocation explain before guessing.

# Step 1: Overall health
GET /_cluster/health?pretty

# Step 2: Identify unassigned shards
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state:desc

# Step 3: Get authoritative explanation for unassigned shard
GET /_cluster/allocation/explain
{
  "index": "your-index",
  "shard": 0,
  "primary": false
}

# Step 4: Check disk thresholds (common cause)
GET /_cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation.disk*

# Common fix for disk threshold exceeded:
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}

**Why**: `GET /_cluster/allocation/explain` tells you exactly why a shard won't assign (disk full, no eligible node, node excluded, etc.). Guessing without this leads to misdiagnosis.

---

JVM Heap Configuration

# In jvm.options (or opensearch.yml for OS 2.12+):
# Set to 50% of available RAM, never exceed 31GB
-Xms16g
-Xmx16g

# Verify heap in use
GET /_nodes/stats?filter_path=nodes.*.jvm.mem

# Monitor GC pressure
GET /_nodes/stats?filter_path=nodes.*.jvm.gc.collectors.*.collection_time_in_millis
# GC time > 25% of wall clock = heap pressure

**Why**: JVM uses compressed ordinary object pointers (compressed OOPs) when heap is at or below ~31GB. Above this threshold, the JVM switches to 64-bit pointers — pointer size doubles, effective heap capacity drops by ~30%. Set heap to exactly 31g maximum; verify with `GET /_nodes` that compressed OOPs is active.

---

Rolling Upgrade Procedure

# 1. Disable shard allocation before each node
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "primaries"
  }
}

# 2. Flush syncronized (ES 7.6+ / OS: not needed — handled by upgrade)
POST /_flush

# 3. Stop node, upgrade, start node
systemctl stop opensearch
# ... upgrade package ...
systemctl start opensearch

# 4. Wait for node to rejoin
GET /_cat/nodes?v

# 5. Re-enable allocation
PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": null
  }
}

# 6. Wait for green before upgrading next node
GET /_cluster/health?wait_for_status=green&timeout=300s

# 7. Repeat for each node

**Why**: Disabling allocation before stopping a node prevents shard recovery storms. Without this, the cluster starts recovering replicas to other nodes as soon as the node goes down — only to cancel and re-recover when it comes back up. This wastes network and CPU.

---

Snapshot Configuration

# Register S3 snapshot repository
PUT /_snapshot/backups
{
  "type": "s3",
  "settings": {
    "bucket": "es-snapshots-prod",
    "region": "us-east-1",
    "base_path": "snapshots",
    "compress": true,
    "chunk_size": "1gb",
    "server_side_encryption": true
  }
}

# Create snapshot policy (automated)
PUT /_slm/policy/nightly-snapshots
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-snap-{now/d}>",
  "repository": "backups",
  "config": {
    "indices": ["*"],
    "ignore_unavailable": false,
    "include_global_state": true
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

# Verify latest snapshot
GET /_snapshot/backups/_all?pretty&s=start_time:desc

**Why**: `include_global_state: true` captures index templates, ILM policies, and cluster settings — not just data. Without global state, a full cluster restore requires manual recreation of all configuration.

---

Pattern Catalog

Keep JVM Heap at or Below 31GB

**Detection**:

# Check current heap for all nodes
GET /_nodes?filter_path=nodes.*.jvm.mem.heap_max_in_bytes
# Convert bytes to GB and flag if > 31GB
curl -s "$ES_HOST/_nodes/stats/jvm" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for nid, n in data['nodes'].items():
    heap_gb = n['jvm']['mem']['heap_max_in_bytes'] / (1024**3)
    flag = ' *** ABOVE 31GB ***' if heap_gb > 31 else ''
    print(f\"{n['name']}: {heap_gb:.1f}GB{flag}\")
"

**Signal**:

# jvm.options
-Xms64g
-Xmx64g  # 64GB — loses compressed OOPs

**Why this matters**: Above ~31GB, JVM object pointers are 64-bit instead of 32-bit compressed. Memory per object increases significantly. The JVM now needs a lar