index-management

description: Mapping design, ILM policies, index templates, and reindexing strategies with failure mode detection

OpenSearch/Elasticsearch Index Management

> **Scope**: Index mapping, analyzer configuration, ILM policies, index templates, and reindexing. Covers OpenSearch 2.x and Elasticsearch 8.x. Does not cover cluster-level shard allocation (see cluster-operations.md). > **Version range**: OpenSearch 2.0+ / Elasticsearch 8.0+ > **Generated**: 2026-04-08

---

Overview

Index management failures are slow-moving disasters: dynamic mapping enables today, then causes mapping explosion in 3 months. ILM not configured today means manual deletion during a storage emergency. Mapping design is largely irreversible on live indices — reindexing 100GB of data takes hours. Get it right at creation time.

---

Pattern Table

| Pattern | Version | Use When | Avoid When | |---------|---------|----------|------------| | `"dynamic": "strict"` | All versions | Production indices with known schema | Exploratory/dev indices | | `"dynamic": false` | All versions | Log indices with variable fields | Need to query dynamic fields | | `keyword` sub-field on `text` | All versions | Need both full-text and aggregation | Field is never aggregated | | `flattened` field type | ES 7.3+/OS 1.0+ | JSON objects with unknown keys | Need to score individual sub-fields | | Index aliases | All versions | Zero-downtime reindexing | Direct index name in application code | | Rollover API | All versions | Time-series data, log ingestion | Static content indices |

---

Correct Patterns

Explicit Mapping with Strict Dynamic

Define all fields at creation time; reject unknown fields.

PUT /articles
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.mapping.total_fields.limit": 200
  },
  "mappings": {
    "dynamic": "strict",
    "_source": { "enabled": true },
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 }
        },
        "analyzer": "english"
      },
      "content": { "type": "text", "analyzer": "english" },
      "author_id": { "type": "keyword" },
      "published_at": { "type": "date", "format": "strict_date_time" },
      "tags": { "type": "keyword" },
      "view_count": { "type": "long" },
      "metadata": {
        "type": "object",
        "dynamic": false
      }
    }
  }
}

**Why**: `"dynamic": "strict"` rejects documents with unknown fields (returns a 400 error). This prevents the gradual accumulation of unknown field mappings that causes `index.mapping.total_fields.limit` to be reached silently.

---

ILM Policy for Log Indices

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "30gb",
            "max_age": "1d",
            "max_docs": 10000000
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "shrink": { "number_of_shards": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "set_priority": { "priority": 0 },
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

**Why**: Hot phase: active writes, high-priority. Warm phase: force-merge reduces segment count (read optimization), shrink consolidates shards from hot to 1 (saves file descriptors). Cold phase: frozen indices move to object storage. Delete: automatic cleanup.

---

Zero-Downtime Reindex with Alias

# 1. Create new index with updated mapping
PUT /articles-v2
{ ... new mapping ... }

# 2. Reindex from old to new (background)
POST _reindex?wait_for_completion=false
{
  "source": { "index": "articles-v1" },
  "dest": { "index": "articles-v2" }
}

# 3. Monitor progress
GET _tasks?actions=*reindex

# 4. Once complete, atomically swap alias
POST _aliases
{
  "actions": [
    { "remove": { "index": "articles-v1", "alias": "articles" } },
    { "add": { "index": "articles-v2", "alias": "articles" } }
  ]
}

# 5. Verify alias points to new index
GET articles/_alias

**Why**: Application code always queries `articles` alias — never the versioned index name. Alias swap is atomic: no gap between removing old and adding new. Clients see no downtime.

---

Index Template for Consistent Mapping

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "priority": 100,
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy",
      "index.lifecycle.rollover_alias": "logs"
    },
    "mappings": {
      "dynamic": false,
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "message": { "type": "text" },
        "trace_id": { "type": "keyword" }
      }
    }
  },
  "data_stream": {}
}

**Why**: Template applies automatically to new indices matching the pattern. Without a template, each rollover creates an index with default settings — no ILM policy, no explicit mapping.

---

Pattern Catalog

Use Strict or Flattened Mapping for Variable Keys

**Detection**:

# Check current field count on an index
curl -s "$ES_HOST/your-index/_mapping" | python3 -c "
import json, sys
mapping = json.load(sys.stdin)
idx = list(mapping.keys())[0]
props = mapping[idx]['mappings'].get('properties', {})
print(f'Top-level field count: {len(props)}')
"

# Check all indices for field counts
GET /_cat/indices?v&h=index,docs.count
GET /your-index/_mapping | jq '.[] | .mappings.properties | keys | length'

**Sig