/alibabacloud-elasticsearch-log-config-generator

name: alibabacloud-elasticsearch-log-config-generator
description: >
  Use when the user needs to generate a log collection configuration file
  (OpenTelemetry Collector or Elastic Beats / Filebeat) that writes logs to
  Alibaba Cloud Elasticsearch or self-managed Elasticsearch. Covers file logs,
  log API ingestion (OTLP / HTTP webhook), and Kafka log topics. Also matches natural requests such as: write a filebeat.yml, generate
  an otelcol config, send logs to ES, collect nginx or Java application logs to
  Elasticsearch, or sync Kafka logs to ES. Always output one configuration file
  targeting Elasticsearch; never mix technology stacks or emit non-ES outputs.
license: Apache-2.0
compatibility: >
  Works with cloud-hosted or self-managed Elasticsearch. ES 8.12+ is
  recommended; OpenTelemetry mapping mode otel is recommended on ES 8.16+.
  Requires local otelcol-contrib or filebeat for configuration validation and
  runtime execution. This skill does not require online APIs; all references
  live under references/ and should be loaded on demand.
metadata:
  domain: aiops
  category: database
  product: alibabacloud-elasticsearch
  owner: alibabacloud-elasticsearch-team
  contact: alibabacloud-elasticsearch-skills@alibaba-inc.com

Log Collection Configuration Generator (Elasticsearch)

Scope

This skill generates exactly **one** valid configuration file that collects log data and sends it to **Elasticsearch**. It supports two technology stacks, and each request must choose exactly one:

• **OpenTelemetry Collector** (`otelcol-contrib`) - use `filelogreceiver`, `otlpreceiver`, or `kafkareceiver` with `elasticsearchexporter`.
• **Elastic Beats / Filebeat** - use `filestream`, `kafka`, or `http_endpoint` input with `output.elasticsearch`.

Hard Constraints

1. **Generate only one configuration per request.** Do not concatenate or batch multiple configurations. 2. **Use exactly one technology stack.** Never mix OpenTelemetry components and Filebeat sections in the same file. 3. **Confirm before generating.** Do not generate YAML until the user has answered the Step 1 technology stack confirmation question in a separate reply. 4. **The output must be Elasticsearch.** Never use Kafka, S3, Logstash-only, files, or any other non-Elasticsearch destination as the final output. 5. **Logs only.** Inputs must be log-shaped sources: files, log webhooks, or Kafka topics that contain logs. If the user asks for a clearly different non-log input, refuse and explain why. 6. **No Syslog or Fluent forward input sources.** Do not generate collector or Filebeat listener configurations for Syslog or Fluent forward protocols. If the user asks for Syslog or Fluent forward as the source, refuse and ask them to choose file, OTLP, HTTP webhook, or Kafka log ingestion instead. 7. **Use local knowledge first.** Authoritative references live in the `references/` directory next to this `SKILL.md`. When a component option is covered locally, do not search the web; read the matching file under `references/`. 8. **Keep the design minimal.** Generate only fields requested by the user or required for the file to load. For common but optional features such as batching, retries, or ILM, recommend them and wait for explicit user confirmation before adding them. 9. **Never inline secrets.** Reference all secrets through `${env:VAR}` for OTel or `${VAR}` for Filebeat.

Local References (Prefer Over Web Search)

Before writing YAML, read the matching file under `references/` to verify field names and defaults. The list below is complete. If the required option is not covered, ask the user before writing it.

Security And RAM Permissions

• `references/ram-policies.md` - declares `required_permissions` for Alibaba Cloud RAM review.

OpenTelemetry Collector - `references/opentelemetry/`

Components:

• `01-configuration-basics.md` - top-level structure, pipelines, environment variables, TLS.
• `02-filelogreceiver.md` - file tailing: include/exclude, multiline, operators, storage.
• `03-elasticsearchexporter.md` - `endpoint`/`endpoints`/`cloudid`, authentication, mapping mode, index, sending_queue.
• `04-kafkareceiver.md` - Kafka log ingestion, SASL, encoding.
• `05-otlpreceiver.md` - OTLP gRPC/HTTP log endpoints.
• `06-batchprocessor.md` - batching.
• `07-attributesprocessor.md` - add/delete/mask/extract attributes, log filtering.
• `08-resourcedetectionprocessor.md` - host, cloud, and k8s metadata.
• `09-transformprocessor.md` - OTTL transforms, including setting `elastic.mapping.mode`.
• `12-filestorage-extension.md` - persistent offsets and queues.

Transitive references used by the components above and already localized under hard constraint #5:

• `13-confighttp.md` - shared HTTP client/server configuration: compression algorithms gzip/zstd/snappy/zlib/deflate/lz4, `compression_params.level`, headers, timeouts, keep-alive, CORS, maximum body size. Used by `elasticsearchexporter` and `otlpreceiver/http`.
• `14-configtls.md` - TLS settings: `ca_file`, `cert_file`, `key_file`, `min_version`, `insecure_skip_verify`, mTLS `client_ca_file`, TPM. Used by `confighttp` and `configgrpc`.
• `15-configauth.md` - authentication extension wiring with `auth.authenticator: <ext-name>` and common client/server authentication extensions.
• `16-configgrpc.md` - gRPC client/server settings: `max_recv_msg_size_mib`, keepalive, `compression`, `balancer_name`. Used by `otlpreceiver/grpc`.
• `17-exporterhelper.md` - `sending_queue`, including persistent queues with `storage`, `retry_on_failure`, `timeout`, and batcher. Inherited by `elasticsearchexporter`.
• `18-ottl-overview.md` - OpenTelemetry Transformation Language (OTTL) overview, statements, and contexts. Used by `transformprocessor` and `filterprocessor`.
• `19-ottl-functions.md` - OTTL syntax and the full editor/converter catalog: `set`, `replace_pattern`, `IsMatch`, `ParseJSON`, and more.
• `20-ottl-log-paths.md` - `ottllog` context paths such as `log.body`, `log.s