/cx-data-pipeline

name: cx-data-pipeline
description: >
  Use this skill when the user asks to "set up parsing", "create parsing rule",
  "extract fields from logs", "regex extraction", "log parsing",
  "enrich logs", "add context to logs", "custom enrichment table", "lookup table",
  "geo enrichment", "create metric from logs", "events to metrics",
  "convert logs to metrics", "generate metrics from events",
  "recording rule", "precomputed metrics", "PromQL recording",
  "configure data pipeline", "transform log data", "data processing rules",
  "rule group", "enrichment settings", "E2M definition", "labels cardinality",
  "bulk delete rules", "enrichment limits", "search enrichment table",
  "what should I convert to metrics", "E2M not producing metrics", "E2M no series",
  "reduce log cost with E2M", "logs to metrics aggregation", "spans to metrics",
  or wants to configure how Coralogix processes, enriches, or transforms ingested data.
metadata:
  version: "0.1.0"

Data Pipeline Skill

Use this skill when configuring how Coralogix processes, enriches, and transforms data. It covers parsing rules (extract structured fields from raw logs), enrichments (add context from lookup tables), Events2Metrics (derive metrics from log/span events), and recording rules (precompute PromQL expressions).

---

CLI Commands

| Command | Subcommands | Purpose | |---|---|---| | `cx parsing-rules` | `list`, `get`, `create`, `update`, `delete`, `bulk-delete`, `usage-limits` | Manage log parsing rules | | `cx enrichments` | `list`, `add`, `remove`, `overwrite`, `limit`, `settings` | Manage enrichment rules | | `cx enrichments custom` | `list`, `get`, `create`, `update`, `delete`, `search` | Manage custom enrichment tables | | `cx e2m` | `list`, `get`, `create`, `update`, `delete`, `labels-cardinality`, `limits` | Manage Events2Metrics definitions | | `cx recording-rules` | `list`, `get`, `create`, `update`, `delete` | Manage Prometheus recording rule groups |

Key flags:

• All create/update operations use `--from-file <path>` (or `-` for stdin)
• All commands support `-o json` for structured output and `-p <profile>` for profile selection
• `cx parsing-rules update` and `cx recording-rules update` require both `--from-file` and the rule group ID
• `cx enrichments custom search` requires `--id <table-id>` and `--query <text>`
• `cx parsing-rules bulk-delete` requires `--ids <id1> <id2> ...`

---

Working with JSON Payloads

These commands use complex JSON structures. **Always template from an existing resource** to avoid format errors:

# 1. Get an existing resource as a template
cx parsing-rules get <rule-group-id> -o json > template.json

# 2. Modify the template (change fields, remove the ID for create operations)

# 3. Create or update
cx parsing-rules create --from-file template.json
cx parsing-rules update --from-file template.json <rule-group-id>

This pattern applies to all create/update operations across all 4 commands. It prevents payload format errors that are the #1 cause of failed attempts.

---

Parsing Rules Workflow

1. List Existing Rules

cx parsing-rules list -o json
cx parsing-rules list -o json | jq '[.[] | {id, name, enabled, rule_count: (.rules | length)}]'

2. Get a Template

cx parsing-rules get <existing-rule-group-id> -o json > rule-template.json

3. Create New Rule Group

Edit the template for your new service, then:

cx parsing-rules create --from-file rule-template.json

4. Verify Parsing

Query recent logs to confirm fields are extracted (load `cx-telemetry-querying` for log querying):

cx logs 'source logs | filter $d.subsystem == "my-service" | limit 10' -o json

5. Check Usage Limits

cx parsing-rules usage-limits -o json

---

Enrichment Workflow

1. List Enrichment Rules

cx enrichments list -o json
cx enrichments settings -o json
cx enrichments limit -o json

2. Create Custom Enrichment Table (if needed)

cx enrichments custom list -o json
cx enrichments custom create --from-file table-definition.json

`table-definition.json` must use the v5 JSON shape (inline file content, not multipart `file=@...`):

{
  "name": "IP Lookup",
  "description": "Maps IPs to locations",
  "file": {
    "textual": "ip,city\n1.2.3.4,London",
    "extension": "csv",
    "name": "lookup.csv",
    "size": 24
  }
}

For updates, include `customEnrichmentId` (number) plus the same fields.

3. Add Enrichment Rules

cx enrichments add --from-file enrichment-rules.json

`enrichment-rules.json` must use `requestEnrichments` (not `enrichments` from list output). Each `enrichmentType` is an object, not a string:

{
  "requestEnrichments": [
    {
      "fieldName": "sourceIPs",
      "enrichmentType": { "geoIp": { "withAsn": true } }
    }
  ]
}

Other types: `{"aws": {"resourceType": "ec2"}}`, `{"suspiciousIp": {}}`, `{"customEnrichment": {"id": 1}}`.

4. Search Custom Table Data

cx enrichments custom search --id <table-id> --query "search term"

5. Verify Enriched Fields

Query logs on hot storage (FrequentSearch tier) to confirm enriched fields appear. Avoid querying archive for verification - ingestion delays can cause false negatives.

cx logs 'source logs | filter $d.enriched_field != null | limit 5' -o json

---

Events2Metrics Workflow

E2M derives Prometheus metrics from log/span events. See **[`references/e2m-schemas.md`](references/e2m-schemas.md)** for the full JSON wire format, enums, and cardinality rules.

How E2M is computed (read this first)

E2M aggregates events **as they stream through the real-time ingestion pipeline** into metric series (~1-min resolution). It is **forward-only** — metrics start from the moment the E2M is created; there is **no backfill**.

All ingested data flows through the pipeline; a **TCO policy** routes each stream into a tier, and the tier decides what