/lhm-data-validation-skill

name: lhm-data-validation-skill
version: 0.1.1
description: Validates data consistency across heterogeneous data sources based on the Alibaba Cloud Lakehouse Migration Center (LHM) product.

LHM Data Validation Service (aliyun CLI)

Drives the entire data validation workflow through the aliyun CLI (`aliyun-cli-lhm` plugin). This Skill is **execution-first**: once the user specifies the data sources, tables, and check type, match the function directly and trigger execution; load reference documents on demand only for complex diagnosis and design scenarios.

> **Network layer**: all API calls are uniformly wrapped by `scripts/aliyun_cli.py` as `aliyun lhm <command>` subprocess invocations, keeping the same call shape as the original Python SDK (`client.xxx(request)` → `resp.body.data`). The request models are provided by the lightweight local module `scripts/lhm_models.py` (the former `alibabacloud_lhm20250116` SDK is deprecated and no longer installed). The aliyun CLI and the aliyun-cli-lhm plugin must be installed before running (see README). > > ⚠️ **Sandbox environment compatibility**: in the evaluation system's sandbox environment, the `aliyun` CLI may be wrapped as a proxy, and actual calls require using `aliyun_real`. The `find_aliyun_binary()` function in `scripts/aliyun_cli.py` already includes this detection logic (it prefers `aliyun_real`). But if you invoke the `aliyun` command directly through Bash, detect it first: > ```bash > if command -v aliyun_real >/dev/null 2>&1; then ALIYUN_CMD="aliyun_real"; else ALIYUN_CMD="aliyun"; fi > ${ALIYUN_CMD} lhm <command> ... > ```

> ⛔ **Plugin prerequisite — hard stop (mirrors the LHM dispatcher's Step 2.1 Hard Stop Gate)**: every atomic skill in this package reaches the service through `aliyun lhm <command>`, so the `aliyun-cli-lhm` plugin (>= 0.1.1, i.e. `~/.aliyun/plugins/aliyun-cli-lhm/manifest.json` must exist) is a hard prerequisite for *all* checks — count / metric / batch / template / report. Verify it before anything else; when this Skill is entered directly rather than through the dispatcher, run the dispatcher's `scripts/install_lhm_plugin.sh` and treat a non-zero exit as fatal. > > If the plugin is unavailable, **stop immediately**: report the environment error verbatim and terminate. Do **not** ask the user for `ds_id`, aliases, database/table names, check type, or sampling rate; do **not** run `run.py check`, any validation script, or any dependency install; and **never output a row count, consistency ratio, or pass/fail verdict that did not come from a real call**. Successfully installed local Python dependencies are never proof that the cloud path works.

Quick Execution Path

0. **Pre-check** (on first run or after environment changes):

   python atomic-skills/lhm-common/scripts/run.py check --profile data-validation

Continue only when `ready=true`; when `ready=false`, guide the user to fix things per the `fix_guide` entries in `blocking_items`. On first use, run `setup` for guided configuration. 1. **Read configuration**: credentials and data sources are read preferentially from `~/.lhm/data_validation_config.yaml`. Users may reference data sources by alias (e.g., `mc_source`, `sr_target`) without memorizing ds_ids. 2. **Build the client**: `build_client()`. Every `aliyun lhm` call carries the fixed parameters `--endpoint` / `--region`, read from the `lhm` section of `~/.lhm/credentials.json` (`lhm.endpoint` / `lhm.region_id`); an explicit `build_client(region='singapore')` overrides the region, and the environment variables (`LHM_ENDPOINT` / `REGION_ID`) then built-in defaults are fallbacks. Credentials are resolved by the aliyun CLI default credential chain — never read from any config file. 3. **Prepare data sources**: `src_ds = (ds_id, ds_name, ds_type)` / `dst_ds = (ds_id, ds_name, ds_type)`, where `ds_id` is the only SDK input. When aliases are used, resolve them from the `data_sources` section of data_validation_config.yaml. 4. **Invoke by intent** (prefer the wrapper functions in `scripts/common.py`):

• **Template selection** (before metric checks): use 1001(MIX) by default. For custom rules, list available templates with `list_templates()`, or load `knowledge/patterns/template-guide.md` for guided configuration. Templates can also be CRUD-managed directly via `atomic-skills/lhm-template-manage`. For manual console configuration, see `knowledge/patterns/template-manual-setup.md`
• Count check → `run_count_check()` or `run_batch_check(check_type=0)`
• Metric check → `run_metric_check()` or `run_batch_check(check_type=1)`
• View/download reports → `summarize_batch()` / `download_report()`
• Rerun failures → `rerun_failed()`
• If the raw SDK APIs are needed (scenarios not covered by the wrapper functions), search `references/api-integration.md`

5. **Poll until completion**: `poll_exec_status(client, task_id, batch_id)`, targeting `exec_status=4`.

Core Parameters

| Parameter | Description | Example | |-----------|-------------|---------| | `region` | Center node | `hangzhou` (default), `singapore` | | `src_ds` / `dst_ds` | Data source triple `(ds_id, ds_name, ds_type)` | `('ds-001', '源端MySQL', 'MySQL')` | | `check_type` | Check type | `0` count / `1` metric / `2` weak content | | `task_mode` | Creation mode | `0` per-table / `1` batch | | `match_rule` | Batch matching rule | `src_db|dst_db|*`; full syntax in `references/batch_match_rules.md` | | `check_template_id` | Check template ID. Built-in: 1001(MIX)/1002(NUM)/1003(LEN); custom: a user-provided UUID | `1001` (default) |

User Intent Quick Reference

| User intent | Corresponding capability | |-------------|--------------------------| | Run a count check | `run_count_check()` / `run_batch_check(check_type=0)` | | Run a metric check | `run_metric_check()` / `run_batch_check(check_type=1)` | | View results / download reports | `summarize_batch()` / `download_report()` | | Diagnose the root cause of differe