/finding-data-lake-assets

name: finding-data-lake-assets
description: >-
  Resolve data lake and lakehouse asset references across Glue Data Catalog, S3, S3
  Tables, and Redshift. Triggers on: find the table, where is our data, which table
  has, locate dataset, find data for, search catalog, what tables match, Redshift
  table, lakehouse table, data lake table, warehouse table, reverse lookup S3 path.
  Do NOT use for: full catalog audits (use exploring-data-catalog), running queries
  (use querying-data-lake), creating tables (use creating-data-lake-table).
metadata:
  version: "2"
  argument-hint: "'[table-name|keyword|column-name|s3://path]'"

Find Data Lake Assets

Overview

Resolves data lake asset references to concrete catalog entries. Acts as a resolver for other skills and direct user requests. Covers Glue, S3, S3 Tables, and Redshift. Optimized for low token usage — return the answer fast and get out of the way.

**Constraints for parameter acquisition:**

• You MUST accept a single argument: table name, keyword, column name, or S3 path
• You MUST accept the argument as direct input or a pointer to a file containing the spec
• You MUST ask for the target AWS region if not already set
• You MUST confirm ambiguous input before searching (e.g., "Did you mean table X or bucket Y?")
• You MUST respect the user's decision to abort at any step

Common Tasks

You MUST execute commands using AWS MCP server tools when connected — they provide validation, sandboxed execution, and audit logging. Fall back to AWS CLI only if MCP is unavailable. You MUST explain each step before executing.

1. Verify Dependencies

Check for required tools and AWS access before searching.

**Constraints:**

• You MUST verify AWS MCP server tools (`aws___call_aws`) are available; fall back to AWS CLI if not
• You MUST confirm credentials with `aws sts get-caller-identity`
• You MUST inform the user about any missing tools and ask whether to proceed

2. Consult Catalog Context (experimental — suggested first lookup)

The customer may publish **context skill assets** in the Glue Data Catalog that map their business language to the real tables — canonical names and aliases, join keys, metrics, usage notes, descriptions — that the raw schema does not carry. When present, this catalog is often enough to answer the request on its own.

These are the **Glue Discovery** operations (`SearchAssets` / `GetAsset` / `ListIterableForms` / `BatchGetIterableForms`) — a distinct metadata-search surface, NOT the legacy `glue search-tables` used in Step 5. They are **experimental** — not available in every CLI build. Gate the lookup on two checks first:

1. **Availability.** Confirm the `GetAsset` operation exists in the caller's Glue CLI model (redirect output so the CLI pager cannot block a non-interactive agent):

   aws glue get-asset help > /dev/null 2>&1
   # exit 0 = available. exit 2 (with "Invalid choice" in stderr) = not in this CLI (skip).
   # any other non-zero (network/credential error) = inconclusive; treat as unavailable.

If it is not available, skip this step and go to the normal search workflow (Steps 3-7). 2. **User opt-in.** If available, ask the user: "I can check the Glue Data Catalog for customer-authored context using an experimental SearchAssets/GetAsset API. Use it? (yes/no)". Proceed only on an explicit yes; otherwise skip to Steps 3-7.

**How this model differs:** Discovery indexes **assets** (not databases/tables). Every asset has an `Id` that is an **ARN**, and every lookup after `SearchAssets` keys off that ARN via the identifier — there is no `--database-name`/`--table-name`. CLI flags are kebab-case (`--search-text`, `--max-results`, `--filter-clause`); top-level response fields are PascalCase (`Id`, `AssetName`, `Forms`). NOTE: a `*.Content` value is itself a JSON STRING with its own camelCase schema (e.g. `dataLocation`, `dataFormat`, `isPartitionKey`) — parse it as embedded JSON, do not expect PascalCase inside. The operations you need:

| Operation | Input → Output | |---|---| | `search-assets` | `--search-text` (+ optional `--filter-clause`) → `Items[]` of `{Id, AssetName, Type, Namespace, AssetTypeId, UpdatedAt}` (NOTE: search items do NOT include a description — call `get-asset` for `Description`/`Forms`) | | `get-asset` | `--identifier <Id, an ARN>` → one asset's `{Description, Forms, IterableForms}`. `Forms."amazon::Table".Content` is JSON `{dataLocation, dataFormat, type}`; advertises column availability via `IterableForms: {"columns": {...}}` | | `list-iterable-forms` | `--asset-identifier <table ARN> --iterable-form-name columns` → that table's columns `Items[]` of `{ItemId, ItemName, Description}` (ItemId = `<table-ARN>#<columnName>`) | | `batch-get-iterable-forms` | `--asset-identifier <table ARN> --iterable-form-name columns --item-identifiers <id1> <id2> ...` (space-separated) → `Items[]` of `{ItemName, Forms}` where `Forms.Column.Content` is JSON `{"type": "...", "isPartitionKey": ...}` |

aws glue search-assets --search-text '<user request terms>' --max-results 5
# Id is a full ARN, e.g. arn:aws:glue:us-west-2:123456789012:table/<db>/<table>
aws glue get-asset --identifier "arn:aws:glue:<region>:<account>:table/<db>/<table>"

`search-assets` returns only identity fields (no description), so to judge relevance you MUST `get-asset` the top candidates (up to ~5) and read their `Description` / `Forms` — do NOT pick by rank alone. Only pass ARNs whose `Type` is a Glue table (`amazon.glue::GlueTable`) to `list-iterable-forms`.

**Narrow with `--filter-clause`** when the request names a database or asset type (filterable: `type`, `amazon.glue::GlueTable.databaseName`, `dataFormat`, `createdAt`):

aws glue search-assets --search-text 'sales' --max-results 5 \
  --filter-clause '{"AttributeFilter": {"Attribute": "amazon.glue::GlueTable.databaseName", "Operator": "equals", "Value": {"StringValue": "<database-name, e.g. sales>"}}}'

**Column name