/dbt-sf-to-bq-translator

name: dbt-sf-to-bq-translator
metadata:
  version: "1.1.3"
  category: BigDataAndAnalytics
description: >-
  Translates Snowflake dbt SQL models to Standardized BigQuery SQL. Handles SQL
  compilation, Jinja macro placeholder masking, BigQuery Translation Service migration
  workflows, AST-based config transformations, explicit type casting, JSON extraction
  standardization, and deduplication. Use when migrating Snowflake dbt pipelines
  or models to BigQuery on Google Cloud. Don't use for generic BigQuery queries or
  non-Snowflake SQL migrations.

dbt Snowflake to BigQuery Translator

You are responsible for: 1. **Dialect Translation**: Translating Snowflake dbt SQL models to Standardized BigQuery SQL on Google Cloud. 2. **Standardization & Compliance**: Enforcing Google-specific standards including copyright headers at the very top of each file, explicit type casting, standardized JSON extraction, and deduplication via `QUALIFY` with `_extracted_at`. 3. **Workflow Integration**: Preserving dbt Jinja constructs and storing the final BigQuery-compatible models.

Follow the instructions given you under `migration_plan/[mig_prefix]/tasks.md`. You will add your progress during operation and summary at the end to the tasks file so that human supervisor can track where you are. You recover from errors by checking the tasks file.

Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:dbt-sf-to-bq-translator)"` and append `agent-skills/1.0 (skill:dbt-sf-to-bq-translator)` to `User-Agent` headers.

Prerequisites & Environment Setup

Before starting the translation, ensure your Google Cloud environment is properly configured: 1. **Google Cloud SDK**: Install the [Google Cloud SDK](https://docs.cloud.google.com/sdk/docs/install-sdk.md.txt) if not already installed. 2. **Authentication**: Authenticate your CLI session:

    gcloud auth login
    gcloud auth application-default login

3. **Project Configuration**: Set your active Google Cloud project:

    gcloud config set project {project_id}

4. **Billing Account**: Verify that an active Google Cloud Billing account is attached to the target project. 5. **Enable Required APIs**: Ensure BigQuery, Migration, and Storage services are enabled:

    gcloud services enable bigquerymigration.googleapis.com storage.googleapis.com bigquery.googleapis.com

6. **Region Selection**: Configure your preferred compute/BigQuery region (default recommended: `us-central1` or `us`). See [Google Cloud Locations](https://cloud.google.com/about/locations):

    gcloud config set compute/region us-central1

Steps

• **Initialization & Setup**: If you do not have a defined `mig_prefix` or if the user wants to start a new translation project, you **MUST** first ask the user for:

1. Migration Project Name (e.g. `my_migration_project`). 2. Input directory containing Snowflake SQL files. 3. Output directory where BigQuery SQL files should be saved. 4. Cloud Storage bucket name for staging translation assets. 5. Google Cloud region (e.g. `us` or `eu`). 6. (Optional) Local path to a directory or `.zip` file containing source database metadata (such as `columns.csv` or `tables.csv`). Once provided, create the tasks checklist file under `migration_plan/[mig_prefix]/tasks.md` with unchecked tasks representing the migration steps.

• **Automated Execution via Bundled Scripts**:

Execute the deterministic end-to-end migration using the bundled translation script `scripts/bulk_translate_via_gcloud.py`:

  python3 scripts/bulk_translate_via_gcloud.py \
    --input {input_dir} \
    --output {output_dir} \
    --bucket {gcs_bucket} \
    --location {region} \
    [--metadata {metadata_path}] \
    [--array-cast-mode {array_cast_mode}] \
    [--struct-columns {struct_columns}]

| Flag | Default | Description | | :--- | :--- | :--- | | `--input` | *(Required)* | Local directory containing source Snowflake dbt `.sql` and `.yml` files. | | `--output` | *(Required)* | Target directory for translated BigQuery dbt `.sql` and `.yml` files. | | `--bucket` | *(Required)* | Cloud Storage bucket name used to stage translation inputs and outputs. | | `--location` | `us` | Google Cloud region for BigQuery Translation Service execution (e.g., `us`, `eu`). | | `--metadata` | `None` | Optional path to a directory or `.zip` archive containing source `tables.csv` and `columns.csv`. | | `--array-cast-mode` | `array_string` | Target conversion for BQMS `ARRAY<JSON>` casts. Use `array_string` (`ARRAY<STRING>`) by default; use `string` (`STRING`), `json_extract_string_array`, or `preserve` as escape hatches when required by the source table schema. | | `--struct-columns` | `None` | Optional comma-separated list of additional BigQuery `STRUCT` column names (e.g., `struct_col,record_col`) to convert from `JSON_VALUE` to native dot notation (`col.field`). |

The migration tools bundled in `scripts/` perform the following coordinated actions:

• `scripts/bulk_translate_via_gcloud.py`: Orchestrates end-to-end bulk migration, automating pre-processing, Cloud Storage upload, BigQuery Translation Service invocation, download, post-processing, and YAML configuration copying.
• `scripts/dbt_translator.py`: Core translation library containing the deterministic AST parser for `config(...)`, Jinja placeholder masking and restoration, JSON/STRUCT/ARRAY extraction sanitization, unimplemented opcode cleanup, macro auditing, casing/join standardization, and copyright header enforcement.

• **Detailed Translation Lifecycle (Executed by Scripts)**:

1. **Compile to Standard SQL using Placeholders (