/spatial-raw-processing

name: spatial-raw-processing
description: Load when converting spatial transcriptomics raw FASTQ pairs through ST-Pipeline into a `raw_counts.h5ad`
  ready for spatial-preprocess. Skip when input is already a count-matrix AnnData (use spatial-preprocess);
  non-spatial bulk / scRNA FASTQ (use bulkrna-read-qc).
trigger: spatial raw processing, raw spatial fastq, spatial fastq, st_pipeline, st pipeline, barcode coordinates, ids file, visium raw fastq, slide-seq fastq, slideseq fastq, upstream spatial processing
tags:
- spatial
- raw-processing
- fastq
- st-pipeline
- visium
- slideseq

spatial-raw-processing

Use from a step

This is CLI_ONLY: ST-Pipeline consumes FASTQ files, barcode coordinates and reference indexes. It is not an in-memory analysis function.

from skills._sdk.notebook import run_cli

output = run_cli("spatial-raw-processing", "--input", "data/run_bundle.json",
                 inputs=["data/run_bundle.json", "data/R1.fastq.gz",
                         "data/R2.fastq.gz", "data/barcodes.tsv"])

Record the reference index version in the module README. The executable example uses synthetic upstream outputs and does not run ST-Pipeline.

When to use

The user has paired-end spatial-transcriptomics FASTQ files (`read1` = spatial barcode + UMI, `read2` = cDNA) plus a STAR genome index, and wants the standard ST-Pipeline run that produces a `raw_counts.h5ad` with one row per spatial spot. Single backend: `st_pipeline` (calls `run_stpipeline` from `skills/spatial/_lib/stpipeline_adapter.py`).

After this skill, chain to `spatial-preprocess` for QC + normalisation. For non-spatial scRNA FASTQ use `sc-fastq-qc`. For bulk RNA-seq read QC use `bulkrna-read-qc`.

Inputs & Outputs

**Inputs**

• Input kinds: `file`, `directory`
• Modalities: visium, slideseq
• File types: `.fastq`, `.fq`, `.json`, `.yaml`, `.yml`
• FASTQ structure: valid first record; `paired` layout
• Directory layouts (any): `paired-fastq`

**Outputs**

• `tables/gene_qc.csv`
• `tables/raw_gene_qc.csv`
• `tables/raw_processing_run_summary.csv`
• `tables/raw_processing_spatial_points.csv`
• `tables/raw_spot_qc.csv`
• `tables/raw_top_genes.csv`
• `tables/run_summary.csv`
• `tables/saturation_curve.csv`
• `tables/spatial_coordinates.csv`
• `tables/spot_qc.csv`
• `tables/stage_summary.csv`
• `tables/top_genes.csv`
• `figures/raw_detected_genes_spatial.png`
• `figures/raw_spot_qc_histograms.png`
• `figures/raw_top_genes_barplot.png`
• `figures/raw_total_counts_spatial.png`
• `figures/st_pipeline_saturation_curve.png`
• `figures/st_pipeline_stage_attrition.png`
• `omicsclaw_stpipeline_run.json`
• `raw_counts.h5ad`
• `st_pipeline.stderr.txt`
• `st_pipeline.stdout.txt`
• `report.md`
• `result.json`
• Processed AnnData (`saves_h5ad`) — adds `obs`: `barcode`, `x_array`, `y_array`; `obsm`: `spatial`

Flow

1. Parse args (or load bundle JSON / YAML from positional `--input`). 2. `_apply_effective_defaults` fills missing parameter values (threads, trimming, UMI ranges, etc.). 3. `_validate_real_run_bundle`: check `read1` / `read2` / `ids` / `ref-map` exist and are well-typed; reject duplicate read1=read2; verify FASTQ extension. 4. Call `run_stpipeline(...)` which shells out to ST-Pipeline (requires the `stpipeline` binary on PATH or `--stpipeline-repo` + `--bin-path`). 5. Wrap the resulting count matrix into AnnData with `X = raw_counts`, `layers["counts"]`, `raw = raw_counts_snapshot`. 6. Save `raw_counts.h5ad` and `result.json`. Print "next: spatial-preprocess on raw_counts.h5ad".

Gotchas

• **All input failures raise typed exceptions wrapped in `SystemExit(1)`.** `spatial_raw_processing.py` catches `DataError` / `DependencyError` / `ParameterError` / `ProcessingError` and re-raises as `SystemExit(1)`. The originating raises live in `_validate_real_run_bundle` — it raises `ParameterError(f"Missing required parameter: {key}")` for missing `read1`/`read2`/`ids`; raises `DataError(...)` for non-existent files; raises `DataError("Resolved read1/read2 inputs must be FASTQ files.")` for non-FASTQ extensions; raises `ParameterError` for read1==read2; raises `DataError` for missing / wrong-type STAR index dir; raises `DataError` only when `--ref-annotation` was *provided* but the path is missing or not a file (the param itself is optional — omitting it doesn't raise).
• **`--read1` / `--read2` / `--ids` / `--ref-map` are all required for real runs** (not enforced by argparse `required=True`, validated later). Missing any → `ParameterError`. Demo mode skips this validation entirely.
• **The output filename is always `raw_counts.h5ad`** (`spatial_raw_processing.py`). It's not configurable — the contract is consumed by `spatial-preprocess`. Multiple runs to the same `--output` will overwrite.
• **QC tables and figures are written.** `tables/spot_qc.csv` and `figures/raw_total_counts_spatial.png` describe the count matrix; upstream stage and saturation plots depend on available pipeline metrics.
• **Demo mode skips ST-Pipeline entirely.** `spatial_raw_processing.py` calls `create_demo_upstream_outputs(...)` to fabricate a synthetic `raw_counts.h5ad`. Useful for plumbing checks; does NOT exercise the FASTQ → matrix code path.
• **`--platform` is a metadata label only.** `_build_parser` documents it as "Label recorded in outputs"; ST-Pipeline doesn't branch on it. Common values: `visium`, `visium_hd`, `slideseq`, custom strings.

Key CLI

# Demo (synthetic raw_counts.h5ad — does NOT run ST-Pipeline)
python skills/spatial/spatial-raw-processing/spatial_raw_processing.py --demo --output /tmp/spatial_raw_demo

# Real run with explicit args
python skills/spatial/spatial-raw-processing/spatial_raw_processing.py \
  --read1 sample_R1.fastq.gz --read2 sample_R2.fastq.gz \
  --ids barcodes.tsv \
  --ref-map /refs/star_index_human \
  --ref-annotation /refs/genes.gtf \
  --exp-name visium_001 --platform visium \
  --threads 16 \
  --output results/

# Real run from bundle JSON
python skills/spatia