/pipeline-guide

name: pipeline-guide
description: Access ENCODE uniform analysis pipelines, generate user-specific Nextflow/WDL pipelines, manage compute resources, and integrate with cloud platforms. Use when the user wants to understand ENCODE pipelines, run pipelines on their own data, generate custom Nextflow workflows from ENCODE pipeline code, check compute requirements (CPU/GPU/memory), run pipelines in background, or integrate with Google Cloud, AWS, or other cloud platforms. Also use when the user asks about ENCODE pipeline outputs, processing standards, software versions, or wants to replicate ENCODE processing. Covers local execution, HPC, and cloud deployment with resource-aware scheduling. Use this skill for ANY pipeline execution, workflow generation, or compute resource management task involving ENCODE data.

ENCODE Pipeline Guide and Custom Workflow Generation

When to Use

• User wants to understand ENCODE uniform analysis pipelines or run them on their own data
• User asks about "ENCODE pipeline", "Nextflow", "WDL", "processing standards", or "pipeline requirements"
• User needs to generate a custom Nextflow/WDL workflow based on ENCODE pipeline specifications
• User wants to know compute requirements (CPU, GPU, memory, storage) for running pipelines
• Example queries: "how do I run the ENCODE ChIP-seq pipeline?", "what are the compute requirements for Hi-C processing?", "generate a Nextflow pipeline for my ATAC-seq data"

Understand ENCODE pipelines, generate user-specific workflows in Nextflow/WDL, and manage compute resources for local, HPC, and cloud execution.

ENCODE Uniform Analysis Pipelines

ENCODE uses standardized pipelines for each assay type, ensuring reproducibility across all datasets. All pipelines are:

• **Open source**: GitHub (github.com/ENCODE-DCC)
• **Containerized**: Docker and Singularity images
• **Written in WDL**: Workflow Description Language (Cromwell execution engine)
• **Portable**: Local, HPC (SLURM, SGE, PBS), or cloud (Google Cloud, AWS, Azure)

Pipeline Repository Map

The official ENCODE pipelines are the WDL workflows below. The `pipeline-*` skills in this toolkit are independent Nextflow implementations that follow the same standards; they are written and maintained by the ENCODE Toolkit author, not by the ENCODE DCC. Each skill builds its own image from the `scripts/Dockerfile` it ships with.

| Assay | Official ENCODE pipeline (WDL) | Primary Tools | Toolkit skill (Nextflow) | |-------|-------------------------------|---------------|--------------------------| | ChIP-seq | `ENCODE-DCC/chip-seq-pipeline2` | BWA, MACS2, IDR | `pipeline-chipseq` | | ATAC-seq | `ENCODE-DCC/atac-seq-pipeline` | Bowtie2, MACS2, IDR | `pipeline-atacseq` | | RNA-seq | `ENCODE-DCC/rna-seq-pipeline` | STAR, RSEM | `pipeline-rnaseq` | | DNase-seq | `ENCODE-DCC/dnase-seq-pipeline` | BWA, Hotspot2 | `pipeline-dnaseseq` | | WGBS | `ENCODE-DCC/dna-me-pipeline` | Bismark, MethylDackel | `pipeline-wgbs` | | Hi-C | `ENCODE-DCC/hic-pipeline` | BWA, Juicer, HiCCUPS | `pipeline-hic` | | CUT&RUN | none published by ENCODE | Bowtie2, SEACR/MACS2 | `pipeline-cutandrun` | | scRNA-seq | see the ENCODE portal pipeline pages | STARsolo | — | | scATAC-seq | see the ENCODE portal pipeline pages | Chromap | — |

ENCODE also publishes images for some pipelines on Docker Hub, for example `encodedcc/chip-seq-pipeline:v2.2.1` and `encodedcc/atac-seq-pipeline:v2.2.0`. Those images are built for the WDL workflows and are not what the Nextflow skills here run.

Literature Foundation

| Reference | Year | Relevance | Citations | |-----------|------|-----------|-----------| | Di Tommaso et al. "Nextflow enables reproducible computational workflows" | 2017 | Nextflow workflow manager | ~2,800 | | Ewels et al. "The nf-core framework for community-curated bioinformatics pipelines" | 2020 | nf-core community pipelines | ~1,900 | | Kurtzer et al. "Singularity: Scientific containers for mobility of compute" | 2017 | Singularity containers for HPC | ~2,500 | | Merkel "Docker: lightweight Linux containers for consistent development and deployment" | 2014 | Docker containerization | ~3,000 | | ENCODE Project Consortium "Expanded encyclopaedias of DNA elements" | 2020 | ENCODE Phase 3 standards | ~1,200 | | Gruening et al. "Bioconda: sustainable and comprehensive software distribution" | 2018 | Bioconda packaging ecosystem | ~1,400 |

Pipeline Output Types by Assay

ChIP-seq Pipeline

| Output Type | Format | Description | Use For | |------------|--------|-------------|---------| | alignments | bam | Filtered, deduplicated | Reprocessing, visualization | | signal of unique reads | bigWig | Unique read signal | Genome browser | | fold change over control | bigWig | Normalized signal | Comparative visualization | | IDR thresholded peaks | bed narrowPeak | Reproducible peaks | Peak analysis (gold standard) | | pseudoreplicated peaks | bed narrowPeak | Single-replicate peaks | When only 1 replicate | | optimal IDR peaks | bed narrowPeak | Pooled replicate peaks | Most complete peak set |

ATAC-seq Pipeline

| Output Type | Format | Description | Use For | |------------|--------|-------------|---------| | alignments | bam | No-mito, deduplicated | Reprocessing | | signal of unique reads | bigWig | Signal track | Genome browser | | IDR thresholded peaks | bed narrowPeak | Reproducible peaks | Accessibility analysis | | pseudoreplicated peaks | bed narrowPeak | Single-replicate | Backup peaks |

RNA-seq Pipeline

| Output Type | Format | Description | Use For | |------------|--------|-------------|---------| | alignments | bam | STAR-aligned | Visualization, reprocessing | | gene quantifications | tsv | Gene-level counts (RSEM) | Differential expression | | transcript quantifications | tsv | Transcript-level counts | Isoform analysis | | signal of unique reads | bigWig | Strand-specific signal | Genome browser |

WGBS Pipeline

| Output Type | Format | Description | Use For | |-----