/pipeline-hic

name: pipeline-hic
description: "Execute ENCODE Hi-C pipeline from FASTQ to contact matrices and loop calls. Child of pipeline-guide. Provides Nextflow execution with Docker and cloud deployment. Use when processing Hi-C data, generating contact matrices, or calling loops. Trigger on: Hi-C pipeline, chromatin conformation, contact matrix, loop calling, TAD detection, Juicer, HiCCUPS, 3D genome."

ENCODE Hi-C Pipeline: FASTQ to Contact Matrices and Loops

When to Use

• User wants to run a Hi-C processing pipeline from FASTQ to contact matrices and loop calls
• User asks about "Hi-C pipeline", "contact matrix", "loop calling", "Juicer", "HiCCUPS", or "TAD detection"
• User needs to process Hi-C data for 3D genome structure analysis
• Example queries: "process my Hi-C FASTQs", "generate contact matrices from Hi-C", "call chromatin loops with HiCCUPS"

Execute the ENCODE Hi-C pipeline for chromatin conformation capture data, producing multi-resolution contact matrices and loop calls.

Pipeline Overview

FASTQ -> FastQC (raw reads)
      -> bwa mem -SP5M (both mates in one call) -> {sample}.paired.bam
         -> pairtools parse -> sort -> dedup -> select UU
              |
              +-> Juicer pre -> .hic -> HiCCUPS -> loops (BEDPE)
              |
              +-> cooler cload + zoomify -> .mcool

Not run by this workflow

Adapter trimming, TAD calling, A/B compartment calling, and every cooltools analysis are outside this workflow. They are documented as manual, optional steps only: trimming in `references/01-qc-trimming.md`, distance decay and compartments in `references/04-matrix-generation.md`. cooltools and bedtools are not installed in the container image, so those manual commands need the conda environment (`environments/hic-env.yml` in the `bioinformatics-installer` skill) or a separate install.

ENCODE Repository

• **GitHub**: `ENCODE-DCC/hic-pipeline`
• **Container**: built from `scripts/Dockerfile` in this skill (`docker build -t encode-toolkit/pipeline-hic:1.0.0 scripts/`); override with `--container`
• **WDL**: Available for Cromwell execution
• **This skill**: Nextflow DSL2 reimplementation for portability

Core Tools and Versions

Versions are those installed by `scripts/Dockerfile`, which is what the workflow runs.

| Tool | Version | Purpose | Citation | |------|---------|---------|----------| | BWA-MEM | 0.7.18 | Alignment (both mates, `-SP5M`) | Li & Durbin 2009 | | pairtools | 1.1.2 | Pair classification, dedup | Open2C | | Juicer tools | 2.20.00 | .hic generation, HiCCUPS | Durand et al. 2016 | | cooler | 0.9.3 | .cool/.mcool generation | Abdennur & Mirny 2020 | | samtools | 1.19 | BAM operations | Li et al. 2009 | | FastQC | 0.12.1 | Read quality | Andrews (Babraham) | | MultiQC | 1.21 | Aggregated QC | Ewels et al. 2016 |

The conda alternative (`hic-env.yml`) ships no juicer_tools jar, only a JRE, so `.hic` generation and HiCCUPS are unavailable on that route; conversely it provides cooltools and bedtools, which the container image does not.

Key Literature

1. **Rao et al. 2014** - "A 3D Map of the Human Genome at Kilobase Resolution Reveals Principles of Chromatin Looping" (Cell, ~5,000 citations) DOI: 10.1016/j.cell.2014.11.021

2. **Lieberman-Aiden et al. 2009** - "Comprehensive Mapping of Long-Range Interactions Reveals Folding Principles of the Human Genome" (Science, ~6,000 citations) DOI: 10.1126/science.1181369

3. **Durand et al. 2016** - "Juicer Provides a One-Click System for Analyzing Loop-Resolution Hi-C Experiments" (Cell Systems, ~2,000 citations) DOI: 10.1016/j.cels.2016.07.002

4. **Abdennur & Mirny 2020** - "Cooler: scalable storage for Hi-C data and other genomically labeled arrays" (Bioinformatics) DOI: 10.1093/bioinformatics/btz540

5. **Amemiya et al. 2019** - "The ENCODE Blacklist" (Scientific Reports, ~1,372 citations) DOI: 10.1038/s41598-019-45839-z

Execution

Quick Start (Local)

nextflow run scripts/main.nf \
    -profile local \
    --reads '/data/fastq/*_R{1,2}.fastq.gz' \
    --bwa_index '/ref/bwa_index/GRCh38.fa' \
    --chrom_sizes '/ref/hg38.chrom.sizes' \
    --outdir results/ \
    -resume

SLURM HPC

nextflow run scripts/main.nf \
    -profile slurm \
    --container /path/to/pipeline-hic.sif \
    --reads '/data/fastq/*_R{1,2}.fastq.gz' \
    --bwa_index '/ref/bwa_index/GRCh38.fa' \
    --chrom_sizes '/ref/hg38.chrom.sizes' \
    --outdir results/ \
    -resume

Cloud (GCP / AWS)

# Google Cloud Batch
nextflow run scripts/main.nf -profile gcp \
    --container us-docker.pkg.dev/<project>/<repo>/pipeline-hic:1.0.0 \
    --gcp_project <project> \
    --gcp_workdir gs://<bucket>/work \
    --reads 'gs://<bucket>/fastq/*_R{1,2}.fastq.gz' \
    --bwa_index gs://<bucket>/ref/GRCh38.fa \
    --chrom_sizes gs://<bucket>/ref/hg38.chrom.sizes \
    --outdir gs://<bucket>/results

# AWS Batch
nextflow run scripts/main.nf -profile aws \
    --container <account>.dkr.ecr.<region>.amazonaws.com/pipeline-hic:1.0.0 \
    --aws_queue <job-queue> \
    --aws_workdir s3://<bucket>/work \
    --reads 's3://<bucket>/fastq/*_R{1,2}.fastq.gz' \
    --bwa_index s3://<bucket>/ref/GRCh38.fa \
    --chrom_sizes s3://<bucket>/ref/hg38.chrom.sizes \
    --outdir s3://<bucket>/results

`--outdir` only sets where results are published; Google Batch and AWS Batch stage every task through the work directory, and the workflow stops with an error if it or the project/queue is missing.

Resource Requirements

| Step | CPUs | RAM | Time (2B contacts) | |------|------|-----|---------------------| | BWA alignment | 8 | 16 GB | 4-6 hours | | pairtools parse + sort | 4 | 16 GB | 2-3 hours | | pairtools dedup | 4 | 16 GB | 1-2 hours | | Juicer pre + hic | 4 | 64 GB | 2-4 hours | | HiCCUPS | 4 | 16 GB | 1-2 hours | | **Total** | **8** | **64 GB** | **8-16 hours** |

The RAM column is each step's first-attempt request. Every process asks for that m