ENCODE Pipeline Guide and Custom Workflow Generation
When to Use
- User wants to understand ENCODE uniform analysis pipelines or run them on their own data
- User asks about "ENCODE pipeline", "Nextflow", "WDL", "processing standards", or "pipeline requirements"
- User needs to generate a custom Nextflow/WDL workflow based on ENCODE pipeline specifications
- User wants to know compute requirements (CPU, GPU, memory, storage) for running pipelines
- Example queries: "how do I run the ENCODE ChIP-seq pipeline?", "what are the compute requirements for Hi-C processing?", "generate a Nextflow pipeline for my ATAC-seq data"
Understand ENCODE pipelines, generate user-specific workflows in Nextflow/WDL, and manage compute resources for local, HPC, and cloud execution.
ENCODE Uniform Analysis Pipelines
ENCODE uses standardized pipelines for each assay type, ensuring reproducibility across all datasets. All pipelines are:
- Open source: GitHub (github.com/ENCODE-DCC)
- Containerized: Docker and Singularity images
- Written in WDL: Workflow Description Language (Cromwell execution engine)
- Portable: Local, HPC (SLURM, SGE, PBS), or cloud (Google Cloud, AWS, Azure)
Pipeline Repository Map
| Assay | GitHub Repository | Primary Tools | Container |
|---|
| ChIP-seq | ENCODE-DCC/chip-seq-pipeline2 | BWA, MACS2, IDR | encodedcc/chip-seq-pipeline:v2.2.1 |
| ATAC-seq | ENCODE-DCC/atac-seq-pipeline | Bowtie2, MACS2, IDR | encodedcc/atac-seq-pipeline:v2.2.0 |
| RNA-seq | ENCODE-DCC/rna-seq-pipeline | STAR, RSEM | encodedcc/rna-seq-pipeline:v1.2.0 |
| DNase-seq | ENCODE-DCC/dnase-seq-pipeline | BWA, Hotspot2 | encodedcc/dnase-seq-pipeline |
| WGBS | ENCODE-DCC/dna-me-pipeline | Bismark/bwa-meth, MethylDackel | encodedcc/dna-me-pipeline |
| Hi-C | ENCODE-DCC/hic-pipeline | BWA, Juicer, HiCCUPS | encodedcc/hic-pipeline |
| scRNA-seq | ENCODE-DCC/scrna-seq-pipeline | STARsolo, Cellranger | — |
| scATAC-seq | ENCODE-DCC/scatac-seq-pipeline | Chromap, SnapATAC2 | — |
| CUT&RUN | ENCODE-DCC/cutandrun-pipeline | Bowtie2, SEACR/MACS2 | — |
Literature Foundation
| Reference | Year | Relevance | Citations |
|---|
| Di Tommaso et al. "Nextflow enables reproducible computational workflows" | 2017 | Nextflow workflow manager | ~2,800 |
| Ewels et al. "The nf-core framework for community-curated bioinformatics pipelines" | 2020 | nf-core community pipelines | ~1,900 |
| Kurtzer et al. "Singularity: Scientific containers for mobility of compute" | 2017 | Singularity containers for HPC | ~2,500 |
| Merkel "Docker: lightweight Linux containers for consistent development and deployment" | 2014 | Docker containerization | ~3,000 |
| ENCODE Project Consortium "Expanded encyclopaedias of DNA elements" | 2020 | ENCODE Phase 3 standards | ~1,200 |
| Gruening et al. "Bioconda: sustainable and comprehensive software distribution" | 2018 | Bioconda packaging ecosystem | ~1,400 |
Pipeline Output Types by Assay
ChIP-seq Pipeline
| Output Type | Format | Description | Use For |
|---|
| alignments | bam | Filtered, deduplicated | Reprocessing, visualization |
| signal of unique reads | bigWig | Unique read signal | Genome browser |
| fold change over control | bigWig | Normalized signal | Comparative visualization |
| IDR thresholded peaks | bed narrowPeak | Reproducible peaks | Peak analysis (gold standard) |
| pseudoreplicated peaks | bed narrowPeak | Single-replicate peaks | When only 1 replicate |
| optimal IDR peaks | bed narrowPeak | Pooled replicate peaks | Most complete peak set |
ATAC-seq Pipeline
| Output Type | Format | Description | Use For |
|---|
| alignments | bam | No-mito, deduplicated | Reprocessing |
| signal of unique reads | bigWig | Signal track | Genome browser |
| IDR thresholded peaks | bed narrowPeak | Reproducible peaks | Accessibility analysis |
| pseudoreplicated peaks | bed narrowPeak | Single-replicate | Backup peaks |
RNA-seq Pipeline
| Output Type | Format | Description | Use For |
|---|
| alignments | bam | STAR-aligned | Visualization, reprocessing |
| gene quantifications | tsv | Gene-level counts (RSEM) | Differential expression |
| transcript quantifications | tsv | Transcript-level counts | Isoform analysis |
| signal of unique reads | bigWig | Strand-specific signal | Genome browser |
WGBS Pipeline
| Output Type | Format | Description | Use For |
|---|
| alignments | bam | Bisulfite-converted | Reprocessing |
| methylation state at CpG | bed bedMethyl | Per-CpG levels | Methylation analysis |
Hi-C Pipeline
| Output Type | Format | Description | Use For |
|---|
| contact matrix | hic | Interaction frequencies | TAD/compartment calling |
| chromatin interactions | bedpe | Called loops | Loop analysis |
Choosing the Right Output Files
Decision Table
| Analysis Goal | File Type | Output Type | Priority |
|---|
| Visualization | bigWig | fold change over control (ChIP) / signal of unique reads (others) | preferred_default=True |
| Peak overlap | bed narrowPeak | IDR thresholded peaks | Highest confidence |
| Quantitative | tsv / bed | gene quantifications / methylation state | Pipeline defaults |
| Custom processing | fastq | reads | When ENCODE pipeline doesn't match |
encode_list_files(experiment_accession="ENCSR...", preferred_default=True)
Step 1: Assess User Compute Resources
Before generating any pipeline, check available resources:
System Check Commands
# CPU cores
nproc # Linux
sysctl -n hw.ncpu # macOS
# Memory
free -h # Linux
sysctl -n hw.memsize | awk '{print $1/1024/1024/1024 " GB"}' # macOS
# Disk space
df -h /path/to/data/
# GPU (if applicable)
nvidia-smi # NVIDIA GPU
# Note: Most ENCODE pipelines do NOT require GPU
# Docker availability
docker --version
docker info | grep "Total Memory"
# Singularity (for HPC)
singularity --version
Minimum Resource Requirements by Pipeline
| Pipeline | Min CPU | Min RAM | Min Disk | GPU | Time Estimate (per sample) |
|---|
| ChIP-seq | 4 cores | 16 GB | 50 GB | No | 2–4 hours |
| ATAC-seq | 4 cores | 16 GB | 50 GB | No | 2–4 hours |
| RNA-seq | 8 cores | 32 GB | 100 GB | No | 4–8 hours (index build) |
| WGBS | 8 cores | 48 GB | 200 GB | No | 12–24 hours |
| Hi-C | 8 cores | 64 GB | 200 GB | No | 8–16 hours |
| scRNA-seq | 8 cores | 64 GB | 100 GB | No | 4–8 hours |
Resource Scaling
- CPU: Alignment steps are parallelizable; doubling cores approximately halves alignment time
- RAM: Genome index loading is the bottleneck; STAR requires ~32 GB for human genome
- Disk: FASTQ + BAM + intermediate files can exceed 100 GB per sample
- Network: ENCODE downloads at ~50–200 MB/s; plan for transfer time
Step 2: Generate Custom Nextflow Workflows
When the user needs to run ENCODE-style processing, generate Nextflow workflows that mirror ENCODE pipeline logic.
Why Nextflow Over WDL
- Broader adoption: Nextflow is used by nf-core, most HPC centers, and cloud platforms
- Native container support: Docker, Singularity, Podman
- Cloud integration: AWS Batch, Google Cloud Life Sciences, Azure Batch natively
- Resource management: Built-in CPU/memory/time limits per process
- Resume capability: Failed runs restart from last successful step
Nextflow Pipeline Template
#!/usr/bin/env nextflow
nextflow.enable.dsl=2
// Pipeline parameters
params.reads = null // Input FASTQ path
params.genome =
