nextflow

Nextflow

Overview

Nextflow is a workflow language and runtime for building reproducible, portable, scalable data pipelines. It is dominant in bioinformatics but works for any data-heavy computation. nf-core is a community curating production-grade Nextflow pipelines, reusable modules, and the nf-core tooling on top of Nextflow.

Key ideas:

• Dataflow programming: pipelines are process tasks connected by channels. Nextflow infers execution order and parallelism from data dependencies — there is no explicit scheduler to write.
• Write once, run anywhere: the same pipeline runs locally, on HPC (SLURM, SGE, LSF, PBS), and on cloud (AWS Batch, Google Batch, Azure Batch, Kubernetes) by changing config/profiles, not code.
• Reproducibility: per-task containers (Docker/Singularity/Apptainer/Conda/Wave) + -resume caching + pinned pipeline revisions.
• DSL2 is the modern, required syntax: modular process/workflow/include definitions.

This skill covers both running existing pipelines and developing your own (Nextflow language + nf-core conventions, testing with nf-test, configuration, and deployment).

When to Use This Skill

Use this skill when the user wants to:

• Run an nf-core or custom Nextflow pipeline, or debug a failing/resuming run.
• Write or modify .nf scripts, nextflow.config, profiles, or nextflow_schema.json.
• Author or test nf-core-style modules/subworkflows (main.nf, meta.yml, tests/, nf-test).
• Configure executors, containers, or resources; scale to HPC or cloud.
• Build a reproducible scientific/bioinformatics workflow (even if "Nextflow" is not named).
• Understand processes, channels, operators, take/emit, publishDir, ext.args, meta maps.

Setup

Nextflow needs Bash and Java 17 or newer (17–25 supported). Verify with java -version.

# Install Nextflow (self-contained launcher)
curl -s https://get.nextflow.io | bash      # creates ./nextflow
sudo mv nextflow /usr/local/bin/             # put on PATH
nextflow info                                # verify

# Or via conda/bioconda (also gets a managed Java)
conda create -n nf -c bioconda -c conda-forge nextflow nf-core

# nf-core tools (Python) for creating/linting/running nf-core assets
pip install nf-core            # or: conda install -c bioconda nf-core
nf-core --version

Pin the engine for reproducibility: export NXF_VER=24.10.0 (use an [edge] release only if needed). For air-gapped/HPC, see references/running-pipelines.md (offline mode) and references/configuration.md.

Two Modes of Work

Decide which path the user is on — it changes everything:

| Goal | Start here | |------|-----------| | Run an existing pipeline (nf-core or a .nf you were given) | references/running-pipelines.md | | Develop a new pipeline / module / subworkflow | references/language.md + references/developing.md | | Configure / scale (HPC, cloud, containers, resources) | references/configuration.md + references/containers.md | | Test modules/pipelines | references/testing.md |

Quick Start

Run an nf-core pipeline

Always smoke-test with the bundled test profile first; it uses tiny data and proves your environment works.

# 1. Confirm setup works (downloads pipeline + tiny test data)
nextflow run nf-core/rnaseq -profile test,docker --outdir results

# 2. Real run: pin a revision (-r), pick a container engine, pass inputs
nextflow run nf-core/rnaseq -r 3.14.0 \
  -profile docker \
  --input samplesheet.csv \
  --genome GRCh38 \
  --outdir results \
  -resume

• -profile (single dash) selects bundled config profiles; combine them comma-separated, e.g. test,docker. Container/infra profiles (docker, singularity, conda) are mutually exclusive — pick one.
• --input, --genome, --outdir (double dash) are pipeline parameters. nf-core pipelines take a samplesheet CSV, not loose files.
• -resume reuses cached results from the last run. -r <version> pins a release for reproducibility.

Use nf-core pipelines launch <name> for an interactive, schema-validated way to build the command and a -params-file. See references/running-pipelines.md.

Write a minimal pipeline

#!/usr/bin/env nextflow

process SAYHELLO {
    tag "$greeting"
    publishDir "results", mode: 'copy'

    input:
    val greeting

    output:
    path "${greeting}.txt"

    script:
    """
    echo '$greeting world' > ${greeting}.txt
    """
}

workflow {
    channel.of('hello', 'bonjour', 'hola') | SAYHELLO
}

nextflow run main.nf            # add -resume on reruns

The full language (processes, channels, operators, DSL2 workflows with take/main/emit, modules) is in references/language.md.

Core Concepts at a Glance

• Process: a unit of work that runs a script (Bash by default). Declares input:, output:, optional directives (resources, container, publishDir, tag, errorStrategy), and a script:/shell:/exec: block. Each task runs in its own isolated work directory (work/xx/yy…).
• Channel: the async queues that connect processes. Queue channels are consumable streams; value channels hold a single reusable value. Created with factories like channel.of, channel.fromPath, channel.fromFilePairs, channel.value.
• Operator: transforms/combines channels — map, filter, collect, groupTuple, join, combine, mix, flatten, branch, multiMap, splitCsv, view, set.
• Workflow: composes processes. DSL2 workflows can declare take: (inputs), main: (logic), emit: (named outputs) and be included as subworkflows. The unnamed workflow {} is the entry point.
• Module: a .nf file exposing processes/workflows via include { NAME } from './path' (supports as aliasing).
• Configuration: nextflow.config sets params, process directives, executor, container engines, and named profiles. Selectors withName:/withLabel: target specific processes. See references/configuration.md.
• meta map (nf-core): the convention of carrying a metadata map ([ id:'sample1', single_end:false ]) alongside files in input/output tuples so samples stay labeled through the pipeline. See references/developing.md.

nf-core tools CLI

nf-core tools (v3+) group subcommands under pipelines, modules, and subworkflows. (Bare forms like nf-core lint still work but warn — prefer the grouped form.)

| Command | Purpose | |---------|---------| | nf-core pipelines list | List/search nf-core pipelines (--json, keywords) | | nf-core pipelines create | Scaffold a new pipeline from the nf-core template | | nf-core pipelines launch <name> | Interactive, schema-driven run command + params file | | nf-core pipelines download <name> | Download pipeline + containers for offline/HPC use | | nf-core pipelines lint | Lint a pipeline against nf-core standards (run in repo root) | | nf-core pipelines schema build | Build/edit nextflow_schema.json via web GUI | | nf-core pipelines create-params-file <name> | Generate a documented YAML params file | | nf-core pipelines bump-version / sync | Bump version / sync with template updates | | nf-core modules list/info/install/update/remove | Manage modules from nf-core/modules | | nf-core modules create / lint / test | Author, lint, and nf-test a module | | nf-core modules patch / bump-versions | Patch an installed module / bump tool versions | | nf-core subworkflows install/create/lint/test | Same lifecycle for subworkflows |

Full command reference, flags, and examples: references/nf-core-tools.md.

Essential nextflow CLI

| Command | Purpose | |---------|---------| | nextflow run <pipeline> -profile <p> --outdir <dir> | Run a pipeline (path, .nf, or user/repo) | | -resume | Reuse cached results from prior run | | -r <rev> | Run a specific git revision/tag/branch | | `-pa