data-pipeline-manager

Data Pipeline Manager

Purpose

The Data Pipeline Manager skill provides comprehensive guidance for designing, implementing, and troubleshooting data processing pipelines with emphasis on quality validation, error handling, and reliability. Data pipelines are a critical component of modern bioinformatics and data science workflows, yet they are a prevalent source of failures in automated systems. Poor data quality, format mismatches, missing files, and transient errors can cascade through pipelines, causing failures that are difficult to diagnose and fix.

This skill addresses these challenges by providing structured approaches to pipeline design, validation strategies at each stage, robust error handling patterns, and monitoring techniques. Whether you're building a new RNA-seq analysis pipeline, debugging a failed ETL job, or ensuring data quality across complex multi-stage workflows, this skill provides the frameworks and best practices needed for success.

The skill integrates closely with the bioinformatician skill (which implements specific bioinformatics pipelines) and the systems-architect skill (which designs the overall system architecture), providing the critical middle layer of pipeline orchestration, validation, and reliability.

When to Use This Skill

Use the Data Pipeline Manager skill when you need to:

Pipeline Design and Implementation

• Design new data processing pipelines from scratch
• Set up RNA-seq, ChIP-seq, or other genomics processing pipelines
• Build ETL (Extract, Transform, Load) workflows for data integration
• Create data transformation pipelines for machine learning preprocessing
• Implement batch processing systems for high-throughput data
• Design real-time streaming data pipelines

Debugging and Troubleshooting

• Debug failed pipeline runs and identify root causes
• Investigate data quality issues causing downstream failures
• Trace errors through multi-stage pipeline workflows
• Identify bottlenecks and performance issues
• Diagnose format mismatches between pipeline stages
• Resolve dependency and ordering issues in complex workflows

Quality Assurance

• Implement validation checks at each pipeline stage
• Ensure data quality and completeness throughout processing
• Verify format consistency between stages
• Validate output correctness and expected ranges
• Implement data integrity checks and checksums
• Monitor data quality metrics over time

Error Recovery and Reliability

• Implement retry logic for transient failures
• Design checkpoint and resume capabilities
• Handle partial failures in parallel processing
• Implement rollback mechanisms for failed stages
• Design idempotent pipeline operations
• Create robust error logging and alerting systems

Pipeline Optimization

• Optimize pipeline performance and resource usage
• Parallelize independent pipeline stages
• Reduce data transfer between stages
• Implement caching strategies for expensive operations
• Balance throughput and latency requirements
• Scale pipelines to handle increasing data volumes

Core Workflow

The Data Pipeline Manager follows a structured six-stage workflow that ensures robust, reliable pipeline operation:

1. Design Phase

Define Pipeline Architecture

Begin by clearly defining the pipeline's purpose, inputs, outputs, and transformations. Use a structured approach:

• Inputs: Specify all input data sources, formats, schemas, and expected volume
• Transformations: Document each processing stage, its purpose, and dependencies
• Outputs: Define output formats, destinations, and success criteria
• Dependencies: Map dependencies between stages to identify sequential vs. parallel operations
• Resources: Estimate compute, memory, and storage requirements for each stage
• Constraints: Identify time, cost, and resource constraints

Choose Pipeline Pattern

Select the appropriate pipeline pattern based on your requirements:

• Sequential: Each stage depends on the previous stage's output
• Parallel: Multiple independent branches process different data or stages
• Conditional: Branching logic based on data content or intermediate results
• Hybrid: Combination of patterns for complex workflows

Design for Observability

Build in observability from the start:

• Define metrics to track at each stage (records processed, time taken, errors)
• Plan logging strategy (what to log, where, at what level)
• Design monitoring dashboards and alerting thresholds
• Implement tracing for distributed pipeline components

2. Input Validation

Pre-Flight Checks

Before processing begins, validate all inputs thoroughly:

• Existence: Verify all input files and data sources exist and are accessible
• Format: Check file formats match expected specifications (FASTQ, BAM, CSV, JSON)
• Completeness: Ensure files are complete and not corrupted (checksums, file size)
• Schema: Validate data schema matches expected structure (columns, data types)
• Non-Empty: Verify files contain actual data, not just headers or empty content
• Encoding: Check text encoding is correct (UTF-8, ASCII)
• Permissions: Verify read/write permissions for all required paths

Content Validation

Beyond basic format checks, validate data content:

• Value Ranges: Check numeric values fall within expected ranges
• Required Fields: Ensure all mandatory fields are present and non-null
• Referential Integrity: Verify foreign key relationships and identifiers
• Consistency: Check related fields are consistent (e.g., start < end coordinates)
• Quality Metrics: For bioinformatics data, check quality scores, read lengths
• Sample Metadata: Verify sample identifiers match expected manifest

Early Failure Strategy

Fail fast on input validation errors rather than processing invalid data:

• Report all validation errors clearly with specific file, line, and field information
• Provide actionable error messages explaining what's wrong and how to fix it
• Stop processing before consuming expensive resources
• Log validation errors for tracking and trend analysis

3. Transform Phase

Stage-by-Stage Processing

Execute each transformation stage with careful attention to reliability:

• Isolation: Process each stage independently with clear input/output contracts
• Error Handling: Wrap transformations in try-catch blocks with specific error handling
• Progress Tracking: Log progress at regular intervals (every N records or time period)
• Resource Monitoring: Track memory, CPU, and disk usage to detect resource issues
• Intermediate Validation: Validate data between stages to catch errors early
• Checkpointing: Save intermediate results for resumability after failures

Transformation Best Practices

Follow these practices for robust transformations:

• Idempotency: Design operations that can be safely retried without side effects
• Atomic Operations: Make changes atomically where possible (write to temp, then rename)
• Batch Processing: Process data in appropriately sized batches for memory efficiency
• Error Context: Capture rich error context (input data, parameters, environment)
• Graceful Degradation: Continue processing other data when individual records fail
• Type Safety: Use strong typing and validation to prevent type errors

Handling Large Data

For large-scale data processing:

• Stream data rather than loading entire datasets into memory
• Use chunking or windowing for processing large files
• Implement backpressure mechanisms to prevent overwhelming downstream stages
• Use efficient file formats (Parquet, Arrow) for intermediate data
• Consider distributed processing frameworks (Spark, Dask) for very large datasets

4. Output Validation

Structure Validation

Verify output structure meets specifications:

• Format: Confirm output files are in the correct format
• **Schema