exploratory-data-analysis

Exploratory Data Analysis for Scientific Data

Overview

Exploratory data analysis (EDA) is the systematic examination of scientific data files to understand their structure, content, quality, and characteristics before formal analysis. This knowhow covers methodology for detecting file types, selecting appropriate analysis approaches, assessing data quality, and generating comprehensive reports across all major scientific data domains.

Key Concepts

Scientific Data Type Categories

| Category | Common Formats | Typical Analysis | Key Libraries | |----------|---------------|-----------------|---------------| | Tabular | CSV, TSV, XLSX, Parquet | Summary statistics, distributions, correlations, missing values | pandas, polars | | Sequence | FASTA, FASTQ, SAM/BAM | Length distribution, quality scores, GC content, alignment stats | BioPython, pysam | | Image/Microscopy | TIFF, ND2, CZI, DICOM | Dimensions (XYZCT), intensity stats, metadata, calibration | tifffile, aicsimageio, nd2reader | | Spectral | mzML, SPC, JCAMP, FID | Peak detection, baseline, S/N ratio, resolution | pymzml, nmrglue, pyteomics | | Structural | PDB, CIF, MOL, SDF | Atom counts, bond validation, B-factors, completeness | BioPython, RDKit, MDAnalysis | | Array/Tensor | NPY, HDF5, Zarr, NetCDF | Shape, dtype, value range, NaN/Inf check, chunk structure | numpy, h5py, zarr, xarray | | Omics | H5AD, MTX, VCF, BED | Feature/sample counts, sparsity, annotation completeness | scanpy, pyranges, cyvcf2 |

Format Detection Strategy

1. Extension-based: Map file extension to category (primary method)
2. Magic bytes: Check file header for binary format identification (HDF5: \x89HDF, GZIP: \x1f\x8b)
3. Content sniffing: For ambiguous extensions (.txt, .dat, .csv), inspect first lines for delimiters, headers, or format markers
4. Compound extensions: Handle .ome.tiff, .nii.gz, .tar.gz by checking from the rightmost extension inward

Data Quality Dimensions

• Completeness: Missing values, empty fields, truncated records
• Consistency: Matching dimensions, compatible dtypes, cross-field validation
• Validity: Values within expected ranges, proper encoding, format compliance
• Provenance: Instrument metadata, software versions, processing history
• Uniqueness: Duplicate records, redundant features

Decision Framework

Data file received
├── What is the file type?
│   ├── Known extension → Look up in format reference
│   ├── Unknown extension → Magic bytes / content sniffing
│   └── Directory (e.g., .d, .zarr) → Check internal structure
│
├── What category does it belong to?
│   ├── Tabular → Summary stats, distributions, correlations
│   ├── Sequence → Length/quality distributions, composition
│   ├── Image → Dimensions, channels, intensity, metadata
│   ├── Spectral → Peaks, baseline, resolution, S/N
│   ├── Structural → Atom/bond validation, geometry checks
│   ├── Array → Shape, dtype, value range, sparsity
│   └── Omics → Feature counts, sample QC, annotation check
│
├── How large is the file?
│   ├── Small (<100 MB) → Load fully, comprehensive analysis
│   ├── Medium (100 MB–1 GB) → Sample or lazy evaluation
│   └── Large (>1 GB) → Stream/chunk, representative sampling
│
└── What is the analysis goal?
    ├── Pre-pipeline QC → Focus on completeness, format compliance
    ├── Data understanding → Statistics, distributions, patterns
    ├── Troubleshooting → Compare against expected format/values
    └── Documentation → Full report with recommendations

Quick Reference: Analysis Approach by Data Type

| Data Type | First Check | Core Analysis | Visualization | |-----------|------------|---------------|---------------| | Tabular | dtypes, shape, nulls | describe(), correlations, outliers | histograms, scatter, heatmap | | Sequence | record count, format | length dist., quality, composition | quality plots, length histogram | | Image | dimensions, bit depth | intensity stats, channel info | thumbnail, histogram | | Spectral | scan count, m/z range | peak detection, TIC, baseline | spectrum plot, TIC chromatogram | | Structural | atom/residue count | B-factors, missing residues | Ramachandran, contact map | | Array | shape, dtype | statistics, NaN check | slice visualization | | Omics | genes × cells matrix | sparsity, QC metrics | violin plots, PCA |

Best Practices

1. Always check file integrity first — verify file is complete (not truncated) and readable before deep analysis. Check file size against expectations
2. Sample large files before full analysis — for files with millions of records, analyze a representative sample (first N records, random sample, or stratified sample) to get quick feedback
3. Use lazy/streaming readers when available — pl.scan_parquet(), h5py dataset slicing, pysam indexed access prevent memory overflows
4. Validate metadata against data — cross-check stated dimensions vs actual data, verify column count matches header, confirm timestamps are monotonic
5. Report data quality quantitatively — "5.2% missing values in column X" is more useful than "some missing values". Include completeness percentages, outlier counts, and format compliance scores
6. Consider data provenance — note instrument type, software version, processing steps, and any preprocessing already applied. This context affects downstream analysis choices
7. Generate actionable recommendations — don't just describe the data; suggest specific preprocessing steps (normalization method, imputation strategy), appropriate analyses, and potential issues to watch for
8. Preserve analysis reproducibility — include library versions, parameter choices, and random seeds in reports so EDA can be reproduced
9. Check for batch effects early — in multi-sample datasets, compare distributions across batches, plates, or runs before combining
10. Handle vendor-specific formats carefully — many instruments produce proprietary formats (.nd2, .czi, .raw). Document which reader library and version was used, as format support varies

Common Pitfalls

1. Assuming CSV means clean tabular data — CSV files can have inconsistent delimiters, mixed encodings, embedded newlines, or malformed quoting. How to avoid: Use pd.read_csv(engine='python') for robustness; check encoding with chardet

1. Ignoring missing value encoding — scientific data uses diverse null representations: NA, NaN, -999, empty string, #N/A, .. How to avoid: Specify na_values parameter; check for sentinel values in numeric columns

1. Drawing conclusions from truncated files — large file transfers can fail silently. How to avoid: Check file size, verify record counts against expected values, check for EOF markers

1. Applying wrong reader to file — some extensions are ambiguous (.raw = Thermo MS, XRD, or image; .d = Agilent directory or generic data). How to avoid: Use magic bytes and context (source instrument) to disambiguate

1. Memory overflow on large datasets — loading a 10 GB CSV into a pandas DataFrame will fail. How to avoid: Check file size first; use chunked reading, lazy evaluation, or sampling for files >100 MB

1. Ignoring coordinate systems and units — microscopy data may use pixels vs microns; spectroscopy data may use wavelength vs wavenumber vs energy. How to avoid: Extract and report units from metadata; verify calibration information

1. Treating all columns as independent — scientific tabular data often has hierarchical structure (replicates nested within conditions). How to avoid: Identify experimental design from column names and metadata before computing correlations

1. Skipping format-specific quality metrics — generic statistics miss domain-specific issues (e.g., Phred quality scores in FASTQ, R-factors in crystallography, mass accuracy in MS). How to avoid: Consult the format reference for domain-specific QC metrics