imaging-data-commons

Imaging Data Commons

Overview

Use the idc-index Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.

Current IDC Data Version: v23 (always verify with IDCClient().get_idc_version())

Primary tool: idc-index (GitHub)

CRITICAL - Check package version and upgrade if needed (run this FIRST):

import idc_index

REQUIRED_VERSION = "0.11.14"  # Must match metadata.idc-index in this file
installed = idc_index.__version__

if installed < REQUIRED_VERSION:
    print(f"Upgrading idc-index from {installed} to {REQUIRED_VERSION}...")
    import subprocess
    subprocess.run(["pip3", "install", "--upgrade", "--break-system-packages", "idc-index"], check=True)
    print("Upgrade complete. Restart Python to use new version.")
else:
    print(f"idc-index {installed} meets requirement ({REQUIRED_VERSION})")

Verify IDC data version and check current data scale:

from idc_index import IDCClient
client = IDCClient()

# Verify IDC data version (should be "v23")
print(f"IDC data version: {client.get_idc_version()}")

# Get collection count and total series
stats = client.sql_query("""
    SELECT
        COUNT(DISTINCT collection_id) as collections,
        COUNT(DISTINCT analysis_result_id) as analysis_results,
        COUNT(DISTINCT PatientID) as patients,
        COUNT(DISTINCT StudyInstanceUID) as studies,
        COUNT(DISTINCT SeriesInstanceUID) as series,
        SUM(instanceCount) as instances,
        SUM(series_size_MB)/1000000 as size_TB
    FROM index
""")
print(stats)

Core workflow:

1. Query metadata → client.sql_query()
2. Download DICOM files → client.download_from_selection()
3. Visualize in browser → client.get_viewer_URL(seriesInstanceUID=...)

When to Use This Skill

• Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images
• Selecting image subsets by cancer type, modality, anatomical site, or other metadata
• Downloading DICOM data from IDC
• Checking data licenses before use in research or commercial applications
• Visualizing medical images in a browser without local DICOM viewer software

Quick Navigation

Core Sections (inline):

• IDC Data Model - Collection and analysis result hierarchy
• Index Tables - Available tables and joining patterns
• Installation - Package setup and version verification
• Core Capabilities - Essential API patterns (query, download, visualize, license, citations, batch)
• Best Practices - Usage guidelines
• Troubleshooting - Common issues and solutions

Reference Guides (load on demand):

| Guide | When to Load | |-------|--------------| | index_tables_guide.md | Complex JOINs, schema discovery, DataFrame access | | use_cases.md | End-to-end workflow examples (training datasets, batch downloads) | | sql_patterns.md | Quick SQL patterns for filter discovery, annotations, size estimation | | clinical_data_guide.md | Clinical/tabular data, imaging+clinical joins, value mapping | | cloud_storage_guide.md | Direct S3/GCS access, versioning, UUID mapping | | dicomweb_guide.md | DICOMweb endpoints, PACS integration | | digital_pathology_guide.md | Slide microscopy (SM), annotations (ANN), pathology workflows | | bigquery_guide.md | Full DICOM metadata, private elements (requires GCP) | | cli_guide.md | Command-line tools (idc download, manifest files) | | parquet_access_guide.md | Direct Parquet queries via GCS (no idc-index install needed) |

IDC Data Model

IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):

• collection_id: Groups patients by disease, modality, or research focus (e.g., tcga_luad, nlst). A patient belongs to exactly one collection.
• analysis_result_id: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.

Use collection_id to find original imaging data, may include annotations deposited along with the images; use analysis_result_id to find AI-generated or expert annotations.

Key identifiers for queries: | Identifier | Scope | Use for | |------------|-------|---------| | collection_id | Dataset grouping | Filtering by project/study | | PatientID | Patient | Grouping images by patient | | StudyInstanceUID | DICOM study | Grouping of related series, visualization | | SeriesInstanceUID | DICOM series | Grouping of related series, visualization |

Index Tables

The idc-index package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.

Complete index table documentation: Use https://idc-index.readthedocs.io/en/latest/indices_reference.html for quick check of available tables and columns without executing any code.

Important: Use client.indices_overview to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.

Available Tables

| Table | Row Granularity | Loaded | Description | |-------|-----------------|--------|-------------| | index | 1 row = 1 DICOM series | Auto | Primary metadata for all current IDC data | | prior_versions_index | 1 row = 1 DICOM series | Auto | Series from previous IDC releases; for downloading deprecated data | | collections_index | 1 row = 1 collection | fetch_index() | Collection-level metadata and descriptions | | analysis_results_index | 1 row = 1 analysis result collection | fetch_index() | Metadata about derived datasets (annotations, segmentations) | | clinical_index | 1 row = 1 clinical data column | fetch_index() | Dictionary mapping clinical table columns to collections | | sm_index | 1 row = 1 slide microscopy series | fetch_index() | Slide Microscopy (pathology) series metadata | | sm_instance_index | 1 row = 1 slide microscopy instance | fetch_index() | Instance-level (SOPInstanceUID) metadata for slide microscopy | | seg_index | 1 row = 1 DICOM Segmentation series | fetch_index() | Segmentation metadata: algorithm, segment count, reference to source image series | | ann_index | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series | | ann_group_index | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm | | contrast_index | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) | | volume_geometry_index | 1 row = 1 CT/MR/PT series | fetch_index() | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite regularly_spaced_3d_volume flag | | rtstruct_index | 1 row = 1 RTSTRUCT series | fetch_index() | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |

Auto = loaded automatically when IDCClient() is instantiated fetch_index() = requires client.fetch_index("table_name") to load

Joining Tables

Key columns are not explicitly labeled, the following is a subset that can be used in joins.

| Join Column | Tables | Use Case | |-------------|--------|----------| | collection_id | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data | | SeriesInstanceUID | index, prior_versions_index, sm_index, sm_instance_index | Link series across tables; connect to slide microscopy details | | StudyInstanceUID | index, prior_versions_index | Link studies across current and historical data | | PatientID | index, prior_versions