NCI Imaging Data Commons (idc-index)

Overview

NCI Imaging Data Commons (IDC) is the largest public collection of cancer imaging data, hosting 175+ DICOM collections (CT, MR, PET, slide microscopy, segmentations, structured reports). The idc-index Python client ships the entire IDC metadata catalog as a parquet file bundled inside the pip wheel; IDCClient() loads it into DuckDB, so sql_query() runs locally with zero network calls.

Image downloads stream from public AWS S3 (default) or Google Cloud Storage buckets via the bundled s5cmd executable. No GCP/AWS credentials, no BigQuery billing, no service account JSON.

When to Use

Searching publicly available cancer imaging datasets by modality, cancer type, anatomical site, or DICOM tag
Building reproducible ML cohorts (segmentation, classification, multimodal) from versioned IDC releases
Querying DICOM metadata at scale using SQL across all 175+ collections without any downloads
Downloading specific DICOM series for local processing or model training
Generating OHIF/Slim viewer URLs to share or inspect series interactively in a browser
Use pydicom-medical-imaging instead when you only need to read, edit, or anonymize DICOM files that you already have locally
For whole-slide pathology preprocessing (tiling, stain normalization) after download, use histolab instead

Prerequisites

Python packages: idc-index (>=0.12), pandas, pydicom (for reading DICOM files after download)
Data requirements: none for querying. For downloads, free disk space matching series_size_MB
Environment: no authentication required. All data is publicly accessible. The wheel bundles both the parquet index and the s5cmd executable used for high-speed S3 transfers
Rate limits: none for local SQL queries (DuckDB on local parquet). Bulk downloads are limited by network bandwidth, not by API quotas

# Skip when already provisioned in a pixi or conda env
pip install idc-index pydicom

Quick Start

from idc_index import IDCClient

client = IDCClient()
print('IDC version:', client.get_idc_version())
print('Collections:', len(client.get_collections()))
print('First 5:', client.get_collections()[:5])

df = client.sql_query("""
    SELECT collection_id, COUNT(DISTINCT SeriesInstanceUID) AS n_series
    FROM index
    WHERE Modality = 'CT'
    GROUP BY collection_id
    ORDER BY n_series DESC LIMIT 5
""")
print(df)

Core API

Module 1: Client Initialization and Collections

IDCClient() is the single entry point. It loads the parquet index, registers it as the DuckDB table named index, and validates the bundled s5cmd. get_collections() returns a plain Python list of lowercase collection IDs such as nsclc_radiomics, lidc_idri, and tcga_gbm.

from idc_index import IDCClient

client = IDCClient()
collections = client.get_collections()
print("total:", len(collections), "type:", type(collections).__name__)
print("lung-related:", [c for c in collections if "lung" in c or "nsclc" in c][:5])

Module 2: sql_query (DuckDB Over the Local Index)

The recommended cohort-selection API. The table name is index; additional tables (prior_versions_index and optional sm_index, clinical_index) are auto-registered when installed. Returns a pandas DataFrame. Use this for any filter involving Modality, BodyPartExamined, series_size_MB, or arbitrary DICOM tags. The legacy get_series(collection_id=..., modality=...) signature no longer exists in idc-index.

# Cohort: small CT series in NSCLC Radiomics, sorted by size for cheap testing
df = client.sql_query("""
    SELECT SeriesInstanceUID, StudyInstanceUID, PatientID, Modality, series_size_MB
    FROM index
    WHERE collection_id = 'nsclc_radiomics'
      AND Modality = 'CT'
    ORDER BY series_size_MB ASC
    LIMIT 5
""")
print(df[["PatientID", "Modality", "series_size_MB"]])

# Cross-collection lung CT count, using DuckDB ILIKE for case-insensitive matching
df = client.sql_query("""
    SELECT collection_id, COUNT(DISTINCT SeriesInstanceUID) AS n
    FROM index
    WHERE Modality = 'CT' AND BodyPartExamined ILIKE '%LUNG%'
    GROUP BY collection_id
    ORDER BY n DESC LIMIT 5
""")
print(df)

Module 3: Hierarchical Browsing (Patients, Studies, Series)

For DICOM-hierarchy navigation (Collection -> Patient -> Study -> Series), use the typed helpers. Each accepts an outputFormat of dict (default), df, or list. Note: get_dicom_series() takes a studyInstanceUID (not collection_id). Use sql_query() if you want to filter series by collection or modality.

patients = client.get_patients('nsclc_radiomics', outputFormat='df')
print("patients:", patients.shape, list(patients.columns)[:5])

# Walk down the hierarchy from one patient
pid = patients["PatientID"].iloc[0]
studies = client.get_dicom_studies(pid, outputFormat='df')
print("studies for", pid, ":", studies.shape)

study_uid = studies["StudyInstanceUID"].iloc[0]
series = client.get_dicom_series(study_uid, outputFormat='df')
print("series in study:", series.shape, "modalities:", series["Modality"].unique().tolist())

Module 4: download_from_selection (Modern Download Path)

The preferred download method. Accepts any combination of collection_id, patientId, studyInstanceUID, seriesInstanceUID, sopInstanceUID, or crdc_series_uuid (each a string or list). Filters apply in sequence. Use dry_run=True to size the cohort before pulling bytes. Files land under dirTemplate (default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID).

import tempfile, glob, os

# Pick the smallest CT series for a fast smoke test (about 40 MB)
df = client.sql_query("""
    SELECT SeriesInstanceUID FROM index
    WHERE collection_id = 'nsclc_radiomics' AND Modality = 'CT'
    ORDER BY series_size_MB ASC LIMIT 1
""")
uid = df["SeriesInstanceUID"].iloc[0]

out = tempfile.mkdtemp(prefix='idc_dl_')
client.download_from_selection(
    downloadDir=out,
    seriesInstanceUID=[uid],
    quiet=True,
    show_progress_bar=False,
    source_bucket_location='aws',  # 'aws' (default) or 'gcs'
)
files = glob.glob(os.path.join(out, "**/*.dcm"), recursive=True)
print(f"downloaded {len(files)} DICOM files to {out}")

# Size a cohort before downloading
client.download_from_selection(
    downloadDir='./preview',
    collection_id='nsclc_radiomics',
    dry_run=True,
)

Module 5: download_dicom_series (Convenience Wrapper)

Single-series download. Internally calls download_from_selection(seriesInstanceUID=...). Accepts a string or list of UIDs.

import tempfile, glob, os
out = tempfile.mkdtemp(prefix='idc_dl_')
client.download_dicom_series(
    seriesInstanceUID=uid,            # string or list
    downloadDir=out,
    quiet=True,
    show_progress_bar=False,
)
print("files:", len(glob.glob(os.path.join(out, "**/*.dcm"), recursive=True)))

Module 6: get_viewer_URL (Browser Visualization)

Returns a shareable URL to the IDC OHIF (radiology) or Slim (slide microscopy) viewer. Auto-selects the viewer based on modality if viewer_selector is omitted.

url = client.get_viewer_URL(seriesInstanceUID=uid)
print("Open in browser:", url)

# Force a specific viewer
url_v2 = client.get_viewer_URL(seriesInstanceUID=uid, viewer_selector='ohif_v2')
print(url_v2)

Module 7: Inspecting Downloaded DICOM with pydicom

After download, files are organized under dirTemplate. Use pydicom for header and pixel inspection.

import pydicom, glob, os
dcm_files = sorted(glob.glob(os.path.join(out, "**/*.dcm"), recursive=True))
ds = pydicom.dcmread(dcm_files[0])
print("PatientID:", ds.PatientID)
print("Modality :", ds.Modality)
print("Rows x Cols:", ds.Rows, "x", ds.Columns)
print("Pixel array:", ds.pixel_array.shape)

Key Concepts

Local-First, Auth-Free Architecture

The IDC client is unusual: client = IDCClient() is fully offline for queries. The parque

imaging-data-commons

Como adicionar

Cole no README do seu repo

Skills relacionadas

xlsx

how-it-works

weekly-digests

mem-search

Receba novas skills de Dados e Análise toda segunda