/imaging-data-commons

name: "imaging-data-commons"
description: "Query and download NCI Imaging Data Commons (IDC) cancer radiology and pathology datasets via the idc-index Python client. No authentication required: the parquet index ships inside the pip wheel, SQL runs locally via DuckDB, and DICOM downloads stream from public S3/GCS buckets through s5cmd. Use sql_query() for DuckDB cohort selection, get_collections/get_patients/get_dicom_studies/get_dicom_series for hierarchical browsing, download_from_selection() for downloads, and get_viewer_URL() for OHIF/Slim links. Use pydicom-medical-imaging for local DICOM reading; histolab for whole-slide pathology preprocessing."
license: "MIT"

NCI Imaging Data Commons (idc-index)

Overview

NCI Imaging Data Commons (IDC) is the largest public collection of cancer imaging data, hosting 175+ DICOM collections (CT, MR, PET, slide microscopy, segmentations, structured reports). The idc-index Python client ships the entire IDC metadata catalog as a parquet file bundled inside the pip wheel; IDCClient() loads it into DuckDB, so sql_query() runs locally with zero network calls.

Image downloads stream from public AWS S3 (default) or Google Cloud Storage buckets via the bundled s5cmd executable. No GCP/AWS credentials, no BigQuery billing, no service account JSON.

When to Use

• Searching publicly available cancer imaging datasets by modality, cancer type, anatomical site, or DICOM tag
• Building reproducible ML cohorts (segmentation, classification, multimodal) from versioned IDC releases
• Querying DICOM metadata at scale using SQL across all 175+ collections without any downloads
• Downloading specific DICOM series for local processing or model training
• Generating OHIF/Slim viewer URLs to share or inspect series interactively in a browser
• Use pydicom-medical-imaging instead when you only need to read, edit, or anonymize DICOM files that you already have locally
• For whole-slide pathology preprocessing (tiling, stain normalization) after download, use histolab instead

Prerequisites

• **Python packages**: `idc-index` (>=0.12), `pandas`, `pydicom` (for reading DICOM files after download)
• **Data requirements**: none for querying. For downloads, free disk space matching `series_size_MB`
• **Environment**: **no authentication required**. All data is publicly accessible. The wheel bundles both the parquet index and the `s5cmd` executable used for high-speed S3 transfers
• **Rate limits**: none for local SQL queries (DuckDB on local parquet). Bulk downloads are limited by network bandwidth, not by API quotas

# Skip when already provisioned in a pixi or conda env
pip install idc-index pydicom

Quick Start

from idc_index import IDCClient

client = IDCClient()
print('IDC version:', client.get_idc_version())
print('Collections:', len(client.get_collections()))
print('First 5:', client.get_collections()[:5])

df = client.sql_query("""
    SELECT collection_id, COUNT(DISTINCT SeriesInstanceUID) AS n_series
    FROM index
    WHERE Modality = 'CT'
    GROUP BY collection_id
    ORDER BY n_series DESC LIMIT 5
""")
print(df)

Core API

Module 1: Client Initialization and Collections

IDCClient() is the single entry point. It loads the parquet index, registers it as the DuckDB table named index, and validates the bundled s5cmd. get_collections() returns a plain Python list of lowercase collection IDs such as nsclc_radiomics, lidc_idri, and tcga_gbm.

from idc_index import IDCClient

client = IDCClient()
collections = client.get_collections()
print("total:", len(collections), "type:", type(collections).__name__)
print("lung-related:", [c for c in collections if "lung" in c or "nsclc" in c][:5])

Module 2: sql_query (DuckDB Over the Local Index)

The recommended cohort-selection API. The table name is index; additional tables (prior_versions_index and optional sm_index, clinical_index) are auto-registered when installed. Returns a pandas DataFrame. Use this for any filter involving Modality, BodyPartExamined, series_size_MB, or arbitrary DICOM tags. The legacy get_series(collection_id=..., modality=...) signature no longer exists in idc-index.

# Cohort: small CT series in NSCLC Radiomics, sorted by size for cheap testing
df = client.sql_query("""
    SELECT SeriesInstanceUID, StudyInstanceUID, PatientID, Modality, series_size_MB
    FROM index
    WHERE collection_id = 'nsclc_radiomics'
      AND Modality = 'CT'
    ORDER BY series_size_MB ASC
    LIMIT 5
""")
print(df[["PatientID", "Modality", "series_size_MB"]])

# Cross-collection lung CT count, using DuckDB ILIKE for case-insensitive matching
df = client.sql_query("""
    SELECT collection_id, COUNT(DISTINCT SeriesInstanceUID) AS n
    FROM index
    WHERE Modality = 'CT' AND BodyPartExamined ILIKE '%LUNG%'
    GROUP BY collection_id
    ORDER BY n DESC LIMIT 5
""")
print(df)

Module 3: Hierarchical Browsing (Patients, Studies, Series)

For DICOM-hierarchy navigation (Collection -> Patient -> Study -> Series), use the typed helpers. Each accepts an outputFormat of dict (default), df, or list. Note: get_dicom_series() takes a studyInstanceUID (not collection_id). Use sql_query() if you want to filter series by collection or modality.

patients = client.get_patients('nsclc_radiomics', outputFormat='df')
print("patients:", patients.shape, list(patients.columns)[:5])

# Walk down the hierarchy from one patient
pid = patients["PatientID"].iloc[0]
studies = client.get_dicom_studies(pid, outputFormat='df')
print("studies for", pid, ":", studies.shape)

study_uid = studies["StudyInstanceUID"].iloc[0]
series = client.get_dicom_series(study_uid, outputFormat='df')
print("series in study:", series.shape, "modalities:", series["Modality"].unique().tolist())

Module 4: download_from_selection (Modern Download Path)

The preferred download method. Accepts any combination of collection_id, patientId,