/dask

name: dask
description: Distributed computing for larger-than-RAM pandas/NumPy workflows. Use when you need to scale existing pandas/NumPy code beyond memory or across clusters. Best for parallel file processing, distributed ML, integration with existing pandas code. For out-of-core analytics on single machine use vaex; for in-memory speed use polars.
allowed-tools: Read Write Edit Bash
license: BSD-3-Clause license
compatibility: Requires Python 3.10+ and dask 2025.1+. DataFrame workflows need pandas 2+ and PyArrow 16+. Cloud paths (s3://, gcs://) need s3fs or gcsfs. Cluster deployment uses dask.distributed (included with dask[complete]).
metadata:
  version: "1.2"
  skill-author: K-Dense Inc.

Dask

Overview

Dask is a Python library for parallel and distributed computing that enables three critical capabilities:

• **Larger-than-memory execution** on single machines for data exceeding available RAM
• **Parallel processing** for improved computational speed across multiple cores
• **Distributed computation** supporting terabyte-scale datasets across multiple machines

Dask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs.

**Current upstream:** dask **2026.3.0** (PyPI, March 2026). Docs: [docs.dask.org](https://docs.dask.org/en/stable/). Since **2025.1.0**, the expression-based DataFrame API with query planning is the only implementation — do not install `dask-expr` separately or set `dataframe.query-planning: False`.

Quick Start

Installation

uv pip install "dask>=2025.1"

For a typical pandas/NumPy workflow with the distributed scheduler and dashboard:

uv pip install "dask[complete]"

Remote object storage (S3, GCS, Azure):

uv pip install s3fs    # s3:// paths
uv pip install gcsfs   # gs:// paths

Requires **Python 3.10+** (3.9 support dropped in 2024.12). DataFrame I/O requires **PyArrow 16+** (as of dask 2026.1.2).

When to Use This Skill

This skill should be used when:

• Process datasets that exceed available RAM
• Scale pandas or NumPy operations to larger datasets
• Parallelize computations for performance improvements
• Process multiple files efficiently (CSVs, Parquet, JSON, text logs)
• Build custom parallel workflows with task dependencies
• Distribute workloads across multiple cores or machines

Core Capabilities

Dask provides five main components, each suited to different use cases:

1. DataFrames - Parallel Pandas Operations

**Purpose**: Scale pandas operations to larger datasets through parallel processing.

**When to Use**:

• Tabular data exceeds available RAM
• Need to process multiple CSV/Parquet files together
• Pandas operations are slow and need parallelization
• Scaling from pandas prototype to production

**Reference Documentation**: For comprehensive guidance on Dask DataFrames, refer to `references/dataframes.md` which includes:

• Reading data (single files, multiple files, glob patterns)
• Common operations (filtering, groupby, joins, aggregations)
• Custom operations with `map_partitions`
• Performance optimization tips
• Common patterns (ETL, time series, multi-file processing)

**Quick Example**:

import dask.dataframe as dd

# Read multiple files as single DataFrame
ddf = dd.read_csv('data/2024-*.csv')

# Operations are lazy until compute()
filtered = ddf[ddf['value'] > 100]
result = filtered.groupby('category').mean().compute()

**Key Points**:

• Operations are lazy (build task graph) until `.compute()` called
• Use `map_partitions` for efficient custom operations
• Convert to DataFrame early when working with structured data from other sources

2. Arrays - Parallel NumPy Operations

**Purpose**: Extend NumPy capabilities to datasets larger than memory using blocked algorithms.

**When to Use**:

• Arrays exceed available RAM
• NumPy operations need parallelization
• Working with scientific datasets (HDF5, Zarr, NetCDF)
• Need parallel linear algebra or array operations

**Reference Documentation**: For comprehensive guidance on Dask Arrays, refer to `references/arrays.md` which includes:

• Creating arrays (from NumPy, random, from disk)
• Chunking strategies and optimization
• Common operations (arithmetic, reductions, linear algebra)
• Custom operations with `map_blocks`
• Integration with HDF5, Zarr, and XArray

**Quick Example**:

import dask.array as da

# Create large array with chunks
x = da.random.random((100000, 100000), chunks=(10000, 10000))

# Operations are lazy
y = x + 100
z = y.mean(axis=0)

# Compute result
result = z.compute()

**Key Points**:

• Chunk size is critical (aim for ~100 MB per chunk)
• Operations work on chunks in parallel
• Rechunk data when needed for efficient operations
• Use `map_blocks` for operations not available in Dask

3. Bags - Parallel Processing of Unstructured Data

**Purpose**: Process unstructured or semi-structured data (text, JSON, logs) with functional operations.

**When to Use**:

• Processing text files, logs, or JSON records
• Data cleaning and ETL before structured analysis
• Working with Python objects that don't fit array/dataframe formats
• Need memory-efficient streaming processing

**Reference Documentation**: For comprehensive guidance on Dask Bags, refer to `references/bags.md` which includes:

• Reading text and JSON files
• Functional operations (map, filter, fold, groupby)
• Converting to DataFrames
• Common patterns (log analysis, JSON processing, text processing)
• Performance considerations

**Quick Example**:

import dask.bag as db
import json

# Read and parse JSON files
bag = db.read_text('logs/*.json').map(json.loads)

# Filter and transform
valid = bag.filter(lambda x: x['status'] == 'valid')
processed = valid.map(lambda x: {'id': x['id'], 'value': x['value']})

# Convert to DataFrame for analysis
ddf = processed.to_dataframe()

**Key Points**:

• Use for initial data cleaning, then convert to DataFrame/Arr