/dask

name: dask
description: "Parallel/distributed computing. Scale pandas/NumPy beyond memory, parallel DataFrames/Arrays, multi-file processing, task graphs, for larger-than-RAM datasets and parallel workflows."

Dask

Overview

Dask is a Python library for parallel and distributed computing that enables three critical capabilities:

• **Larger-than-memory execution** on single machines for data exceeding available RAM
• **Parallel processing** for improved computational speed across multiple cores
• **Distributed computation** supporting terabyte-scale datasets across multiple machines

Dask scales from laptops (processing ~100 GiB) to clusters (processing ~100 TiB) while maintaining familiar Python APIs.

When to Use This Skill

This skill should be used when:

• Process datasets that exceed available RAM
• Scale pandas or NumPy operations to larger datasets
• Parallelize computations for performance improvements
• Process multiple files efficiently (CSVs, Parquet, JSON, text logs)
• Build custom parallel workflows with task dependencies
• Distribute workloads across multiple cores or machines

Core Capabilities

Dask provides five main components, each suited to different use cases:

1. DataFrames - Parallel Pandas Operations

**Purpose**: Scale pandas operations to larger datasets through parallel processing.

**When to Use**:

• Tabular data exceeds available RAM
• Need to process multiple CSV/Parquet files together
• Pandas operations are slow and need parallelization
• Scaling from pandas prototype to production

**Reference Documentation**: For comprehensive guidance on Dask DataFrames, refer to `references/dataframes.md` which includes:

• Reading data (single files, multiple files, glob patterns)
• Common operations (filtering, groupby, joins, aggregations)
• Custom operations with `map_partitions`
• Performance optimization tips
• Common patterns (ETL, time series, multi-file processing)

**Quick Example**:

import dask.dataframe as dd

# Read multiple files as single DataFrame
ddf = dd.read_csv('data/2024-*.csv')

# Operations are lazy until compute()
filtered = ddf[ddf['value'] > 100]
result = filtered.groupby('category').mean().compute()

**Key Points**:

• Operations are lazy (build task graph) until `.compute()` called
• Use `map_partitions` for efficient custom operations
• Convert to DataFrame early when working with structured data from other sources

2. Arrays - Parallel NumPy Operations

**Purpose**: Extend NumPy capabilities to datasets larger than memory using blocked algorithms.

**When to Use**:

• Arrays exceed available RAM
• NumPy operations need parallelization
• Working with scientific datasets (HDF5, Zarr, NetCDF)
• Need parallel linear algebra or array operations

**Reference Documentation**: For comprehensive guidance on Dask Arrays, refer to `references/arrays.md` which includes:

• Creating arrays (from NumPy, random, from disk)
• Chunking strategies and optimization
• Common operations (arithmetic, reductions, linear algebra)
• Custom operations with `map_blocks`
• Integration with HDF5, Zarr, and XArray

**Quick Example**:

import dask.array as da

# Create large array with chunks
x = da.random.random((100000, 100000), chunks=(10000, 10000))

# Operations are lazy
y = x + 100
z = y.mean(axis=0)

# Compute result
result = z.compute()

**Key Points**:

• Chunk size is critical (aim for ~100 MB per chunk)
• Operations work on chunks in parallel
• Rechunk data when needed for efficient operations
• Use `map_blocks` for operations not available in Dask

3. Bags - Parallel Processing of Unstructured Data

**Purpose**: Process unstructured or semi-structured data (text, JSON, logs) with functional operations.

**When to Use**:

• Processing text files, logs, or JSON records
• Data cleaning and ETL before structured analysis
• Working with Python objects that don't fit array/dataframe formats
• Need memory-efficient streaming processing

**Reference Documentation**: For comprehensive guidance on Dask Bags, refer to `references/bags.md` which includes:

• Reading text and JSON files
• Functional operations (map, filter, fold, groupby)
• Converting to DataFrames
• Common patterns (log analysis, JSON processing, text processing)
• Performance considerations

**Quick Example**:

import dask.bag as db
import json

# Read and parse JSON files
bag = db.read_text('logs/*.json').map(json.loads)

# Filter and transform
valid = bag.filter(lambda x: x['status'] == 'valid')
processed = valid.map(lambda x: {'id': x['id'], 'value': x['value']})

# Convert to DataFrame for analysis
ddf = processed.to_dataframe()

**Key Points**:

• Use for initial data cleaning, then convert to DataFrame/Array
• Use `foldby` instead of `groupby` for better performance
• Operations are streaming and memory-efficient
• Convert to structured formats (DataFrame) for complex operations

4. Futures - Task-Based Parallelization

**Purpose**: Build custom parallel workflows with fine-grained control over task execution and dependencies.

**When to Use**:

• Building dynamic, evolving workflows
• Need immediate task execution (not lazy)
• Computations depend on runtime conditions
• Implementing custom parallel algorithms
• Need stateful computations

**Reference Documentation**: For comprehensive guidance on Dask Futures, refer to `references/futures.md` which includes:

• Setting up distributed client
• Submitting tasks and working with futures
• Task dependencies and data movement
• Advanced coordination (queues, locks, events, actors)
• Common patterns (parameter sweeps, dynamic tasks, iterative algorithms)

**Quick Example**:

from dask.distributed import Client

client = Client()  # Create local cluster

# Submit tasks (executes immediately)
def process(x):
    return x ** 2

futures = client.map(process, range(100))

# Gather results
results = client.gather(futures)

client.close()

**Key Points**:

• Requires distributed cli