/vaex

name: vaex
description: Use this skill for processing and analyzing large tabular datasets (billions of rows) that exceed available RAM. Vaex excels at out-of-core DataFrame operations, lazy evaluation, fast aggregations, efficient visualization of big data, and machine learning on large datasets. Apply when users need to work with large CSV/HDF5/Arrow/Parquet files, perform fast statistics on massive datasets, create visualizations of big data, or build ML pipelines that do not fit in memory.
allowed-tools: Read Write Edit Bash Grep Glob
license: MIT license
metadata:
  version: "1.1"
  skill-author: K-Dense Inc.
compatibility: Requires Python 3.10+ (3.12+ recommended with vaex 4.19.0). Install with uv pip install vaex. Optional s3fs/gcsfs/adlfs for cloud I/O.

Vaex

Overview

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

Installation

Install the full meta-package (recommended):

uv pip install vaex

Minimal install (pick only what you need):

uv pip install vaex-core vaex-viz vaex-hdf5 vaex-ml

The `vaex` package is a meta-package that pulls in `vaex-core`, `vaex-viz`, `vaex-hdf5`, `vaex-ml`, and other sub-packages. Arrow support is built into `vaex-core` (the separate `vaex-arrow` package is deprecated). `vaex-distributed` is deprecated in favor of vaex-enterprise.

**Version notes (vaex 4.19.0+):** Python 3.12 and NumPy v2 require vaex >= 4.19.0. On Windows, you may need Python dev headers to build the `annoy` dependency.

When to Use This Skill

Use Vaex when:

• Processing tabular datasets larger than available RAM (gigabytes to terabytes)
• Performing fast statistical aggregations on massive datasets
• Creating visualizations and heatmaps of large datasets
• Building machine learning pipelines on big data
• Converting between data formats (CSV, HDF5, Arrow, Parquet)
• Needing lazy evaluation and virtual columns to avoid memory overhead
• Working with astronomical data, financial time series, or other large-scale scientific datasets

**Vaex vs alternatives:** Use **polars** when data fits in RAM and you need maximum in-memory speed. Use **dask** when you need distributed pandas/NumPy across a cluster. Use **vaex** for single-machine, out-of-core analytics on tabular data that exceeds RAM via memory-mapped HDF5/Arrow files.

Core Capabilities

Vaex provides six primary capability areas, each documented in detail in the references directory:

1. DataFrames and Data Loading

Load and create Vaex DataFrames from various sources including files (HDF5, CSV, Arrow, Parquet), pandas DataFrames, NumPy arrays, and dictionaries. Reference `references/core_dataframes.md` for:

• Opening large files efficiently
• Converting from pandas/NumPy/Arrow
• Working with example datasets
• Understanding DataFrame structure

2. Data Processing and Manipulation

Perform filtering, create virtual columns, use expressions, and aggregate data without loading everything into memory. Reference `references/data_processing.md` for:

• Filtering and selections
• Virtual columns and expressions
• Groupby operations and aggregations
• String operations and datetime handling
• Working with missing data

3. Performance and Optimization

Leverage Vaex's lazy evaluation, caching strategies, and memory-efficient operations. Reference `references/performance.md` for:

• Understanding lazy evaluation
• Using `delay=True` for batching operations
• Materializing columns when needed
• Caching strategies
• Asynchronous operations

4. Data Visualization

Create interactive visualizations of large datasets including heatmaps, histograms, and scatter plots. Reference `references/visualization.md` for:

• Creating 1D and 2D plots
• Heatmap visualizations
• Working with selections
• Customizing plots and subplots

5. Machine Learning Integration

Build ML pipelines with transformers, encoders, and integration with scikit-learn, XGBoost, and other frameworks. Reference `references/machine_learning.md` for:

• Feature scaling and encoding
• PCA and dimensionality reduction
• K-means clustering
• Integration with scikit-learn/XGBoost/CatBoost
• Model serialization and deployment

6. I/O Operations

Efficiently read and write data in various formats with optimal performance. Reference `references/io_operations.md` for:

• File format recommendations
• Export strategies
• Working with Apache Arrow
• CSV handling for large files
• Server and remote data access

Quick Start Pattern

For most Vaex tasks, follow this pattern:

import vaex

# 1. Open or create DataFrame
df = vaex.open('large_file.hdf5')  # or .csv, .arrow, .parquet
# OR
df = vaex.from_pandas(pandas_df)

# 2. Explore the data
print(df)  # Shows first/last rows and column info
df.describe()  # Statistical summary

# 3. Create virtual columns (no memory overhead)
df['new_column'] = df.x ** 2 + df.y

# 4. Filter with selections
df_filtered = df[df.age > 25]

# 5. Compute statistics (fast, lazy evaluation)
mean_val = df.x.mean()
stats = df.groupby('category').agg({'value': 'sum'})

# 6. Visualize (df.viz is the recommended accessor since vaex 4.0)
df.viz.heatmap(df.x, df.y, limits='99.7%', show=True)
# Legacy: df.plot1d() and df.plot() still work on the DataFrame

# 7. Export if needed
df.export_hdf5('output.hdf5')

Working with References

The reference files contain detailed information about each capability area. Load references into context based on the specific task:

• **Basic operations**: Start with `references/core_dataframes.md` and `references/data_processing.md`
• **Performance issues**: Check `references/performance.md`
• **Visualization tasks**: Use `references/visualization.md`
• **ML pipelines**: Reference `references/machine_learnin