/vaex

name: vaex
description: Processes large tabular scientific datasets with Vaex expressions, filtered views, streamed statistics, binned visualizations, and file conversion. Use for larger-than-RAM HDF5, Arrow, CSV, or Parquet analysis, virtual feature engineering, or Vaex ML preprocessing; distinguishes these operations from estimators and conversions that materialize data.
allowed-tools: Read Write Edit Bash Grep Glob
license: MIT license
compatibility: Requires Python 3.9-3.12 for vaex-core 4.19.0; tested on Python 3.12. Install vaex-core plus vaex-hdf5, vaex-viz, or vaex-ml as needed. Package installation and remote data need network access; local workflows need no credentials.
metadata:
  version: "1.3"
  skill-author: K-Dense Inc.
  last-reviewed: "2026-10-01"

Vaex

When to use

Use Vaex for columnar analysis on a single machine when data exceeds RAM, especially repeated reductions and histograms over local Vaex HDF5 or Arrow files. Expressions and virtual columns defer computation; reductions normally execute immediately. Out-of-core storage does not make every operation memory bounded: sorting, joins, large group dictionaries, materialization, and many estimator fits need substantial RAM.

Installation and verified scope

Use a separate environment; the repository's default Python is newer than this release supports:

uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex/bin/python "vaex-core==4.19.0" "vaex-hdf5==0.15.0" "vaex-viz==0.6.0"
# Optional ML (also installs its declared estimator dependencies):
uv pip install --python .venv-vaex/bin/python "vaex-ml==0.19.0"

On Windows use `.venv-vaex\Scripts\python.exe` as the interpreter path. The `vaex` 4.19.0 metapackage installs more integrations; it is not needed for the core workflow. Core 4.19.0 declares Python `>=3.9,<3.13`, pandas `<3`, Dask `<2024.9`, and NumPy `<3`. Do not upgrade these constraints independently. Arrow support is in core; FITS needs `vaex-astro`. Compatible binary wheels determine platform availability; compiling the optional `annoy` dependency requires a C++ toolchain, not just Python headers.

Native checks used Python 3.12, core 4.19.0, HDF5 0.15.0, viz 0.6.0, ML 0.19.0, NumPy 2.5.3, pandas 2.3.3, PyArrow 25.0.1 and Matplotlib 3.11.2 on macOS ARM. See [review and verification](references/review.md) for evidence and optional-integration limits. These are correctness checks on small synthetic inputs, not performance benchmarks.

Workflow

1. Establish row identity, units, schema, missing-value codes, and expected counts. Inspect CSV raw headers before parsers rename duplicates; supply explicit types for IDs and late-appearing values. Keep dates, time zones, and sampling cadence explicit. 2. Open files with `vaex.open`. HDF5 must use a compatible table layout; arbitrary HDF5 scientific arrays are not automatically a Vaex table. CSV opening performs indexing/schema work; Parquet must decode compressed data. Neither is an instant, zero-memory operation. 3. Select needed columns and use expressions for derived values. A virtual column avoids a full stored array but still needs expression metadata and evaluation buffers. 4. Record filters/selections and missingness before reductions. Batch independent statistics with `delay=True`, then `df.execute()` and each promise's `.get()`. 5. Validate counts, units, join cardinality, and numerical results against a small independently computed subset. Binned or approximate summaries need explicit limits/resolution. 6. Plot aggregated grids or a bounded sample. A count heatmap and a mean heatmap answer different questions; show coverage and avoid hiding rare/extreme observations silently. 7. Export directly in chunks; exporting evaluates virtual columns without needing `materialize()` first. Reopen and check counts/schema/values before replacing source data.

Small executable example

Run in a writable working directory; output names must not refer to existing data.

from pathlib import Path
import numpy as np
import vaex

out = Path('vaex-example.hdf5')
if out.exists():
    raise FileExistsError(out)
df = vaex.from_arrays(
    x=np.arange(1., 7.), y=np.arange(6.) ** 2,
    category=np.array(['A', 'B', 'A', 'B', 'A', 'B']),
)
df['energy'] = df.x ** 2 + df.y
selected = df[df.x >= 3]
mean_task = selected.energy.mean(delay=True)
count_task = selected.count(delay=True)
selected.execute()
assert count_task.get() == 4
assert np.isclose(mean_task.get(), 35.0)
summary = df.groupby('category', agg={
    'rows': vaex.agg.count(), 'energy_sum': vaex.agg.sum('energy'),
})
assert int(summary.rows.sum()) == len(df)
df.export_hdf5(str(out), chunk_size=2)
reopened = vaex.open(str(out))
assert reopened.get_column_names() == df.get_column_names()
assert np.allclose(reopened.energy.to_numpy(), df.energy.to_numpy())

For a large real input, replace the in-memory fixture with `vaex.open('input.hdf5')`. The small `.to_numpy()` comparison above is a fixture check; do not apply it to a whole larger-than-RAM dataset. Compare sampled rows and streamed summaries instead.

Reference map

• [Core DataFrames](references/core_dataframes.md): loaders, expression/array distinctions, inspection and schema.
• [Data processing](references/data_processing.md): filtering, missingness, strings/dates, grouped statistics and joins.
• [Performance](references/performance.md): delayed/async execution, caching, buffers, materialization and profiling.
• [Visualization](references/visualization.md): supported `df.viz` methods, grid geometry, finite plotting limits and widgets.
• [Machine learning](references/machine_learning.md): train-only fitting, native transformers, estimator memory and state transfer.
• [I/O](references/io_operations.md): chunked CSV conversion, HDF5/Arrow/Parquet round trips and remote boundaries.

Failure checks

• `df.x.mean()` returns a computed result; it is not a lazy expression.
• Use `df.percen