/lance-format

name: lance-format
description: Deep reference for Lance v10 - the open columnar lakehouse format for multimodal AI - and its Rust crate workspace plus pylance. Covers the 2.x file format and structural encodings, the table format (manifests, fragments, transactions, OCC), vector / scalar / full-text indexes, MemWAL, schema evolution, time travel, namespaces, and object-store config. Use when building directly on the Lance crates or reading `.lance` datasets; this is the Lance format and engine (`lance-format/lance`), not the LanceDB product built on top of it.
metadata:
  version: "0.12.0"
  upstream: "lance-format/lance@v10.0.0-beta.7"
  openclaw:
    homepage: https://github.com/tenequm/skills/tree/main/skills/lance-format
    emoji: "🗄️"

Lance v10 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x faster than Parquet for random access." It is not one format but a stack of interoperating specs: a **file format**, a **table format**, **index formats**, **catalog specs**, and a **namespace client spec**. The Rust workspace at `lance-format/lance` implements all of them plus Python (`pylance`) and Java bindings.

This skill tracks **`v10.0.0-beta.7`** (the `lance-format/lance` git tag), the current development frontier. Pin against tags, not `main` - Lance ships beta tags every few days and `next`-format encodings can change.

**Which tag to pin.** Lance's release train bumps the major automatically: `ci/publish_beta.sh` compares the release root to HEAD and, if any PR in the range carries the GitHub `breaking-change` label, re-roots at `MAJOR+1`. That is what happened here - the `9.1.0-beta.*` dev line was mechanically renamed to `10.0.0-beta.*` on 2026-07-23 (`chore: bump to 10.0.0-beta.1 based on breaking change detection`), so **`v9.1.0` was never tagged** and `v9.1.0-beta.8` is the direct ancestor of `v10.0.0-beta.7`. A major bump therefore signals "some labeled breaking change landed", not a redesign.

**`v9.0.0` final shipped 2026-07-24 and is now the stable pin** (it supersedes `v8.0.0`, 2026-07-01). It lives on the `release/v9.0` stabilization branch - already at `9.0.1-beta.0` - and is *not* an ancestor of `main`. crates.io carries **finals only** (currently `lance 9.0.0`); every beta and rc is a git tag with no published crate, so a beta pin means a git dependency.

Three layers of reference, load what the task needs:

• `references/lance-reference.md` - the distilled deep reference. Any concrete schema,

parameter, proto, or constraint.

• `references/performance.md` - ALL official performance guidance combined, plus

field-verified remote-storage practices. Load for any performance, tuning, maintenance-cost, or "why is this slow" question.

• `references/docs/` - a **verbatim mirror of the official docs** (`docs/src` at the tracked

tag): every guide, quickstart, and format spec, unedited. Load when you need the full official text. Complete file map below.

This file is the orientation: read it first, then jump into what you need.

Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

• **Lance** - the format and engine. The `lance-format/lance` repo; the `lance` /`lance-*`

Rust crates; `pylance`. It gives you datasets, the file/table format, indexes, commits, scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own Rust/Python code. **This skill is about Lance.**

• **LanceDB** - a separate database *product* (`lancedb/lancedb`) built on top of Lance. It

adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK parity, and managed Cloud / Enterprise tiers. Not covered here.

If you are linking the `lance` crate in `Cargo.toml`, you are using Lance directly - use this skill. If a question is about LanceDB internals, the storage layer underneath it is still Lance, so this skill remains the authority for the format itself.

The crate workspace

26 crate directories under `rust/`. `lance` is the public entry point; the rest are layers beneath it. Full table with descriptions and citations in `references/lance-reference.md` section 2.

| Crate | Role | |-------|------| | `lance` | Public entry point - `Dataset`, scanner, indexes, commits | | `lance-table` | Table format - manifest, feature flags, commit handlers, row IDs | | `lance-file` | File format - file reader/writer | | `lance-encoding` | Structural encodings, compression (internal, not for external use) | | `lance-index` | Scalar / vector / FTS / system indexes | | `lance-index-core` | Shared index primitives extracted from `lance-index` (new in the 9.1/10.0 dev line, PR #7713) | | `lance-io` | Object store, I/O schedulers | | `lance-core` | Shared `Error`/`Result`, cache, datatypes | | `lance-datafusion` | DataFusion glue (exec, expr, planner, UDFs) | | `lance-linalg` | SIMD L2 / dot / cosine / hamming kernels | | `lance-select` | Row-selection primitives - `RowAddrMask`, `RowIdMask`, `IndexExprResult` (extracted from `lance-core`/`lance-index` in v7.1.0-beta.2) | | `lance-tokenizer` | FTS tokenizer stack (simple, ngram, jieba, lindera, stemmers) | | `lance-derive` | `#[derive(DeepSizeOf)]` proc-macro for Arrow-aware memory accounting (new in v8, PR #6229; replaced the external `deepsize` crate) | | `lance-geo` | Geospatial UDFs (feature-gated `geo`) | | `lance-namespace` / `-impls` / `-datafusion` | Namespace trait, Directory/REST impls, DataFusion catalog bridge | | `lance-arrow`, `lance-tools`, `fsst`, `lance-bitpacking`, ... | Arrow extensions, CLI, compression sub-crates |

All share `version = "10.0.0-beta.7"` except the two Arrow-tracking crates, `lance-arrow-scalar` and `lance-arrow-stats`, both pinned at `=58.0.0`. Workspace: edition 2024, `rust-version = 1.91.0` (MSRV; the pinned build toolchain in `rust-toolchain.toml` is 1.97.0, PR #7712), `resolver = "3"`; notable deps arrow 58, datafusion 54, opendal 0.57, `object_store` 0.13.2,