/lance-format

name: lance-format
description: Deep reference for Lance v15 columnar format, its Rust crates, and pylance - file encodings, table format, indexes, schema evolution, time travel. Use when building on the Lance crates or reading .lance datasets, not the LanceDB product.
metadata:
  version: "0.22.0"
  categories: "development, integrations"
  topics: "lance, columnar-format, vector-search, rust, lakehouse"
  upstream: "lance-format/lance@v15.0.0-beta.10"
  openclaw:
    homepage: https://github.com/tenequm/skills/tree/main/skills/lance-format
    emoji: "🗄️"

Lance v15 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x faster than Parquet for random access." It is not one format but a stack of interoperating specs: a **file format**, a **table format**, **index formats**, **catalog specs**, and a **namespace client spec**. The Rust workspace at `lance-format/lance` implements all of them plus Python (`pylance`) and Java bindings.

This skill tracks **`v15.0.0-beta.10`** (the `lance-format/lance` git tag), the current development frontier; **`v14.0.2`** is the stable pin, released 2026-10-29. Pin against tags, not `main` - Lance ships beta tags every few days and `next`-format encodings can change. Version landscape below.

Three layers of reference, load what the task needs:

• **The deep reference** - any concrete schema, parameter, proto, or constraint. Split by topic:

| File in `references/` | Covers | Sections | |------|--------|----------| | `format-file.md` | What Lance is, the 28 crates, file format, data types | 1-4 | | `format-table.md` | Dataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL | 5-10 | | `indexes.md` | Vector / scalar / FTS / geo indexes, distributed builds | 11-12 | | `ops.md` | Object store, capability matrix, source map | 13, 15, 16 | | `changelog-v7-v15.md` | The full v7 -> v15 delta | 14 |

Cross-references written as "section N" resolve through `references/lance-reference.md`.

• `references/performance.md` - ALL performance guidance. Part A routes to the official text and

adds the source-derived changes upstream has not documented; Part B is field-verified remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this slow" question.

• `references/docs/` - a **verbatim mirror of the official docs** (`docs/src` at the tracked

tag): every guide, quickstart, and format spec, unedited. Load when you need the full official text. Directory map below.

`references/maintenance.md` covers refreshing this skill against a new upstream tag. Upstream also ships its own operational agent skill, `skills/lance-table-maintenance/` in the repo (#9610) - health indicators and maintenance operations; this skill stays the format reference.

Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

• **Lance** - the format and engine. The `lance-format/lance` repo; the `lance` /`lance-*`

Rust crates; `pylance`. It gives you datasets, the file/table format, indexes, commits, scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own Rust/Python code. **This skill is about Lance.**

• **LanceDB** - a separate database *product* (`lancedb/lancedb`) built on top of Lance. It

adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK parity, and managed Cloud / Enterprise tiers. Not covered here - and since v15 the upstream `quickstart/index.md` is a LanceDB quickstart, so it is left out of the mirror.

**The wider ecosystem** (separate repos, own version lines, none covered here): Flink streaming writes (`lance-flink`), PostgreSQL reads via `pglance`, read-only Apache Doris queries, a Cypher graph engine (`lance-graph`), a dataset browser (`lance-data-viewer`), agentic context management (`lance-context`), and namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, AWS Glue, Iceberg, BigLake, Dataproc and OneLake. The canonical docs site is **`lance.org`**. Generated per-language SDK docs live at `lance-format.github.io/lance-python-doc` for Python and [javadoc.io](https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html) for Java - the matching `lance-format.github.io/lance-java-doc` path 404s.

Linking the `lance` crate in `Cargo.toml` means you are using Lance directly - use this skill. For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the authority for the format itself.

**The wrapper can hide format features.** LanceDB's `create_table` cannot enable stable row IDs; only pylance's `write_dataset(enable_stable_row_ids=True)` can. If a format-level capability matters to your design, check whether the wrapper exposes it before assuming the underlying format settles the question - and reach for `pylance` directly when it does not.

The crate workspace

28 crate directories under `rust/` at `v15.0.0-beta.10` - #10251 split DataFusion-free execution helpers into **`lance-datafusion-exec`** (27 through every v14 tag). **`lance` is the public entry point** - `Dataset`, scanner, indexes, commits; everything else (`lance-table`, `lance-file`, `lance-encoding`, `lance-index`, `lance-io`, `lance-core`, `lance-datafusion`, `lance-linalg`, `lance-namespace*`, `lance-jsonb`, ...) is a layer beneath it. Edition 2024; Python bindings need 3.10+.

| Line | arrow | datafusion | pyo3 | MSRV | |------|-------|------------|------|------| | v15 betas (#10397, from beta.5) | 59.2 | 55 | 0.29 | 1.95.0 | | v14.0.x | 58 | 54 | 0.28 | 1.94.0 |

Full table with roles, versions, and every workspace dep in `references/format-file.md` section 2.

**Build prerequisites differ by line.** Through v14, six crates compile protobufs in `build.rs`, so every consumer build needs a real `protoc` (or the `protoc` cargo feature, which vendors `protobuf-src`); a toolchain-manager