/tiledbvcf

name: tiledbvcf
description: Stores and retrieves genomic variant calls with TileDB-VCF. Use for indexed single-sample VCF/BCF ingestion, incremental cohorts, region and sample queries, streaming results, allele statistics, QC, and VCF/BCF export locally or through TileDB Cloud.
license: MIT license
compatibility: Requires a native TileDB-VCF installation and Python 3.9-3.12 for the reviewed Conda builds. bcftools compresses/indexes input fixtures. Installation and remote storage need network access; TileDB Cloud needs its Python client, account token, and storage permissions.
metadata:
  version: "1.3"
  skill-author: Jeremy Leipzig
  last-reviewed: "2026-10-01"
  upstream-version: "TileDB-VCF 0.40.3; tiledb-cloud 0.14.4"

TileDB-VCF

When to use

Use for cohort variant storage, incremental sample ingestion, interval queries, and exporting subsets for downstream genomics. TileDB-VCF stores records; it does not perform joint variant calling, association testing, normalization, or population structure adjustment. Dataset size alone does not determine whether cloud execution is appropriate: benchmark the intended sample, region, and attribute workload.

This skill targets released **TileDB-VCF 0.40.3**. Native Python/CLI checks used two synthetic single-sample VCFs on macOS ARM64. Cloud client **0.14.4** contracts were checked against its released wheel and mocked dispatch; hosted queries and jobs were not executed. See [installation and verification notes](references/verification.md).

Install and verify the native stack

The `tiledbvcf` Python distribution is not published on PyPI at this review. The official **`tiledb` Conda channel** supplies `tiledbvcf-py` 0.40.3 for macOS ARM64, macOS x86-64, and Linux x86-64, with Python 3.9-3.12 builds. `tiledb` on PyPI is TileDB-Py and does not install TileDB-VCF. Native Apple Silicon no longer requires forcing `CONDA_SUBDIR=osx-64`.

For the tested macOS ARM64 environment, pins below avoid versioned-library import failures in the otherwise successful current Conda solve:

conda create -n tiledb-vcf -c tiledb -c conda-forge \
  python=3.12 tiledbvcf-py=0.40.3 \
  azure-core-cpp=1.16.2 azure-storage-blobs-cpp=12.16.0 \
  azure-storage-files-datalake-cpp=12.14.0 capnproto=1.4.0 c-blosc2=2.23.1
conda activate tiledb-vcf
python -c 'import tiledbvcf; print(tiledbvcf.version)'
tiledbvcf version

The equivalent Micromamba solve/install was executed. These extra ABI pins are a verified macOS ARM64 workaround, not a claim about all platforms. Preserve the resolved environment for production. Other-platform installation and official Docker images are alternatives documented upstream, not tested here.

Prepare and create a cohort

1. Record reference assembly, contig naming/lengths, callers, normalization rules, sample identity, and file checksums. Do not mix assemblies or assume `1` and `chr1` are equivalent. Confirm compatible headers across inputs. 2. Each input must contain **one sample**, be coordinate sorted, and have an index. Use BGZF-compressed VCF or BCF with a matching `.csi`/`.tbi`. Inspect `bcftools query -l sample1.vcf.gz`; filenames are not sample identifiers. 3. Create the dataset explicitly, then ingest. Opening `mode="w"` alone does not create its schema. Materialize frequently read fields at creation time. 4. Validate a small known interval and a round-trip export before scaling.

For existing sorted plain-text VCFs (run once per input; output names must be new):

bcftools view -Oz -o sample1.vcf.gz sample1.vcf
bcftools index -c sample1.vcf.gz
bcftools view -Oz -o sample2.vcf.gz sample2.vcf
bcftools index -c sample2.vcf.gz

The following local examples were exercised with sample IDs `S1` and `S2`, contig `chr1`, and coordinates 1-100; substitute the verified IDs and intervals in real data.

import tiledbvcf

uri = "cohort"
config = {"sm.compute_concurrency_level": "2", "sm.io_concurrency_level": "2"}
with tiledbvcf.Dataset(uri, mode="w", tiledb_config=config) as ds:
    ds.create_dataset(extra_attrs=["fmt_GT", "fmt_DP"])
    ds.ingest_samples(
        ["sample1.vcf.gz"], threads=2, total_memory_budget_mb=512
    )

# Incremental ingestion uses the existing dataset; do not call create_dataset again.
with tiledbvcf.Dataset(uri, mode="w", tiledb_config=config) as ds:
    ds.ingest_samples(
        ["sample2.vcf.gz"], threads=2, total_memory_budget_mb=512
    )

The schema-v4 data array has `contig`, start-coordinate, and sample **dimensions**; headers and optional statistics are separate arrays in the dataset group. Fields not explicitly materialized remain in the INFO/FORMAT payloads. Query names such as `pos_start` need not match raw storage column names.

Upstream supports parallel thread/process ingestion. Assign distinct sample work and coordinate lifecycle/maintenance operations; do not blindly re-ingest samples. `resume=True` supports interrupted ingestion, not arbitrary duplicate correction.

Query completely and interpret coordinates correctly

| Surface | Coordinate convention | | --- | --- | | Python `regions=["chr1:10-14"]` | 1-based, closed interval | | Returned `pos_start`, `pos_end` | 1-based, inclusive record endpoints | | BED input and `query_bed_start`, `query_bed_end` | 0-based, half-open | | `read_variant_stats()` / `read_allele_count()` column `pos` in 0.40.3 | **0-based**; add one before joining to VCF POS |

Queries return **overlapping records**, not just records starting inside the interval. A deletion spanning positions 10-14 appears in `chr1:14-14`, with `pos_start=10`. For BED `[13,14)`, the equivalent string is `chr1:14-14`. The Python region parser requires explicit `contig:start-end`; bare `"chr1"` is rejected in this release. Use a known contig length for a whole-contig query.

cfg = tiledbvcf.ReadConfig(memory_budget_mb=128, tiledb_config=config)
attrs = ["sample_name", "contig", "pos_start", "pos_end", "alleles", "fmt_GT"]
wi