/datalad

name: datalad
description: Retrieve, version, and publish scientific datasets with DataLad and git-annex, and capture computational provenance with datalad run, rerun, and containers-run. Use when cloning or fetching data from OpenNeuro, DANDI, datasets.datalad.org, or any DataLad dataset; when a file in a dataset reads as a broken symlink or a small pointer instead of real data; when an analysis needs a machine-readable record of how each output was produced so it can be re-executed; or when publishing a dataset to siblings such as a GitHub repository plus a storage remote. Also use to decide between DataLad and plain Git for a data-carrying repository.
compatibility: Needs datalad 1.6.x on Python 3.10+, plus git and git-annex 10.x. git-annex is not written in Python but installs as a prebuilt wheel from PyPI (`uv pip install git-annex`), from a system package manager, or from conda-forge. Container-based provenance also needs datalad-container (1.2.x) and Singularity/Apptainer or Docker. clone, get, and push need network access; credentialed remotes read secrets from the system keyring or from DATALAD_CREDENTIAL_<NAME>_<COMPONENT> environment variables.
license: MIT
allowed-tools: Read Write Edit Bash
metadata:
  version: "1.0"
  skill-author: Dylan Pulver

DataLad

Overview

DataLad is a data management layer over Git and git-annex. Git tracks the dataset structure, small text files, and the history. git-annex tracks the *content* of large files, storing each file as a key and keeping the bytes somewhere that is not necessarily the local repository.

That split is the single most important thing to internalise, because it means a freshly cloned dataset contains the full history and the full file listing while containing almost none of the data. A 100 TB dataset clones in seconds and occupies a few megabytes. The bytes arrive only when asked for, per file, with `datalad get`.

The second thing DataLad adds is provenance. `datalad run` executes a command and commits the result together with a machine-readable record of the command, its inputs, and its outputs. `datalad rerun` reads that record back and re-executes it. This turns "how was this figure produced" from an archaeology problem into a command.

When to use DataLad instead of plain Git

Use DataLad when any of the following holds:

• Files are too large for Git to handle comfortably, or the total exceeds what every

collaborator wants on disk.

• Data lives in more than one place (a lab server, a cluster scratch, S3, a supercomputer)

and you need to know which copies exist.

• The analysis must be re-executable, and a plain commit message is not enough evidence.
• You are consuming published datasets from OpenNeuro, DANDI, or `datasets.datalad.org`,

which are distributed as DataLad datasets.

• The project nests other datasets inside it and you want each one to keep its own

independent history.

Use plain Git when the repository is code and text only, everything fits comfortably in Git, and nobody needs partial checkouts. DataLad on top of a small pure-code repository adds indirection without buying anything.

Installation

# git-annex is NOT written in Python but is available from PyPI if you already
# have git itself installed:
uv pip install git-annex
# You can also install it first from the system
# (Debian/Ubuntu: apt install git-annex; macOS: brew install git-annex;
#  conda-forge: conda install -c conda-forge git-annex)
uv pip install datalad
uv pip install datalad-container   # only for containers-run

datalad wtf --section dependencies   # confirm git-annex version is visible

The PyPI `git-annex` package ships the prebuilt binary as a wheel for Linux, macOS, and Windows rather than building the Haskell sources, so it installs like any other Python dependency and can be pinned in the same environment as DataLad. It does not bring git along with it.

`datalad wtf` prints the resolved environment and is the first thing to run when behaviour looks impossible. An old or missing git-annex is behind a large share of confusing errors.

DataLad itself is MIT licensed. git-annex is a separate tool under the AGPL, which matters only if you redistribute a modified git-annex rather than call it.

The failure that bites first: pointers are not data

After `datalad clone`, annexed files exist as symlinks into `.git/annex/objects/` (or as small pointer files where symlinks are unavailable, such as on Windows or a crippled filesystem). Nothing has downloaded the content yet.

datalad clone https://github.com/OpenNeuroDatasets/ds000001.git
cd ds000001
ls sub-01/anat/            # the file is listed
python -c "import nibabel; nibabel.load('sub-01/anat/sub-01_T1w.nii.gz')"   # fails
datalad get sub-01/anat/sub-01_T1w.nii.gz                                   # now it works

The failure mode to recognise: a tool reports the file as empty, truncated, corrupt, "not a gzip file", or a broken symlink, and the file size on disk is a few hundred bytes. That is a pointer, not a corrupted download. **Run `datalad get` before reading data, and treat "file exists" as insufficient evidence that its content is present.**

Before an analysis touches a directory, fetch it explicitly:

datalad get sub-01/                  # everything under a path
datalad get -r .                     # everything, including subdatasets
datalad get -n -r .                  # subdataset structure only, no file content

`datalad status --annex` reports how much content is present locally, and `git annex whereis <path>` reports which repositories hold a given file. `whereis` reads recorded state and does not contact the remotes, so it tells you what git-annex last learned rather than what is true right now.

See [data-access.md](references/data-access.md) for finding datasets, subdataset behaviour, dropping content safely, and repairing a dataset.

Recording provenance with datalad run

`datalad run` is the reas