Manage external data sources with crab data

Use crab data to bring an external input into the current worktree and keep enough provenance to refresh that input later. Each import creates two local artifacts:

• a materialized file or directory at the output path you choose;
• a credential-free source descriptor under .crab/workflow/sources/.

The descriptor binds the target to its source, resolved revision or provider validator, verified BLAKE3 content identity, and byte count. This makes external inputs reviewable and repeatable alongside Crab workflows.

Command overview

crab data <COMMAND> [OPTIONS]

Every subcommand supports --json and --jsonl. The flags are mutually exclusive.

Start with a versioned input

crab data import ../datasets \
  --path training/samples.parquet \
  --rev main \
  --output data/samples.parquet

crab data status data/samples.parquet

crab data update data/samples.parquet --dry-run
crab data update data/samples.parquet

git status --short
git add data/samples.parquet .crab/workflow/sources/
git commit -m "data: pin training samples"

All commands resolve the current Git worktree root first. Running them from a nested directory still interprets list paths and targets relative to that worktree root. Local source, database, and query-file paths resolve from the shell's current directory when you import them.

List data paths

crab data list inventories either the materialized worktree or a committed Git tree.

crab data list [PATH] [--recursive] [--rev REVISION] [--json | --jsonl]

List the direct children of the worktree:

crab data list

directory	data
file	params.yaml

List every materialized file below data/:

crab data list data --recursive

Inspect the same path as it existed in a commit:

crab data list data --rev HEAD~1 --recursive --json

Without --rev, Crab reads the filesystem and omits Crab's internal .git, .crab, and target directories during normal directory traversal. With --rev, Crab resolves the selector locally and reads the committed Git tree; it does not fetch a missing revision or infer remote Git transport.

If a listed file is the target of a source descriptor, structured output adds its source_id. Directory entries have no byte size or source ID.

Import a local file, directory, or Git revision

crab data import SOURCE --output TARGET \
  [--path SOURCE_PATH] [--rev REVISION] [--json | --jsonl]

Import the live filesystem

crab data import ../datasets \
  --path exports/latest \
  --output data/latest

Without --rev, Crab snapshots the current file or directory, including uncommitted content. If SOURCE is a Git repository, the descriptor records its current HEAD when available, but future updates continue to follow the live source path.

Import committed content

crab data import ../datasets \
  --path exports/train.csv \
  --rev v2.1.0 \
  --output data/train.csv

With --rev, Crab records both the requested selector (v2.1.0) and the resolved commit. A later crab data update resolves the same selector again, so a moving branch or tag can advance to a new commit. A commit hash remains pinned to that commit.

Committed imports support a regular file, a directory, or the repository root. They preserve executable bits. Git symlinks, submodules, and other non-regular tree entries are rejected before Crab installs the target.

Import a URL or object-store object

crab data import-url URL --output TARGET [--json | --jsonl]

Examples:

crab data import-url https://datasets.example.com/train.csv \
  --output data/train.csv

crab data import-url s3://ml-inputs/releases/train.parquet \
  --output data/train.parquet

crab data import-url file:///srv/exports/labels.json \
  --output data/labels.json

HTTP responses must have a successful status. Weak ETags are not used as validators. Object-store downloads verify the streamed byte count against the provider metadata before installation.

SSH/SFTP, WebDAV, HDFS, Google Drive, OSS, and other URL schemes are not implemented. They fail before creating the target or descriptor.

Import a SQLite query

crab data import-db runs a query against a local SQLite database opened in read-only mode and writes one JSON object per row.

crab data import-db sqlite \
  --database warehouse/source.db \
  --query 'select id, label from samples order by id' \
  --output data/samples.jsonl

The query must be non-empty UTF-8 text no longer than 1,024 bytes. Crab stores the trimmed query in the descriptor so crab data update can rerun it. Do not put passwords, tokens, or other secrets in query text.

Rows preserve query order; include an explicit ORDER BY when deterministic ordering matters. Column names become JSON keys. SQLite values map as follows:

For example:

{"id":1,"label":"one","payload":null}
{"id":2,"label":"two","payload":{"$binary_base64":"AQI="}}

Other database connectors fail before writing. Crab snapshots query results; it does not replace database transactions, migrations, or query planning.

Preview or apply an update

crab data update TARGET_OR_ID [--dry-run] [--json | --jsonl]

TARGET_OR_ID can be the descriptor's 64-character ID or its target path:

crab data update data/train.csv --dry-run --json
crab data update data/train.csv

An update follows the source policy recorded at import time:

• live local sources read the same file or directory again;
• revision imports resolve the original requested Git selector again;
• HTTP sources use their strong ETag for a conditional request when available;
• object-store sources compare their version or strong ETag before downloading;
• SQLite sources rerun the stored query against the same database.

Crab considers content hash, byte count, revision, and validator when deciding whether a source changed. Text output is either changed or up-to-date.

--dry-run prevents replacement, but it is not an offline operation: Crab still reads or downloads enough source data to compute and verify the candidate state. Temporary data is discarded afterward.

For an applied change, Crab writes to a sibling temporary path, verifies the candidate, swaps it with the target, and atomically rewrites the descriptor. If the descriptor write fails, Crab restores the prior target. A failed fetch or verification therefore leaves the previous materialization intact.

Check source status

crab data status [TARGET_OR_ID] [--json | --jsonl]

Without a filter, the command reports every descriptor. A filter accepts the same descriptor ID or target path as update.

source:up-to-date	data/train.csv	cache=not-managed,descriptor=present,git=clean,lock=locked,remote=not-checked,source=not-checked,workspace=up-to-date

Status intentionally performs no network I/O. It compares the descriptor with the local target and asks Git whether the target is clean, then reports these dimensions:

Use crab data update TARGET --dry-run when you need a source-aware freshness check. Unlike status, that command can perform filesystem, database, or network I/O.

Understand source descriptors

Descriptors live at:

.crab/workflow/sources/<source_id>.json

The stable source ID is derived from the source kind, credential-free locator, and target path. Updating content does not change the ID.

{
  "schema_version": 1,
  "id": "abababababababababababababababababababababababababababababababab",
  "kind": "repo",
  "locator": "/srv/datasets",
  "revision": "8b58e5155a8f5d21c4c9e6a76b28f176305f4f31",
  "validator": null,
  "content_hash": "b3:cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd",
  "size": 4829012,
  "target": "data/train.csv",
  "metadata": {
    "requested_revision": "main",
    "source_path": "exports/train.csv"
  }
}

Descriptor reads reject invalid identities, unsafe targets, newer schemas, credential-bearing URLs, and metadata keys or values that appear to contain secrets. Descriptor writes use an atomic file replacement.

Use structured output

All six commands support the same output modes:

Example list response:

{
  "schema": "data",
  "version": "1.0",
  "timestamp": "2026-08-21T06:00:00.000Z",
  "data": {
    "operation": "list",
    "entries": [
      {
        "path": "data/train.csv",
        "kind": "file",
        "size": 4829012,
        "source_id": "abababababababababababababababababababababababababababababababab"
      }
    ]
  }
}

Import and update responses put a descriptor plus changed and dry_run under data.entries. Status responses use the same entry fields as list and add a dimensions object.

See Structured Output for envelope, error, and exit-code conventions.

Troubleshoot common failures

Related workflow documentation

• Running Commands: consume imported data as stage dependencies.
• Workflow Pipelines: connect data inputs to multi-stage workflows.
• Workflow Lockfile: pin reproducible workflow state.
• Migrating from DVC: move existing DVC projects to Crab.