How do you work with a repository larger than your disk?

Imagine a 4 TB vision repository and a laptop with 256 GB of storage. The repository fits in Crab. It does not fit on the laptop.

That is fine. Choose when bytes become local:

• Keep pointers for Git work.
• Hydrate a known working set.
• Mount the repository for on-demand reads.

Pick from the access pattern

Repository size is not the deciding factor. Predictability is.

All five choices can use the same commit. The file identity does not change; only the local materialization policy does.

1. Keep pointers for Git work

A normal Crab clone is lazy by default:

crab clone crab://team-data/vision-search
cd vision-search
crab status

Git history, source, and pointer blobs are local. The 820 GB of models and 3.1 TB of datasets stay in object storage.

A pointer still identifies the complete file. It records the content hash and size. Git can compare it, switch branches, and merge it without downloading the payload.

This is different from sparse checkout:

• Sparse checkout: the path may be absent.
• Crab pointer: the path is present, but the large payload is deferred.

A parser expecting a real model or image cannot use the pointer as data. That tool needs hydration or a mount.

2. Hydrate a known set

Suppose training needs one model, tokenizers, and the current dataset slice. Hydrate just those paths:

crab hydrate 'models/current/**'
crab hydrate --manifest .crab/manifests/training.txt

For repeatable jobs, commit the selection:

# .crab/manifests/training.txt
configs/**
models/current/*.safetensors
data/tokenizers/**

Then hydrate the manifest from the commit:

crab hydrate --manifest-ref HEAD:.crab/manifests/training.txt

The manifest makes the disk requirement reviewable. A branch changing its training inputs changes the manifest too.

You can also materialize a selection during clone:

crab clone crab://team-data/vision-search \
  --include 'configs/**' \
  --include 'models/current/**'

Avoid a casual **. A broad glob can turn an 84 GB job into a 4 TB download.

Recover space with dehydrate

dehydrate replaces clean materialized files with their pointers:

crab dehydrate 'models/archive/**'
crab status

Modified files are skipped. Crab will not discard local edits to reclaim disk.

The read cache is separate. Dehydrating a file does not promise that every cached range or application output disappears.

3. Mount for unpredictable reads

An asset browser does not know which file the user will click. Mounting avoids guessing the working set:

crab mount \
  --repo crab://team-data/vision-search \
  --mountpoint /mnt/vision-search \
  --read-only

The application sees ordinary paths. Crab serves each filesystem operation from a stable repository snapshot.

Automatic backend selection prefers NFS when available. Use --backend=fuse to request FUSE; that machine must have fuse3 or macFUSE. Backend choice does not change content verification.

What a mounted read changes

Listing models/ reads snapshot and overlay metadata. It does not fetch every model.

The first 64 KiB read of a managed file resolves its pointer, reconstructs the needed read window from xorb ranges, verifies it, and saves that window in the local cache. A repeat read can stop at the cache.

Sequential readers benefit from adaptive prefetch. Random seeks and memory mapping can touch many windows. Benchmark the real application, not only a single cp.

An unavailable or corrupt range remains an error. Crab does not substitute zero-filled or truncated bytes.

Writes are a different capacity event

A writable mount adds a copy-on-write overlay. The base snapshot remains unchanged until you explicitly commit.

The first write to a pointer-backed file promotes the full file into local overlay backing. Editing one byte of a 12 GB model therefore needs roughly 12 GB of overlay space, plus temporary headroom during promotion.

Inspect and publish changes explicitly:

crab mount diff --mountpoint /mnt/vision-search
crab mount commit --mountpoint /mnt/vision-search \
  -m "Update generated artifacts" \
  --push

Commit freezes writes and checks that the base ref has not moved. If the push fails, Crab keeps the local commit identity and overlay recovery state.

Use crab mount export when you want the overlay in a normal directory before committing it.

Budget four local stores

Do not compare only “repository size” with “free disk.” Track:

1. Hydrated worktree files.
2. Verified read cache.
3. Writable overlay backing.
4. Application scratch and temporary output.

Keep headroom below the filesystem limit. Hydration needs temporary output, overlay promotion may reconstruct a full file, and concurrent reads can grow the cache before eviction catches up.

Cold and warm runs also differ. A cold read includes origin transfer and cache population. A warm read may avoid the network, but the application still reads and processes the requested bytes.

A practical daily workflow

For the example laptop:

1. Clone pointer-only for normal development.
2. Commit a narrow manifest for the 84 GB training set.
3. Mount the archive read-only for occasional exploration.
4. Dehydrate old, clean worktree files before the disk becomes tight.
5. Use a writable mount only with an overlay budget and commit plan.

Before switching branches, refresh the mount or create one at the intended ref. A long-running mount is a stable snapshot; it does not silently follow every branch update.

Unmount cleanly before deleting its mountpoint or cache state. Open processes may still depend on the snapshot and in-flight reads.

For exact lifecycle and backend options, see the crab mount reference.