Git for large files at any scale

Git already gives teams the best shared vocabulary for software: add, commit, branch, push, clone.

The problem is no longer Git’s interface. It is the weight behind it.

A modern commit may need to name source code, a 40 GB checkpoint, a changing dataset, and thousands of binary assets. Moving the heavy files into a second catalog lightens Git, but splits the state the team must reproduce.

Today we are launching Crab around one premise:

Keep one Git history. Put the heavy bytes directly in your object store. Move only the content a person or job actually needs.

Crab stores compact pointers in Git. Content-defined chunks, immutable packed objects, and reconstruction metadata live in S3, Google Cloud Storage, Azure Blob Storage, or an S3-compatible store.

No Crab data server sits between your team and the bucket.

The file became infrastructure

“Large file” used to mean an occasional installer or video. Now binary content is often the product:

• ML: checkpoints, embeddings, tokenized corpora, evaluation sets
• Games: textures, audio, meshes, maps, engine caches
• Data: Parquet, Arrow, databases, scientific arrays
• Media: frames, simulation output, layered source assets
• Agents: more branches, worktrees, experiments, and short-lived states

Git still provides the right coordination unit. A commit is reviewable. A branch is collaborative. A checkout selects an exact historical state.

The physical representation is where the economics break down.

A small edit can create a large object

Imagine four 8 GB checkpoints. Each new version changes only 0.5 GB of encoded bytes.

Ordinary Git still sees four complete blobs. A content-addressed data plane can keep the shared regions once and add only the new identities.

That example is illustrative, not a savings guarantee. Recompressed or encrypted data may change broadly even when the logical edit is small.

One history, two data paths

Crab extends Git at two existing boundaries:

1. A long-running filter=crab process turns selected files into backed pointers during add.
2. git-remote-crab implements the remote-helper protocol for a crab:// remote.

The commit remains the join:

commit 8fc2
├── src/train.rs ───────────── ordinary Git blob
└── models/encoder.bin ─────── Crab pointer: f41a… · 40 GB
                                      │
                                      └── recipe → shards → xorb ranges
                                                             │
                                                             └── your bucket

Git can inspect names and history without downloading model bytes. The object store can hold immutable content without understanding branches or worktrees.

The user sees one repository. The implementation gets two lanes with different scaling behavior.

Reuse starts with identity

Whole-file storage can reuse content only when the complete file is identical. Fixed-size chunks improve the unit, but an insertion near the front shifts every later boundary.

Content-defined chunking can resynchronize after an edit. Distant, unchanged regions retain their earlier identities.

Crab hashes chunks with BLAKE3 and packs new chunk data into immutable xorbs. Shards locate each chunk range. A file recipe preserves reconstruction order.

That creates reuse across several dimensions:

• Versions: a checkpoint can retain regions from its parent.
• Branches: experiments can share their common base.
• Paths: a copy or rename does not imply another physical copy.
• Contributors: identity does not depend on the uploader.
• Time: a new version can reuse content introduced months earlier.

The invariant is intentionally modest: identical chunks have identical identities, so Crab does not need to store proven content twice in one repository data plane.

Add prepares. Push publishes.

A pointer is useful only when its client can later publish a complete reconstruction recipe. That makes add a transaction, not a text substitution.

Add creates a backed pointer

crab add streams the file, chooses stable chunk boundaries, hashes content, flushes local staging, writes the pointer, and then updates Git’s index.

Three decisions remain independent:

1. Git and .gitignore decide whether a path enters history.
2. .gitattributes decides whether Git stores bytes or a Crab pointer.
3. Hydration policy decides whether this worktree materializes the bytes.

Push makes immutable data durable first

A branch must never point to a missing chunk. Crab therefore prepares every immutable dependency before it changes the destination ref.

The expected-old comparison is the authority: move the ref only if it still names the commit this push planned against.

Before publication, an interruption can leave safe immutable orphans. After publication, the visible state is complete.

Clone history. Hydrate bytes.

Clone should not mean “download every large file anybody ever committed.”

A source-only CI job may need no checkpoints. An evaluation job may need one model and one test set. An artist may need the current map but not every cinematic.

Crab clone is lazy by default:

# Fetch Git history and compact pointers.
crab clone crab://team-bucket/vision-search

# Materialize only this job's working set.
crab hydrate 'models/encoder.safetensors'

# Return clean managed files to pointer form.
crab dehydrate 'models/*.safetensors'

Hydration is a proof-oriented read path:

1. Read the pointer identity and logical size.
2. Resolve every ordered recipe term.
3. Coalesce the required xorb ranges.
4. Read from verified cache or canonical origin.
5. Reconstruct chunks in recipe order.
6. Verify the complete output.
7. Materialize only after verification succeeds.

The result is byte-identical, or hydration returns an error. A partial output does not quietly become the worktree file.

Scale has five axes

Repository size is only one kind of pressure.

One huge file

Hashing, chunking, upload, and reconstruction stream through bounded buffers. The whole file does not need to fit in memory.

Deep history

The full-file hash changes with every version, preserving exact identity. Stable chunk regions can still be reused physically.

Many files

Xorbs avoid creating one tiny object-store object per chunk. Shards preserve lookup and ordering without using path names as the physical layout.

Many writers

Different refs can prepare in parallel. Writers to the same ref reconcile at the expected-old transaction; uploading bytes never grants permission to replace history.

Many readers

Verified caches can accelerate reads, but the bucket remains authoritative. A cache miss changes latency, not correctness.

Failure is boring on purpose

Machines stop mid-plan. Credentials expire. Another writer moves the branch. A response disappears after a successful commit.

Crab assigns each failure to a boundary with a visible result and a safe next action.

The visibility rule is compact enough to keep in your head:

before REF TXN  → old tip stays visible → retry is safe
at REF TXN      → expected-old decides  → one winner
after REF TXN   → new tip is complete   → repair moves forward

Garbage collection follows reachability

Deleting a path does not prove its chunks are unused. The same content may still belong to another version, branch, file, workflow artifact, or recovery point.

Crab GC starts from retained roots, walks the complete Git and file-data closure, compares the mark set with repository inventory, and then applies a grace window.

An object is eligible only when both statements are true:

1. No retained root reaches it.
2. It is older than the protection window.

The grace window protects concurrent or interrupted writers. Repository scope protects other teams that share the bucket.

Choose Crab for the right problem

Crab is not the answer to every large object.

Use ordinary Git when it already works. Crab leaves unmatched paths alone.

Use an artifact registry when the content is a released product, not a file edited beside source. Use a domain data system when its query, lineage, or governance model is the primary interface.

Use Crab when four conditions line up:

• Git commits and branches should remain the coordination interface.
• Large versions share meaningful byte regions.
• Worktrees should materialize selected content.
• Durable bytes should live directly in cloud storage you control.

First push in five minutes

Crab ships as one CLI. The installation also provides Git’s remote helper and filter process.

brew install crabbuild/tap/crab

# Or use the checksum-verifying installer.
curl -fsSL https://crab.build/install.sh | bash

Create a repository backed by your bucket:

mkdir vision-search && cd vision-search
crab configure s3://team-bucket/vision-search
crab track '*.safetensors'
crab add .
git commit -m "Add the first model"
crab push origin main

Then verify the ref and its data closure separately:

git ls-remote origin refs/heads/main
crab fsck --json
crab doctor

The premise, tested

“Any scale” does not mean physics stops applying.

A five-terabyte version still contains five terabytes of logical content. A cold object still has restore latency. Re-encoding can erase reuse. Object stores still charge for storage, requests, and transfer.

The goal is more precise: do not force every scale dimension onto every operation.

A large history should not require a large checkout. A large file should not require whole-file memory. A failed upload should not expose a broken ref.

Crab is our attempt to keep those statements true while preserving the workflow developers already understand.

Start with the installation guide, create a first repository, or inspect the complete CLI documentation.

Migrating from Git LFS? Read the migration guide. Working with more data than local disk? Continue with lazy clone and hydration and the local mount workflow.

The first question is not “How large is the repository?”

It is: Which bytes does this operation actually need?

Crab lets Git answer which state you mean—and lets the data plane answer the rest.