Why does Git struggle with large files?

Git can store an 8 GB model. The trouble starts when the repository keeps ten, fifty, or one hundred versions of it.

A normal full clone transfers every reachable version. Mirrors, backups, and scans also have to account for that history, even when a developer only wants to edit Python.

Follow one model through four commits

Consider this repository:

vision-search/
├── src/train.py
├── configs/base.toml
└── models/encoder.safetensors  # 8 GB

The model changes after each training run. The source and configuration should stay in the same commit as the model that uses them.

In a configured Crab repository, the workflow is:

crab track '*.safetensors'
crab ship . -m "Train encoder v1"

# After the next training run
crab ship models/encoder.safetensors -m "Train encoder v2"

crab track writes the .gitattributes rule. crab ship stages the selected files, creates a Git commit, and pushes through Crab's native pipeline.

Assume each new 8 GB checkpoint keeps 7.5 GB of the previous version's encoded bytes:

With ordinary Git blobs, four versions contribute 32 GB of reachable binary objects. With Crab, Git keeps four compact pointers while the bucket adds the 8 GB base and roughly 0.5 GB of new chunks per later version.

That 9.5 GB result is an example, not a guarantee. Recompression, encryption, or a container rewrite can change most bytes and leave little to reuse.

What Crab does at each step

The pointer answers “which model belongs to this commit?” The bucket answers “where are its bytes?”

In this example, the changed weights region gets a new chunk identity. Stable regions keep the same identity, so Crab can reuse them across both file recipes.

Why a bucket URL is not enough

A team can put this in a configuration file:

MODEL_URI=s3://team-models/encoder/latest.safetensors

That keeps the 8 GB file out of Git, but it leaves important questions to scripts and convention:

• Can latest change after the source commit is reviewed?
• How does another machine verify the downloaded bytes?
• What happens if the model upload succeeds but the Git push fails?
• Which object can retention policy delete safely?

A committed Crab pointer records the expected file identity and size. Crab then requires a complete reconstruction recipe before it publishes the ref.

A failed push must leave the old branch valid

A successful upload is not the same as a published commit. Crab moves the visible ref only after the Git objects, large-file data, and reconstruction metadata are durable.

A crab ship failure after the commit step leaves the local commit in place. If the problem was network access or credentials, fix it and run crab push again. Completed immutable uploads can be reused by the retry.

Select a stage below to see what readers observe at each failure boundary.

Different jobs can download different files

A commit identifies the complete repository. It does not force every machine to materialize every large file.

All four jobs use the same commits and refs. Hydration changes the working tree, not repository history.

What you pay for

Crab changes the cost; it does not remove it.

• Every client needs Crab installed.
• Every client that reads or writes model data needs scoped bucket credentials.
• The bucket still charges for storage, requests, retrieval, and network transfer.
• Chunk reuse depends on physical byte overlap, not on how similar two models seem.
• Your team owns retention and lifecycle policy.

When Crab is worth the extra machinery

Crab is a strong fit when all of these are true:

• Large files change often.
• New versions preserve substantial byte regions.
• Source and data must review and roll back as one commit.
• Different jobs need different subsets of the repository.
• The team wants direct control of object storage.

A few stable release archives may work fine in Git LFS or plain object storage. A dataset with its own catalog may not need Git identity at all.

We built Crab for the middle ground: keep Git's history and workflow, but stop making every clone carry every large-file version.

Next, compare Crab and Git LFS using your repository's constraints.