How do you set up a Crab repository?

This tutorial connects an existing Git repository to an S3 path and tracks model files. You will push one commit and verify it from a fresh clone. Replace the sample names with resources you control.

Before you begin

You need Git, an existing repository, and credentials that can read and write the target bucket path. Crab supports Amazon S3, Google Cloud Storage, and Azure Blob Storage. This tutorial uses S3.

Run the setup first in a disposable repository or a non-production object-store prefix. The first pass should prove installation, credentials, tracking, publication, and reconstruction without changing a branch used by other contributors.

Choose a representative file that you are allowed to copy into the test repository. Avoid confidential production data for the tutorial. A deterministic sample helps with hash verification, while a real producer output gives better deduplication evidence during the second push.

Record the CLI version, Git version, provider region, and destination prefix. These details make an access or behavior difference reproducible on another machine.

Install Crab on your platform

Install Crab with the package or installer that matches your development environment. Each release includes the crab command and the git-remote-crab executable that Git needs for crab:// remotes.

On macOS or Linux, install Crab with Homebrew:

brew install crabbuild/tap/crab

You can instead use Crab's checksum-verifying installer on macOS or Linux:

curl -fsSL https://crab.build/install.sh | bash

On Windows, run the PowerShell installer:

irm https://crab.build/install.ps1 | iex

On macOS or Linux, open a new shell if the installer changed your PATH, then verify both entry points:

crab version
command -v git-remote-crab

On Windows, run the equivalent checks in PowerShell:

crab version
Get-Command git-remote-crab

The version command proves that the CLI runs. The helper lookup proves that Git can find the remote helper executable. Repository setup installs the filter configuration later.

Pin CRAB_VERSION in continuous integration and build images when you need repeatable automation. Read Installation & Setup for custom install directories, global filter registration, mount helpers, and platform requirements.

Configure the repository

Run crab configure from the repository root. It connects the remote and installs the Git filter configuration used for tracked files.

cd vision-search
crab configure s3://team-data/vision-search
crab doctor

crab doctor checks the local setup, remote discovery, credentials, and object-store access. Fix any reported error before adding data.

Configuration should create one understandable ownership boundary. Use a dedicated repository prefix rather than the bucket root, and apply permissions to that prefix. A read-only client needs discovery and object reads. A writer also needs the repository's immutable uploads and ref operations.

Do not place secrets in the remote URL, .gitattributes, or committed configuration. Use the provider's supported credential chain so local profiles, short-lived roles, and workload identity can change without rewriting repository history.

Choose which files Crab tracks

Track patterns, not individual versions of the same asset type. Quote the glob so your shell does not expand it before Crab receives it.

crab track '*.safetensors'
git add .gitattributes
git commit -m "Track model files with Crab"

The generated .gitattributes rule sends matching files through Crab's clean and smudge integration. Files that do not match remain ordinary Git blobs.

Inspect the repository configuration

Before adding a large file, review the state that setup created. This separates configuration errors from data-transfer errors later in the tutorial.

git remote -v
git config --local --get-regexp '^filter\.crab\.'
git check-attr filter diff merge -- models/example.safetensors
crab track

The remote should name the intended repository path. Git should report the Crab filter for the sample model path. crab track should list the pattern you added.

Keep .gitattributes in Git history because it determines path representation for every contributor. Keep object-store credentials outside the repository. Crab discovers credentials from the configured provider flow rather than committing secrets with project configuration.

Add and ship a large file

Copy a model into the repository, then let crab ship stage, commit, and push the selected path.

cp /path/to/model-v1.safetensors models/model-v1.safetensors
crab ship models/model-v1.safetensors -m "Add model v1"

Crab stages the file through its filter, then uploads new large-file data and metadata. It pushes the Git objects and moves the remote ref after every dependency is durable.

Inspect the result before continuing. git show HEAD:models/model-v1.safetensors should show a compact pointer rather than the complete model bytes. git status should be clean, and the destination branch should name the commit produced by crab ship.

The local file remains materialized because adding a file does not need to replace your working copy with its pointer. Git's index and your working tree can therefore contain different representations of the same committed path.

Publish a second version to exercise reuse

An initial push proves storage and permissions, but it does not show deduplication. Create a realistic second version that changes a limited region without recompressing or encrypting the complete file.

cp /path/to/model-v2.safetensors models/model-v2.safetensors
crab ship models/model-v2.safetensors -m "Add model v2"
crab stat perf

Review the reported transfer and deduplication counters. The amount of reused data depends on shared bytes between the files. A different file name does not prevent reuse because chunk identity comes from content.

If the second version uploads nearly all bytes, inspect the file format before assuming configuration is wrong. Some serializers reorder data, add nondeterministic fields, or recompress the complete container after a small logical change.

Compare full-file identity and chunk reuse as different measurements. The second model should have a different full-file hash when its content changed. Reused chunk counters show how much physical data remained identical inside that new file identity.

Keep both commits during verification. A fresh client should reconstruct the first version after checking out the earlier commit and the second version after returning to the branch tip. This confirms that reuse did not collapse two distinct file recipes.

Verify from a fresh clone

Clone with Crab so the repository can check out pointers without downloading every large file immediately.

cd ..
crab clone crab://team-data/vision-search vision-search-check
cd vision-search-check
crab status
crab hydrate 'models/model-v1.safetensors'
crab status

After hydration, the working-tree file contains its original bytes. Git history still contains the compact pointer used to locate and verify those bytes.

Compare a cryptographic hash from the source repository with the hydrated result. Run the clone from a machine or environment that does not share the original repository's local cache. Otherwise, a successful hydration could prove only that the writer retained the bytes locally.

Also switch to the earlier commit and hydrate the earlier model version. The repository succeeds only when its retained history can reconstruct every version protected by your policy, not merely the newest file.

Understand what each command proved

The tutorial crosses several independent boundaries:

1. crab --version proves that the CLI runs, but not that Git can invoke its helper and filter modes.
2. crab doctor proves configuration and access checks, but not that a tracked path produces a publishable recipe.
3. git check-attr proves path selection, but not that object storage contains the file.
4. crab ship proves local preparation and remote publication for one commit.
5. A fresh clone and hash comparison prove remote discoverability and byte-exact reconstruction.

Keep these proofs separate while troubleshooting. Reinstalling the binary cannot repair a provider policy, and widening a provider policy cannot fix an uncommitted .gitattributes rule.

Read common failures by boundary

Setup problems become easier to diagnose when you classify the failing boundary:

Do not respond to an access error by widening the bucket policy to every object. Scope permissions to the repository prefix and required provider operations. crab doctor gives a narrower signal after each policy change.

Prepare automation after the manual path works

Continuous integration should install Crab, obtain short-lived credentials, clone lazily, and hydrate a declared working set. A committed manifest makes that set reviewable.

crab clone crab://team-data/vision-search ci-worktree
cd ci-worktree
crab hydrate --manifest .crab/manifests/ci.txt
crab fsck

Avoid crab hydrate --all in automation unless the job consumes every managed file. Narrow hydration reduces transfer, runner disk usage, and accidental restore requests for archived xorbs.

Pin the Crab version in build images or installation steps, and log it with the commit ID. Use a dedicated manifest per job so a lint task does not inherit the model and dataset requirements of a training task.

Treat a fresh-clone verification job as repository health evidence. It should obtain independent credentials, fetch the published ref, hydrate representative files, compare their expected hashes, and run crab fsck. Schedule broader historical checks separately when they would exceed the normal build budget.

For a more explicit verification workflow, continue with How do you verify your first Crab push?. For command options, use the crab configure, crab track, and crab hydrate references.