01 Install & Init
One command to install, one to connect your bucket. Crab auto-detects large files and configures git tracking.
$ crab init s3://my-bucket/repoNow in Private Beta
Git for any file at any scale
Crab is a serverless Git remote storage solution powered by the Xet protocol for chunk-level deduplication — version huge binaries straight into your S3, GCS, or Azure bucket.
Serverless Git for Large Files
Git Repo
commitsbranchespointer blobs
push / pull
Crab
git remote helper
CDC chunking
Dedup & pack
Filter & VFS
xorbs / shards
Cloud Storage
AWS S3Google Cloud StorageAzure Blob
How It Works
Three steps to version and store your large files — no servers, no proprietary lock-in.
02 Ship Your Files
A single command stages, commits, and pushes. Crab handles chunking, deduplication, and parallel upload.
$ crab ship . -m "add model v2"03 Stored in Your Cloud
Files are deduplicated and packed into xorbs in your own S3, GCS, or Azure bucket. No server needed.
$ crab://my-bucket/repo ✓Core Features
Everything you need to version any file
One serverless toolchain across the CLI, desktop app, and cloud — covering large-file Git, deduplicated storage, AI agents, and ML pipelines.
- Standard Git UXgit clone, add, commit, push — unmodified. Crab is a remote helper plus filter driver, so every workflow you already know just works.
- Chunk-Level DedupContent-defined chunking with Gearhash plus a 3-tier session, shard, and database index. Identical chunks upload exactly once.
- Cloud-Native StorageRepositories live in S3, GCS, or Azure Blob. No servers to run, no databases to operate, no LFS endpoint to scale or pay for.
- Virtual FilesystemMount repositories and hydrate file chunks on demand. Browse huge trees without pulling every object into your checkout.
- ML Pipeline WorkflowsVersion datasets, weights, and checkpoints alongside code. Lazy checkout pulls only the chunks each pipeline stage actually needs.
- Desktop ApplicationCross-platform GUI with file previews, integrated terminal, SSH remote workspaces, and Crab operations on top of the CLI.
Comparison
How Crab stacks up
See how Crab compares to other popular tools for managing large files, datasets, and ML assets alongside your code.
| Feature | Crab | Git LFS | DVC | HF Hub |
|---|---|---|---|---|
| Chunk-level deduplication | ||||
| Serverless (no infra) | ||||
| Standard Git CLI | Partial | |||
| Desktop App | ||||
| FUSE virtual filesystem | ||||
| ML pipeline engine | ||||
| Cloud-native storage | ||||
| Open source | Planned | Partial |
Interactive Demo
See it in action
Two common workflows — starting fresh or cloning an existing repo. Full end-to-end from init to push.
Architecture
Content-Defined Chunking & 3-Tier Dedup
Files are split at natural content boundaries using Gearhash CDC. Duplicate chunks are identified across three tiers — minimizing storage costs even for large binary datasets.
Class AExisting — already on remote, skipped entirely.
Class BStaged — in local staging, needs packing & upload.
Class CNew — never seen before, staged, packed, uploaded.
Testimonials
Trusted by ML teams shipping at scale
“Chunk-level dedup turned our 1,212 GB checkpoint repo into a 10 GB push. We swapped Git LFS for Crab in an afternoon and never looked back — pulls are instant and the bucket bill dropped by an order of magnitude.”
FAQ
Frequently asked questions
Everything you need to know about Crab
What is Crab?
Crab is a serverless Git remote storage solution powered by the Xet protocol. It acts as a Git remote helper and filter driver, enabling you to version any file — regardless of size — directly into your own cloud storage bucket (S3, GCS, or Azure Blob).
How is Crab different from Git LFS?
Git LFS stores files as whole objects on a dedicated server you have to run and scale. Crab uses content-defined chunking (CDC) to split files at natural byte boundaries and deduplicates at the chunk level. There's no server — data goes straight to your cloud bucket. Identical chunks are uploaded exactly once, even across different files and repos.
What cloud providers are supported?
Crab supports Amazon S3, Google Cloud Storage (GCS), and Azure Blob Storage. Your data stays in your bucket under your control. We also support S3-compatible endpoints like MinIO and Cloudflare R2.
How does chunk-level deduplication work?
Crab uses Gearhash content-defined chunking (CDC) to split files at natural content boundaries. Each chunk is hashed and checked against a 3-tier deduplication index (session, shard, and database). Chunks that already exist on the remote are skipped — only truly new data is uploaded. This typically saves 50–90% on storage for versioned binary files.
Can I use Crab with my existing Git repos?
Absolutely. Crab works with standard Git commands — git clone, add, commit, push. You add a Crab remote to your existing repo and push. If you're migrating from Git LFS, Crab can detect LFS pointers and store those files alongside xorbs without re-uploading.
How do I get started?
Install Crab with a single command: brew install crabbuild/tap/crab on macOS, or cargo install crab on any platform. Then run crab init in your repo to configure a cloud remote. Check out our docs for a complete quickstart guide.
Quickstart
Get started in seconds
Install the Crab CLI for your platform and initialize your first serverless remote bucket.
# Install using Homebrew$ brew install crabbuild/tap/crab==> Fetching crabbuild/tap/crab...==> Installed crab v0.8.4 # Initialize Crab in your repository$ crab init --bucket my-s3-bucket --region us-west-2Initialized serverless Crab remote.Requires Homebrew. For manual binary installation, see GitHub releases.
Ready to handle any file at any scale?
Install the Crab CLI in minutes, or grab the desktop app for a full GUI with previews, integrated terminal, and remote workspaces.
Built for scale
Numbers that hold up under big repositories
Crab is engineered for ML model weights, datasets, and game assets that break Git LFS. Every metric below comes from the shipping CLI.
SIMD-accelerated Gearhash CDC throughput on a single core.
Concurrent xorb transfers saturate cloud bandwidth on push.
Session, shard, and database index — chunks uploaded once.
Packed chunks balance Range GET cost against round trips.