Optimizing Remote Storage Layout

Every push creates a new pack file in the remote store. Over time, dozens or hundreds of small packs accumulate. This increases S3 listing latency, adds per-object overhead costs, and slows down clone and fetch operations.

Repacking uses Git's geometric policy to roll up smaller packs into fewer, progressively larger ones. Large stable packs stay in place, so maintenance does not repeatedly rewrite the full history.

How Repacking Works

1. Pins the canonical manifest and segmented pack inventory.
2. Selects the smallest suffix whose replacement restores a factor-two geometric progression.
3. Downloads and verifies only that suffix with bounded concurrency after a free-space preflight; stable large packs stay in object storage and are not rewritten.
4. Generates one pack from the exact selected OID union and verifies that no object was added or lost.
5. Uploads only newly generated pack, index, and reverse-index objects, and repairs missing metadata sidecars.
6. Atomically updates the manifest using compare-and-swap (CAS).
7. Rebinds the existing Git visibility and complete commit-graph proofs to the replacement generation, then publishes exact locators and a generation receipt. Repairable post-CAS failures never invalidate the committed manifest.

Old pack files leave the active manifest after the update, but immutable recovery history continues to reference them. Repack therefore reduces the pack count used by clone and fetch. To reclaim old pack bytes, preview and apply an explicit history-retention boundary with crab recover history prune --keep-last N, then run repo-scoped crab gc.

Atomicity and Safety

The manifest update uses CAS to prevent concurrent maintenance or push activity from corrupting state. Apply mode holds the repository maintenance lease. If the manifest changes between repack and commit, the operation fails without replacing newer state; rerun against the new generation. Missing post-CAS evidence is repairable on a rerun without advancing an already consolidated generation. Dry-run reports current bytes; replacement size is unknown until Git performs the repack.

Usage

Preview pack statistics

crab repack --dry-run

repack dry run: 15 packs, 2400000000 bytes, 0.1s

Run the repack

crab repack

repack complete: 15 → 6 packs, 2400000000 → 2300000000 bytes, 4.2s

When to Repack

The repack_auto_threshold config option emits an advisory warning after a fetch when the pack count exceeds a threshold. It does not run consolidation on the developer's push/fetch path:

# .crab/local.toml
repack_auto_threshold = 20

Repack vs. Xorb Optimization

These optimize different things:

CLI Reference

For complete command syntax and all available flags, see the crab repack reference.