Optimizing Xorb Layout

Crab stores file content as content-addressed xorbs. Xorb size and grouping affect storage cost, object count, and access performance. crab optimize xorbs rewrites the xorbs referenced by the repository's canonical metadata into a target size profile.

This is different from crab optimize packs, which consolidates Git pack files. Xorb optimization operates on the content-addressed data layer.

Built-in Profiles

Built-in profiles retain their public Zstd level descriptions; the current Xet xorb encoder maps that config to its supported LZ4 chunk scheme. Custom profiles may select zstd:N, lz4, or none.

When you omit --profile, Crab scans the canonical file index and selects a target from the median file size:

• p50 > 100 MiB: ml
• p50 >= 1 MiB: dataset
• otherwise: code

Usage

Dry-run first:

crab optimize xorbs --profile ml --dry-run

Dry-run reads the canonical shard set and live xorb metadata. It performs no writes. Source-shard scanning is disk-backed and hash-verified, so planning does not retain several complete metadata shards in memory at once.

Apply the rewrite:

crab optimize xorbs --profile ml --apply

Apply consolidates bounded batches of source xorbs, verifies every source and destination size/hash, rebuilds affected Xet shards, publishes candidate indexes, and advances the canonical manifest with compare-and-swap. A repository maintenance lease prevents overlapping destructive maintenance; old xorbs remain immutable until a later GC proves them unreferenced.

Resume or abort an interrupted run:

crab optimize xorbs --resume
crab optimize xorbs --abort

Custom Profiles

Define profiles in .crab/local.toml:

[optimize.xorbs.profiles.my-profile]
target_xorb_bytes = 134217728   # 128 MiB
max_xorbs_per_file = 8
group_by = "file"
compression = "zstd:5"          # or "lz4" / "none"

Custom profiles live under the same command namespace as crab optimize xorbs.

Tier-Aware Optimization

If source xorbs are in archive storage classes, Crab can restore them before processing:

crab optimize xorbs --profile ml --apply --include-cold=false
crab optimize xorbs --profile ml --apply --restore-tier=bulk
crab optimize xorbs --profile ml --apply --output-class=STANDARD_IA

Concurrency

Old source xorbs become orphans and are reclaimed by the next crab gc.

CLI Reference

For complete command syntax and all available flags, see crab optimize.