Optimizing Xorb Layout

Crab stores file content as content-addressed xorbs. Xorb size and grouping affect storage cost, object count, and access performance. crab optimize xorbs rewrites xorbs to match a target profile: larger xorbs for ML models, medium xorbs for datasets, and smaller xorbs for code or random-access workloads.

This is different from crab optimize packs, which consolidates Git pack files. Xorb optimization operates on the content-addressed data layer.

Built-in Profiles

When you omit --profile, Crab scans the file index and selects based on median file size:

• p50 > 100 MiB: ml
• p50 >= 1 MiB: dataset
• otherwise: code

Usage

Dry-run first:

crab optimize xorbs --profile ml --dry-run

Apply the rewrite:

crab optimize xorbs --profile ml --apply

Resume or abort an interrupted run:

crab optimize xorbs --resume
crab optimize xorbs --abort

Custom Profiles

Define profiles in .crab/config.toml:

[restripe.profiles.my-profile]
target_xorb_bytes = 134217728   # 128 MiB
max_xorbs_per_file = 8
group_by = "file"
compression = "zstd:5"

The config section still uses [restripe.profiles] because it describes the xorb rewrite engine’s stored profile configuration.

Tier-Aware Optimization

If source xorbs are in archive storage classes, Crab can restore them before processing:

crab optimize xorbs --profile ml --apply --include-cold=false
crab optimize xorbs --profile ml --apply --restore-tier=bulk
crab optimize xorbs --profile ml --apply --output-class=STANDARD_IA

Concurrency

Old source xorbs become orphans and are reclaimed by the next crab gc.

CLI Reference

For complete command syntax and all available flags, see crab optimize.