Verifying Remote Store Consistency

The store verification layer within crab fsck queries your remote object storage directly to detect inconsistencies between manifests and actual stored objects. While the broader integrity check covers the full data chain, store verification focuses specifically on what's physically present in S3, GCS, or Azure versus what the manifests claim should be there.

What Gets Verified

Pack-list vs. storage

The verifier loads pack-list.json and cross-references it against actual pack objects:

• Listed but missing — a pack ID is in the manifest but the object doesn't exist in storage
• Present but unlisted — a pack object exists but isn't in the manifest

Unreferenced immutable packs are expected after interrupted or conflicted pushes and are protected by the normal GC grace period. fsck does not add them to the committed manifest.

Shard-list vs. storage

Same cross-reference for shard objects under .crab/shards/. Divergence is reported but not automatically repairable.

Data chain integrity

For each file-index entry, the verifier confirms the referenced shard exists either in the shard-list or directly in storage. Missing shards are reported as MissingFileIndex issues.

Push lock expiry

Push locks are checked against their backend-clock lease. Holder-safe repair can remove a genuinely expired lock.

Usage

Verify remote store

crab fsck

When healthy:

crab fsck: repository is clean

Repair detected issues

crab fsck --repair

crab fsck: 2 error(s), 0 info, 2 repaired, 0 repair failure(s)

Stream events for monitoring

crab fsck --jsonl

Each issue is emitted as a warning event, followed by a terminal result — useful for piping into monitoring systems.

Repair Safety

Only specific issue types are auto-repaired:

Repairs are conservative. Expired lock objects are deleted only after the holder-safe expiry check; committed repository content is not deleted.

Current Boundaries

• The current Git-object check does not reconstruct current head and run full Git reachability. Monitor an independent clone/fetch; use crab recover history verify <generation> for strict Git fsck of a historical root.
• The generic object-store API cannot enumerate provider-side incomplete multipart uploads, so configure the provider lifecycle rule that aborts old incomplete uploads.
• Locator or visibility acceleration damage requires crab metadb rebuild followed by another verification run.
• Same-bucket manifest history is recovery history, not an independent backup.

Exit Codes

CLI Reference

For complete command syntax and all available flags, see the crab fsck reference.