How does Crab avoid storing duplicate data?

Crab does not ask, “Have I seen this file before?”

It asks, “Which regions of these bytes already exist?”

A changed 10 GiB checkpoint might contain 8 GiB of familiar chunks. Crab can reuse those chunks and upload only the 2 GiB it cannot prove durable.

1. Content chooses the cut points

Fixed-size chunking cuts every n bytes. Insert one byte near the start and every later offset shifts.

Content-defined chunking examines a rolling window of bytes. The content itself decides when a chunk ends.

fixed size       [ A ][ B ][ C ][ D ]
insert one byte  [ A ][? B][? C][? D]   every offset stays shifted

content defined  [ A ][ B ][ C ][ D ]
insert one byte  [ A ][ N ][ O ][ D ]   boundaries resynchronize

An edit can change several nearby chunks. The useful property is recovery: once the rolling content becomes familiar again, later boundaries can match the earlier version.

Crab's gearhash chunker targets 64 KiB chunks. That is a target, not a promise that every chunk has the same size.

One checkpoint, worked through

Suppose models/ranker.bin is 10 GiB. Version 2 replaces one region but keeps most encoded bytes stable.

crab add models/ranker.bin
git commit -m "Update ranking model"
crab push origin main

Use this illustrative classification:

The chunk reuse ratio is 80%. The upload can be smaller than 2 GiB after compression, but xorb framing and metadata add some overhead. Reuse ratio and upload ratio are related, not identical.

2. A matching hash is only identity

Crab computes a BLAKE3 hash for each chunk. Equal bytes produce the same identity across paths, branches, commits, and file versions.

But a hash does not answer:

• Where are the bytes?
• Are they durable?
• Can another machine read them?

Select each evidence level below. Only the final level can authorize skipping a remote upload.

The distinction prevents a dangerous shortcut. Your local cache may contain a chunk while the shared origin does not. Publishing a pointer from that cache hint would create a file that works only on the writer's machine.

The authority ladder is:

1. Session evidence saves repeated work in one process.
2. Staging evidence proves this client can prepare the bytes.
3. Canonical placement and origin receipts prove remote durability.

Negative local lookup results are also provisional. Another writer may have published the chunk. Refreshing canonical state can turn a planned upload into safe reuse.

3. Identity, location, and order are separate

Crab stores three different facts:

Select a recipe term to follow it into physical storage.

This separation matters in two ways.

First, the same chunk can appear more than once in a file. The recipe repeats its identity at each logical position while storage keeps one physical copy.

Second, Crab can repack a chunk into a better xorb. Its location changes, but the chunk hash and file recipe do not.

The full file also receives a BLAKE3 identity. That final hash catches missing, reordered, or corrupted chunks during hydration.

4. New chunks become xorbs

Uploading one object per 64 KiB chunk would create too many object-store requests. Crab packs new chunks into immutable compressed objects called xorbs.

The default target is 64 MiB of compressed data, bounded between smaller and larger operational limits. A xorb may contain chunks from several files, and one file may reference several xorbs.

new chunks from file A ─┐
new chunks from file B ─┼─► xorb builder ─► immutable xorb
new chunks from file C ─┘                    hash + chunk metadata

Once uploaded, shard metadata publishes each chunk's xorb_hash, chunk_index, and uncompressed size. Readers can resolve recipes without scanning the bucket.

Packing is deliberately separate from file identity:

• Reused chunks keep their proven locations.
• New chunks enter a bounded xorb builder.
• Shards record complete reconstruction terms.
• The file recipe continues to name ordered chunk identities.

An interrupted push may leave an unreachable xorb. That object is still valid; a retry can reuse it, and garbage collection can remove it later if no retained root reaches it.

What kind of files deduplicate well?

Savings follow encoded bytes, not filenames or logical meaning.

Deterministic serialization helps. Stable record ordering, stable headers, and independently compressed partitions preserve byte regions between versions.

Do not weaken encryption or integrity controls for deduplication. When randomized ciphertext is required, plan for whole-stream change and optimize retention or partitioning instead.

Measure three different ratios

Use two representative outputs from the real producer—not zero-filled synthetic files.

# Baseline version
crab add --jsonl models/ranker.bin | tee add-v1.jsonl
git commit -m "Add baseline model"
crab push --jsonl | tee push-v1.jsonl

# Changed version
crab add --jsonl models/ranker.bin | tee add-v2.jsonl
git commit -m "Update model"
crab push --jsonl | tee push-v2.jsonl

Compare:

• Chunk reuse ratio: reused source bytes ÷ total source bytes.
• Upload ratio: uploaded xorb bytes ÷ total source bytes.
• Request density: object-store operations ÷ uploaded GiB.

Keep cold hydration separate. Deduplication reduces new storage and push transfer. A new reader still needs all ranges required to reconstruct the 10 GiB file; a warm cache may avoid more origin traffic.

Next, trace how those proven objects become visible through one ref transaction.