Crab's Multi-Layer Caching Architecture

How Crab Avoids Repeating Network Work

Every byte of your Crab repository lives in cloud storage. There is no server between you and S3, so when Crab needs data it either finds it locally or fetches it from the bucket. The cache's job is to make that second option rare for data and metadata you have already touched.

It works the same way a CPU keeps your laptop responsive: the data you touch most often stays in the closest, fastest layer, and only the rare miss reaches all the way out to main memory. Or, if you prefer kitchens to chips: counter, fridge, pantry, grocery store. The closer the layer, the cheaper the lookup.

What You'll Learn

• Why most Crab operations resolve entirely from your local disk
• How four cache layers cooperate to minimise network calls
• Why the cache repairs itself instead of needing manual cleanup
• How push-time warming avoids round-tripping data you just uploaded

Key Takeaways

• Four layers, one rule — check local state before the object store. L1 → L2 → L3 → remote, in that order.
• Self-healing — Blake3 verification on every read catches and replaces corrupted entries automatically.
• Lower cold-start cost — metadata warming means the first hydrate can spend less time asking where chunks live.
• Push-to-cache — bytes you just uploaded are immediately available locally, no round-trip to the cloud.

How the Four Layers Fit Together

When Crab needs data — for a hydrate, a diff, a checkout, a FUSE read — it checks four places in order, fastest first. Most of the time it never makes it past the second layer.

The waterfall is strict: a hit at any layer returns immediately, and only a complete miss reaches the network. When L4 does run, the bytes it pulls back get written into L1 before they're returned, so the next request takes the fast path.

L1 — The Chunk Cache (Kitchen Counter)

The primary cache lives at ~/.cache/crab/. It's a content-addressed disk cache that stores individual chunks, shard indexes, and full xorb archives — the compressed bundles Crab ships across the wire. Think of it as the kitchen counter: ingredients you reach for every day are right there.

Every read is verified against a Blake3 hash. If a chunk is corrupted by a bad disk write or an interrupted download, Crab notices, evicts the bad entry, and re-fetches it. There's no manual cleanup step — the cache heals itself as a side effect of being used.

It also stays small on its own. The cache uses LRU eviction, where every successful read updates the entry's timestamp. When the cache grows past its size budget, the coldest files get removed first. Data from your active branch stays warm; the experiment you abandoned three months ago drops off naturally.

[cache]
# Maximum cache size on disk (default: 10 GB)
chunk_max_bytes = 10_737_418_240

# Override location — handy on CI runners or fast NVMe drives
root = "/fast-nvme/.cache/crab"

[cache.warming]
prefetch_shards = true
prefetch_concurrency = 8

L2 — The Staging Area (Pantry)

The staging area at .crab/staging/ exists primarily to buffer chunks before a push, but it doubles as a powerful implicit cache. Picture pushing a 2 GB model file: the chunks are staged locally, then uploaded to S3, but the local copies don't disappear the moment the upload finishes.

If a teammate clones the repo an hour later and you need to verify something in that same file, Crab checks the staging area's index first. If the chunks are still there — and they usually are — it reads them locally without touching the network. It's the pantry: not as immediate as the counter, but still in your house.

The staging area also catches the middle case during dedup checks on push. Crab's three-tier dedup check looks like this:

Most chunks short-circuit at one of the first two checks, so the global lookup stays rare and the network stays quiet.

L3 — Metadata Warming (Shopping List)

Here's a problem fresh clones run into. Before any actual data transfer can begin, Crab needs to know where every chunk lives in cloud storage. With no local metadata that means hundreds of sequential lookups just to build the map — pure latency, before a single byte of useful data moves.

Crab works around this by warming metadata in the background during clone. While you're getting set up, it prefetches shard indexes and file-index entries. By the time you run your first crab hydrate, many of the "where is everything?" questions are already answered locally and the network can spend more time on actual file content.

The difference shows up clearly the first time you hydrate after a fresh clone:

It's the difference between wandering every aisle of the store looking for ingredients and walking in with a shopping list already written.

L4 — Remote Fetch With Push-to-Cache

L4 is the fall-through layer: a range request to S3 (or GCS, or Azure) for the bytes nothing else had. It's the only layer that actually moves data over the network, which is exactly why the layers above it work so hard to avoid it.

The most wasteful pattern any storage system can fall into is uploading data and then immediately downloading it again. You just had those bytes in memory — fetching them from the cloud a moment later is pure waste. Crab eliminates the round-trip entirely: at the end of every successful push, uploaded xorbs are written directly into L1.

This is safe because xorbs are immutable and content-addressed. There's no stale-cache problem — the content can't change. Freshly-pushed data lands at the "most recently used" end of the LRU, so it stays warm for the operations that follow.

A simplified version of the eviction logic looks like this:

// Conceptual: keep cache under budget, drop coldest entries first.
fn evict_to_budget(cache: &mut Cache, budget_bytes: u64) {
    while cache.total_bytes() > budget_bytes {
        let coldest = cache.entries_by_atime().next();
        match coldest {
            Some(entry) if entry.is_pinned() => break,
            Some(entry) => cache.remove(&entry.key),
            None => break,
        }
    }
}

The actual implementation is more careful — it batches removals, fsyncs metadata, and cooperates with concurrent reads — but the core idea is the same. Old, untouched entries make room for new ones, and freshly pushed xorbs start at the warm end of the queue.

What This Means in Practice

For a typical development workflow, the layers compound:

1. You clone a repo. L3 metadata warming prefetches shard indexes in the background.
2. You hydrate files. Chunks resolve from the warmed L3 map, and downloads are bandwidth-bound from the very first byte.
3. You make changes and push. Only new chunks upload (dedup handles the rest), and those chunks land in L1 on the way out.
4. You hydrate or diff later. Warm chunks and metadata resolve locally; cold ranges fall through to the object store and then become cacheable.

After the initial clone and first reads, everyday operations touch S3 less often. The cache layers cooperate so that remote fetch becomes the exception for warm working sets, not the default for every operation.