Distributed Locking and Consistency in Crab

Multiple People Can Push Safely at the Same Time

Two teammates hit git push to the same branch within the same minute. One will win, one will need to rebase — and neither should ever silently lose work. Traditional git hosting solves this with a central server that accepts one push and rejects the other.

Crab doesn't have a server. Your repository lives entirely in cloud object storage — S3, GCS, or Azure. So how does it keep things safe when several people push at once?

The answer is three small mechanisms working together: a push lock that prevents wasted uploads, a write journal that survives crashes, and a compare-and-swap ref update that makes the final commit atomic. None of them need a server.

What You'll Learn (TL;DR)

• No silent overwrites. Crab uses compare-and-swap (CAS) so the last writer never silently wins. It works the same way Google Docs flags a conflict when two people edit the same paragraph.
• No wasted uploads. A short-lived lock — like reserving a meeting room before booking it — stops two people from uploading hundreds of megabytes only to have one rejected.
• Crash-safe. A local write journal records what's already uploaded, so an interrupted push resumes instead of starting over.
• No external services. Everything runs on the object store's built-in conditional write primitives. No ZooKeeper, no etcd, no DynamoDB.

How a Safe Push Works, Step by Step

Every push follows a strict order: acquire a lock, upload your data, update the ref atomically, then release the lock. If anything fails midway, the system recovers without leaving the repo in a broken state.

That ordering rule is what makes it safe to read concurrently. Anyone who sees a ref update is guaranteed that all the data behind it already exists. There is never a window where a ref points to missing chunks.

If a push crashes between steps 2 and 3, no other client can see partial state — the ref still points at the old commit. The new data sits in the object store as orphans until either the push retries (and uses them) or garbage collection cleans them up.

No Silent Overwrites with Compare-and-Swap

The most important mechanism is compare-and-swap (CAS). The intuition is simple:

Think of it like editing a paragraph in Google Docs. If someone else changed the same paragraph while you were typing, the doc tells you instead of silently losing their edit. CAS gives git refs the same property.

When Crab updates a ref like refs/heads/main, it doesn't blindly overwrite the current value. Instead it says: "Set this ref to commit abc123, but only if it currently points to def456." If someone else pushed in between, the object store rejects the update. Crab then re-reads the latest state, re-applies your change on top, and tries again.

The whole pattern is a short read-modify-write loop with conflict detection:

// Simplified CAS update loop
for attempt in 0..max_attempts {
    let (current_value, etag) = store.get_with_etag(path).await?;
    let new_value = apply_mutation(current_value);
    match store.conditional_put(path, new_value, etag).await {
        Ok(_)             => return Ok(()),  // No conflict — done.
        Err(CasConflict)  => continue,       // Lost the race — retry with fresh state.
        Err(e)            => return Err(e),  // Real error — bail.
    }
}

The key detail is on the second line: every retry re-reads and re-applies the mutation. Concurrent writers therefore converge — each one folds in the other's changes before writing its own. There is no path where one writer's update silently disappears.

Object stores expose this primitive cheaply. S3 has conditional writes via If-Match ETags, GCS has generation preconditions, Azure has ETags too. Crab leans on whichever the active backend offers. No extra database, no consensus protocol, no clock sync.

Avoiding Wasted Uploads with the Push Lock

CAS makes the final ref update safe, but on its own it would let two people upload 500 MB of data simultaneously — and then reject one of them at the very last step. That's a slow, expensive way to discover a conflict.

The push lock fixes that. It's a small lease file at {repo}/locks/refs/{branch}/lock in the object store itself. Acquiring the lock is a conditional PUT that only succeeds when the file doesn't already exist. No external coordinator needed.

Think of it like reserving a meeting room before booking it. You only commit to the meeting once you know the room is yours; otherwise you'd both show up and one of you wastes the trip.

The lock has four properties that keep it from becoming a liability:

• Acquisition. A conditional PUT — only succeeds if the lock file doesn't exist yet.
• Heartbeat. A background task renews the lease while the push runs, so long uploads don't hit the TTL.
• Expiry. If the holder crashes or loses network, the lock falls off after its TTL (default: 5 minutes). No human intervention.
• Release. A normal delete when the push finishes successfully.

The lock is a serialization optimization, not the safety mechanism. CAS is what guarantees correctness — even if the lock disappeared entirely, you'd still never get a silent overwrite, just more wasted uploads.

Recovering from Crashes with the Write Journal

Pushes can run for a long time. A 500 MB upload over a flaky coffee-shop Wi-Fi connection can take minutes, and laptops sleep, networks drop, and processes get killed. You don't want any of that to mean "start the upload from scratch."

The write journal is a tiny local file that records exactly which xorbs (Crab's compressed bundle of file chunks) and shards have been uploaded so far. When you retry a failed push, Crab reads the journal, checks that the already-uploaded data is still in the store, and resumes from the next missing piece.

The journal also enforces the strict ordering Crab depends on:

Once the ref is updated and the lock released, the journal entry is deleted. The push is officially complete and forgotten.

A Real-World Race: Alice and Bob Push at Once

Here's how all three mechanisms cooperate during a realistic conflict:

1. Alice acquires the lock on main. Bob calls git push a second later, sees the lock is held, and starts polling. He has not uploaded anything yet.

2. Alice uploads, CASes, releases. Her xorbs go up first, her shard manifest second, her ref last. The lock comes off cleanly.

3. Bob acquires the lock. It's free now. He starts uploading. Most of his chunks are deduplicated against Alice's recent push (their branches share history), so the actual data transfer is small.

4. Bob's CAS fails. His expected parent commit is stale — Alice's commit is now HEAD. He sees the same non-fast-forward error you'd get from any git host.

5. Bob pulls and rebases. Alice's data is already cached locally from the metadata sync. After rebasing, Bob pushes again. The journal knows his xorbs are already uploaded, so the retry only updates the manifest and ref. Done in seconds.

The experience is identical to pushing against a traditional git server. No silent overwrites, no surprises, no manual cleanup.

How the Pieces Fit Together

Every step in the sequence is either idempotent (safe to retry) or conditional (fails cleanly on conflict). There is no two-phase commit, no distributed transaction, no consensus protocol — just the object store's conditional writes doing the heavy lifting.

Why This Design Works Without a Server

Availability over strict ordering. Pushes to different branches never block each other. Same-branch pushes are serialized by the lock, but if the holder crashes, the system self-heals within the TTL window — no on-call paging.

No external dependencies. The object store's conditional write primitives provide all the coordination Crab needs. That's one fewer service to operate, monitor, secure, and pay for.

Familiar conflict UX. When conflicts do happen, they surface as the same ! [rejected] (non-fast-forward) message developers already know from GitHub, GitLab, or any other git host. Pull, rebase, push. Same workflow, no central server needed to enforce it.