Run an ML experiment sweep without dirtying your workspace

The runnable fraud model uses learning rate 0.2 and L2 regularization 0.01. The team wants to evaluate nine combinations without repeatedly editing params.json or overwriting the current model.

Crab experiments start from HEAD, apply declared overrides in a temporary worktree, execute the DAG, and record parameters, stage hashes, metrics, and the base commit.

The displayed scores are illustrative. Your experiment records come from the metrics generated by your workflow.

Commit the baseline first

crab run --validate
git status --short
git add crab.yaml crab.lock params.json data/ src/ metrics/ plots/
git commit -m "record fraud model baseline"
crab exp save -n baseline

Temporary worktrees isolate experiment files from the current workspace, but their starting point is still the committed baseline.

Run two named candidates

crab exp run \
  -S train.learning_rate=0.1 \
  -S train.l2=0.0 \
  -n lr-0-1-l2-0 \
  --json

crab exp run \
  -S train.learning_rate=0.2 \
  -S train.l2=0.01 \
  -n lr-0-2-l2-0-01 \
  --json

Inspect and compare before bringing any files into the workspace:

crab exp ls
crab exp show lr-0-2-l2-0-01 --json
crab exp diff lr-0-1-l2-0 lr-0-2-l2-0-01 --json

Use human names for exploration and the immutable experiment IDs returned in JSON for automation.

Queue the full Cartesian product

Two repeatable -S flags with comma-separated values create nine tasks:

crab exp queue \
  -S train.learning_rate=0.1,0.2,0.3 \
  -S train.l2=0.0,0.01,0.1 \
  -m "fraud model learning-rate and L2 sweep" \
  --json

Start bounded local workers:

crab queue start --jobs 2
crab queue status --json

crab queue is a local workstation or CI-runner scheduler, not a distributed cluster scheduler. On GPU machines, choose a job count that matches the training script's device allocation.

Operate tasks by ID

Use the task IDs from queue status:

crab queue logs <task-id>
crab queue kill <task-id>
crab queue remove --failed

For planned shutdown, stop workers gracefully instead of killing active tasks:

crab queue stop
crab queue status

Keep failure logs until you understand whether the problem came from code, input materialization, resources, or a parameter combination.

Select a result deliberately

After the queue finishes:

crab exp ls
crab exp show <winner-id>
crab exp diff baseline <winner-id>

Apply a completed snapshot when you want its files in the current workspace:

crab exp apply <winner-id>

Create a branch boundary when the candidate needs normal Git review:

crab exp promote <winner-id> --branch experiments/fraud-candidate

Inspect the exact subcommand behavior before choosing between apply and promote: applying changes workspace files; promoting creates a Git branch boundary.

Share and clean up

Experiment metadata is local until pushed:

crab exp push <winner-id>

A reviewer can then retrieve and inspect it:

crab exp pull <winner-id>
crab exp show <winner-id>

After preserving selected results:

crab exp remove old-candidate
crab exp gc --keep 100 --dry-run
crab exp clean

Continue by reviewing params, metrics, and plots together.