Workflow Quickstart

This quickstart creates a two-stage pipeline, runs it, records metrics, and pushes the stage cache so another clone or CI job can reuse the result.

Prerequisites

You need a Crab repository with a configured remote:

mkdir ml-project
cd ml-project
git init
crab init crab://my-bucket/ml-project
# Optional: set workflow.enabled false to opt out; fresh configs are enabled.

Fresh Crab configs enable workflow commands. Existing repositories can opt out explicitly with crab config set workflow.enabled false.

Create a Small Project

mkdir -p src data models metrics

cat > data/raw.csv <<'EOF'
x,y
1,2
2,4
3,6
EOF

cat > params.yaml <<'EOF'
train:
  multiplier: 2
EOF

Add tiny scripts for the example:

cat > src/prepare.py <<'EOF'
from pathlib import Path
raw = Path("data/raw.csv").read_text()
Path("data/prepared.csv").write_text(raw.replace(",", "\t"))
EOF

cat > src/train.py <<'EOF'
import json
from pathlib import Path
Path("models/model.txt").write_text("trained\n")
Path("metrics/train.json").write_text(json.dumps({"accuracy": 0.91}))
EOF

Define the Pipeline

Create crab.yaml:

params:
  - params.yaml

metrics:
  - metrics/train.json

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw.csv
    outs:
      - data/prepared.csv

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/prepared.csv
    params:
      - train.multiplier
    outs:
      - models/model.txt
    metrics:
      - metrics/train.json

Validate before running:

crab run --validate

Run and Inspect

crab run
crab workflow status
crab metrics show
crab workflow dag --format mermaid

The first run executes both stages and writes crab.lock. A second run should skip unchanged stages:

crab run --dry

Use --explain-miss when a stage runs and you expected a cache hit:

crab run --dry --explain-miss

Commit the Reproducible State

Commit the workflow definition, params, lockfile, scripts, and small metrics:

git add crab.yaml crab.lock params.yaml src data/raw.csv metrics/train.json
git commit -m "add reproducible workflow"

Large outputs can be tracked with Crab before committing if your project produces models, datasets, or reports that should live in object storage:

crab track "models/**"
crab add models/model.txt
git add .gitattributes models/model.txt
git commit -m "track model output"

Share the Stage Cache

Push git state and stage cache:

crab push
crab workflow push-cache --all

For day-to-day runs, publish cache entries as each stage completes:

crab run --cache-push

A clone or CI job can reuse remote cache entries:

crab clone crab://my-bucket/ml-project
cd ml-project
crab config set workflow.enabled true
crab run --cache-only

--cache-only exits with code 3 on a cache miss. That is useful in CI when a job should prove the committed lockfile and remote cache are complete.

Verify the cache boundary

The clean-clone command is the decisive check because the original workspace still has local outputs and stage-cache entries. In the clone, remove any generated outputs before --cache-only if they were committed for review, then confirm the command restores the declared outputs without executing the Python scripts. Inspect crab workflow status and crab metrics show after replay.

If the clone exits with code 3, rerun the producer with --cache-push or push the existing entries with crab workflow push-cache --all. Do not replace the cache-only check with a normal run when CI is intended to prove publication; normal execution can hide a missing remote cache entry by recomputing it.

Next Steps

• Learn the mental model in Concepts.
• Add richer stages in Authoring Stages.
• Publish cache from CI with Remote Cache and CI.