Build reproducible workflows

Crab Workflow adds reproducible pipeline stages to repositories that already use Crab for large files. You declare a command, its inputs, and its outputs. Crab then records the successful state, reuses matching results, and helps teams compare metrics and experiments without adding a separate data server.

Workflow execution path

Scroll horizontally to explore the full diagram →

Use Workflow when a repository has commands that should be reproducible: data preparation, model training, evaluation, report generation, asset builds, or any expensive step where "same inputs, same outputs" should be reusable.

What Workflow adds

Crab's file layer handles large content in git by storing pointers in commits and bytes in object storage. Workflow sits above that layer:

The workflow layer is additive. A repo can use Crab for large files without Workflow, and a Workflow repo still uses normal git commits and branches.

Separate committed state from local runtime state

A workflow repo usually commits these files:

• crab.yaml and optional *.workflow.yaml files declare stages.
• params.yaml or other params files hold values used by stages.
• crab.lock and optional *.workflow.lock files record the exact state that ran successfully.
• Source files, scripts, configs, and small metrics live in git.
• Large stage outputs can be Crab-managed files backed by object storage.

Local runtime state stays out of git:

• .crab/cache/stages/ stores local stage cache entries.
• .crab/workflow/runs/ stores run journals for crash recovery and audit.
• .crab/workflow/exp/ stores local experiment metadata.
• Queued experiment logs and temporary worktrees are local until pushed.

Use DVC-style concepts with Crab storage

Workflow is inspired by DVC's pipeline and experiment model, but it is designed to live inside Crab's serverless git remote:

• Stages use familiar deps, outs, params, metrics, and plots.
• crab repro is an alias for crab run.
• crab stage add authors DVC-style stage definitions.
• crab exp runs experiments in temporary worktrees.
• crab queue and crab exp queue run batches of experiment overrides.
• crab workflow push-cache --all shares stage cache through the configured Crab remote.

If you are moving from DVC, start with Migrating from DVC.

Follow the learning path

1. Quickstart: run a small pipeline with Workflow.
2. Concepts: understand stages, cache, lockfiles, journals, and experiments.
3. Authoring stages: write production crab.yaml files.
4. Running pipelines: use target modes, cache controls, and JSON output.
5. Experiments and Hydra: compare parameterized runs.
6. Remote cache and CI: share cache results with teams and automation.
7. Operations and troubleshooting: recover from common failures.

Command-level details remain in Automation & Pipelines and CLI Reference.