Automation & Pipelines

Crab includes a workflow engine for running reproducible commands with content-addressed caching. Define multi-step pipelines, track experiments, and let Crab skip steps whose inputs haven't changed.

Key Concepts

• Content-addressed caching — Commands are cached by the hash of their inputs. If nothing changed, the step is skipped.
• Pipeline DAGs — Define dependencies between steps. Crab runs them in topological order.
• Experiment tracking — Log parameters, metrics, and outputs for each run.

Topics in This Section

• Running Commands — Execute cached commands with crab run
• Workflow Pipelines — Define multi-step pipelines
• Workflow Journal — Inspect pipeline execution history
• Workflow Lockfile — Pin pipeline state for reproducibility
• Experiments — Track experiment runs and parameters
• Experiment Queue — Queue and schedule experiment runs
• Parameters — Define and sweep hyperparameters
• Metrics Tracking — Log and compare metrics across runs
• DAG Visualization — Visualize pipeline dependency graphs