Params, Metrics, and Plots

Params explain why a run changed. Metrics and plots explain what changed. Together they make workflow commits and experiments reviewable.

Params

Declare params files at the top level:

params:
  - params.yaml
  - configs/train.yaml

Reference individual keys from stages:

stages:
  train:
    params:
      - train.lr
      - train.epochs

Inspect the current flattened params map:

crab params show
crab params show --json

Compare params across refs:

crab params diff main HEAD
crab params diff --json main HEAD

Use dotted keys for nested values. Keep params files small enough that review diffs are meaningful.

Metrics

Metrics should be generated by stages and committed when they are small:

metrics:
  - metrics/train.json

stages:
  train:
    metrics:
      - metrics/train.json

Example metric file:

{
  "accuracy": 0.91,
  "loss": 0.18
}

Show metrics:

crab metrics show
crab metrics show --json

Compare refs or the current workspace:

crab metrics diff main HEAD
crab metrics diff --json

Prefer stable metric names and units. A rename looks like one metric removed and another added.

Plots

Plots can be declared at the top level or on stages:

plots:
  - metrics/loss.csv:
      x: epoch
      y: val_loss

stages:
  train:
    plots:
      - metrics/loss.csv:
          x: epoch
          y: val_loss

Render plots:

crab plots show
crab plots show --format html --output crab_plots/index.html
crab plots show --show-vega --out plots.vl.json

Compare plots:

crab plots diff main HEAD
crab plots diff --format html --output plot-diff.html

Crab supports CSV, TSV, JSON, YAML, and image plot sources. Use crab plots templates to list built-in and local Vega-Lite templates.

PR Review Pattern

For a pull request that changes training behavior:

crab params diff origin/main HEAD --json > params.diff.json
crab metrics diff origin/main HEAD --json > metrics.diff.json
crab plots diff origin/main HEAD --format html --output plot-diff.html

Attach the JSON summaries and HTML plot output to CI artifacts. Reviewers can see both the setting change and the result change.

Experiments Pattern

Run experiments with named overrides:

crab exp run -S train.lr=0.001 -n low-lr --json
crab exp run -S train.lr=0.01 -n high-lr --json
crab exp show low-lr --json
crab exp diff low-lr high-lr --json

Use stable experiment names for human workflows, and keep UUIDs in automation where exact identity matters.

Hygiene

• Commit small metrics that reviewers should see.
• Store large generated dashboards as Crab-managed outputs or CI artifacts.
• Keep plot source paths stable across branches.
• Prefer JSON for scalar metrics and CSV/TSV for curves.
• Use --json output in automation instead of parsing text.

See Metrics Tracking, Parameters, and DAG Visualization for command-level details.