Review ML parameters, metrics, and plots as one evidence chain

The best recall in a sweep is not automatically the best production model. Fraud systems trade recall against customer friction, precision, latency, and policy thresholds. Review should connect the exact inputs to scalar metrics and curve behavior.

Inspect a nine-run parameter sweep

Scroll horizontally to explore the full diagram →

L2 0.0

L2 0.01

L2 0.1

lr 0.1

lr 0.2

lr 0.3

crab exp show lr-0-2-l2-0-01 --json

Select a result to inspect its illustrative metrics. Crab records the real parameter overrides, stage hashes, and metrics for each experiment; this browser grid is an explanation, not an experiment runner.

Declare stable evidence paths

The evaluation stage in crab.yaml produces a JSON metrics file and a CSV plot source:

crab.yaml

metrics:
  - metrics/evaluation.json

plots:
  - plots/precision_recall.csv:
      x: recall
      y: precision

stages:
  evaluate:
    cmd: python3 src/evaluate.py --config params.json
    deps:
      - src/common.py
      - src/evaluate.py
      - models/fraud-model.pkl
      - data/features.csv
    params:
      - evaluate.threshold
      - evaluate.minimum_recall
    metrics:
      - metrics/evaluation.json
    plots:
      - plots/precision_recall.csv:
          x: recall
          y: precision

Use stable names and units:

metrics/evaluation.json

{
  "accuracy": 1.0,
  "f1": 1.0,
  "precision": 1.0,
  "recall": 1.0,
  "samples": 6,
  "threshold": 0.5
}

Inspect the current run

These commands read recorded files. They do not execute evaluate if evidence is missing:

crab params show --json
crab metrics show --json
crab plots show --format html --output crab_plots/index.html

If a key is absent, verify that its file exists and is declared in the workflow before assuming the reader command is wrong.

Compare the candidate with the baseline

For Git revisions:

crab params diff origin/main HEAD --json > params.diff.json
crab metrics diff origin/main HEAD --json > metrics.diff.json
crab plots diff origin/main HEAD \
  --format html \
  --output plot-diff.html

For isolated experiments:

crab exp show <candidate-id> --json
crab exp diff <baseline-id> <candidate-id> --json

A useful review packet contains:

Evidence	Review question
Parameter diff	Which learning rate, depth, seed, or threshold changed?
Metric diff	Did recall improve without unacceptable false positives or latency?
Plot overlay	Did the curve improve across the operating region, not only at one point?
Stage hashes	Were upstream data and feature stages actually comparable?

Treat quality gates as domain policy

Crab proves workflow identity and recorded bytes. It does not decide that 0.841 recall is safe for production. Put domain checks in evaluation code or CI:

crab run evaluate --json
python3 src/check_quality_gate.py metrics/evaluation.json

The script might require minimum recall, maximum false-positive rate, latency, fairness slices, and schema compatibility. Keep those thresholds reviewable in params or policy files.

Render the same visualization everywhere

List the available plot templates:

crab plots templates

Commit plot declarations and source data when they are small and useful for later comparison. Store bulky rendered dashboards as CI artifacts or Crab-managed outputs.

Make CI emit structured evidence

crab run --validate
crab run evaluate --cache-only --jsonl | tee evaluate.events.jsonl
crab params diff origin/main HEAD --json > params.diff.json
crab metrics diff origin/main HEAD --json > metrics.diff.json
crab plots diff origin/main HEAD --format html --output plot-diff.html

The cache-only execution proves the candidate result was published. The three diffs make it reviewable. Neither substitutes for application-specific quality checks.

Once the candidate passes review, capture its model bytes as an immutable workflow artifact.

Inspect a nine-run parameter sweep

Scroll horizontally to explore the full diagram →

L2 0.0

L2 0.01

L2 0.1

lr 0.1

lr 0.2

lr 0.3

crab exp show lr-0-2-l2-0-01 --json

Declare stable evidence paths

The evaluation stage in crab.yaml produces a JSON metrics file and a CSV plot source:

crab.yaml

metrics:
  - metrics/evaluation.json

plots:
  - plots/precision_recall.csv:
      x: recall
      y: precision

stages:
  evaluate:
    cmd: python3 src/evaluate.py --config params.json
    deps:
      - src/common.py
      - src/evaluate.py
      - models/fraud-model.pkl
      - data/features.csv
    params:
      - evaluate.threshold
      - evaluate.minimum_recall
    metrics:
      - metrics/evaluation.json
    plots:
      - plots/precision_recall.csv:
          x: recall
          y: precision

Use stable names and units:

metrics/evaluation.json

{
  "accuracy": 1.0,
  "f1": 1.0,
  "precision": 1.0,
  "recall": 1.0,
  "samples": 6,
  "threshold": 0.5
}

Inspect the current run

These commands read recorded files. They do not execute evaluate if evidence is missing:

crab params show --json
crab metrics show --json
crab plots show --format html --output crab_plots/index.html

If a key is absent, verify that its file exists and is declared in the workflow before assuming the reader command is wrong.

Compare the candidate with the baseline

For Git revisions:

crab params diff origin/main HEAD --json > params.diff.json
crab metrics diff origin/main HEAD --json > metrics.diff.json
crab plots diff origin/main HEAD \
  --format html \
  --output plot-diff.html

For isolated experiments:

crab exp show <candidate-id> --json
crab exp diff <baseline-id> <candidate-id> --json

A useful review packet contains:

Evidence	Review question
Parameter diff	Which learning rate, depth, seed, or threshold changed?
Metric diff	Did recall improve without unacceptable false positives or latency?
Plot overlay	Did the curve improve across the operating region, not only at one point?
Stage hashes	Were upstream data and feature stages actually comparable?

Treat quality gates as domain policy

Crab proves workflow identity and recorded bytes. It does not decide that 0.841 recall is safe for production. Put domain checks in evaluation code or CI:

crab run evaluate --json
python3 src/check_quality_gate.py metrics/evaluation.json

The script might require minimum recall, maximum false-positive rate, latency, fairness slices, and schema compatibility. Keep those thresholds reviewable in params or policy files.

Render the same visualization everywhere

List the available plot templates:

crab plots templates

Commit plot declarations and source data when they are small and useful for later comparison. Store bulky rendered dashboards as CI artifacts or Crab-managed outputs.

Make CI emit structured evidence

crab run --validate
crab run evaluate --cache-only --jsonl | tee evaluate.events.jsonl
crab params diff origin/main HEAD --json > params.diff.json
crab metrics diff origin/main HEAD --json > metrics.diff.json
crab plots diff origin/main HEAD --format html --output plot-diff.html

The cache-only execution proves the candidate result was published. The three diffs make it reviewable. Neither substitutes for application-specific quality checks.

Once the candidate passes review, capture its model bytes as an immutable workflow artifact.

Review ML parameters, metrics, and plots as one evidence chain

Declare stable evidence paths

Inspect the current run

Compare the candidate with the baseline

Treat quality gates as domain policy

Render the same visualization everywhere

Make CI emit structured evidence

Check the decision, not your memory.

Run an ML experiment sweep without dirtying your workspace

Version and retrieve an immutable ML artifact

Related guides

Review ML parameters, metrics, and plots as one evidence chain

Declare stable evidence paths

Inspect the current run

Compare the candidate with the baseline

Treat quality gates as domain policy

Render the same visualization everywhere

Make CI emit structured evidence

Check the decision, not your memory.

Run an ML experiment sweep without dirtying your workspace

Version and retrieve an immutable ML artifact

Related guides