Skip to main content

Diagnose and recover a red run

A red check is a claim about a run, not yet about your commit. Work the ladder top-down; each rung can end the investigation.

Order of evidence

  1. Fleet firstslipstream_platform_health {} then slipstream_notifications {"min_level":"warn"}. A degraded platform or a fenced executor release (dispatch_frozen) explains a red or a stall without touching the repo.
  2. Pick the runslipstream_status with the repo and the exact pushed 40-hex sha; take the newest non-superseded concluded red of the kind you care about by run_id.
  3. Diagnose itslipstream_diagnose {repo, run_id}: failed stage, bounded error tail, classification, retry chain, next steps.
  4. Only if the tail is not enoughslipstream_log_get {run_id, grep} for the concrete error, then page with offsets.
  5. Recoverslipstream_rerun with the exact failed run_id and matching kind (deploy reruns need confirm: true; a red artifact run needs kind: "artifact"); then slipstream_follow the original sha to a terminal phase.

Reading the diagnosis

FieldValuesRead it as
failure_classcode / recipe / infra / unclassifiedcode blames the change; recipe blames the pipeline definition — fix the recipe, not your code, because neither a rerun nor a code change can help; infra is a platform or resource fault the platform already retries; unclassified needs more evidence before either.
kill_reasonwall_cap / stage_timeout / nullWhich deadline killed the run: the kind's hard ceiling (wall_cap) or a stage's own timeout (stage_timeout). Either says the pipeline is too slow for its bound, not that the change is wrong. Null = not killed by a deadline.
error_classdeadline or emptydeadline: the platform force-concluded the run on its own lifecycle lease. Never consumer evidence. At most one automatic retry follows, and only if the retry gate allowed it — check the retry chain rather than assuming a successor exists.
failure_fingerprinthashSame fingerprint on main before your commit = pre-existing; first occurrence on your branch = look at the diff.
retry chainrun idsThe successor already exists — follow it instead of spawning another.

When a rerun is justified

Transient infra, resource pressure, or contention when diagnosis says so. Not: a deterministic code failure, a near-ceiling run (make it faster instead), a superseded artifact, or anything while the platform reports degraded: true.

Recovery proof

A rerun result alone proves nothing. Re-follow the ORIGINAL sha with slipstream_follow; only its terminal phase (ci_only, live, live_unchanged with a stated unchanged_because) is delivery evidence. See Ship and follow.