.env key: resume a run and
restore a stage checkpoint, both invoked from the browser. They are off
by default and, like every other mutation path in the hub, they never trust
the client — the server re-derives eligibility from ground truth before doing
any work, and every request is recorded to an append-only ledger that is both
the idempotency store and the audit trail. The full threat model lives in
docs/security/cockpit-controls-threat-model.md; this page covers what the
feature does and how to turn it on.
How it works
Off by default, enabled two ways
cockpitControlsEnabled(policy, env) in
src/core/harness/cockpit-actions.js checks, in order:
- the env flag
RSTACK_COCKPIT_CONTROLS(1/true/yes/on) — a global kill switch, and policy.cockpit_controls.enabled === truein.rstack/policy.json— a per-project opt-in.
cockpit projection
(src/observability/dashboard/state/cockpit.js) returns
{ enabled: false, runs: [] } — there is nothing for the client to render —
and POST /api/action returns 403 before touching anything.
Two actions, two risk levels
COCKPIT_ACTION_TYPES defines exactly two actions:
The projection (
buildCockpitProjection) builds one resume-run action per
run and one restore-checkpoint action per checkpointed stage
(rollup.checkpoints.stages), each carrying:
enabled/disabledReason— fromevaluateResumeEligibility(rollup)orevaluateCheckpointEligibility(stage, { stale }), derived from the same compact pipeline rollup the CLI’spipeline statusreads, so the UI can’t offer something execution wouldn’t do. Resume is only offered whennext_action.kindisactive,pending,retry, orfailed; a pending approval, an exhausted retry budget, or a stale snapshot disables it with an explicit reason. Checkpoint restore is only offered whenstage.restorable === true— a corrupt manifest, hash mismatch, missing/extra files, or a pre-manifest legacy checkpoint is surfaced as disabled with the reason, never silently hidden.idempotencyRequired: trueon both actions.- a
confirmblock (title/consequence/target) the UI renders in a confirmation modal before submitting.
The route re-verifies from ground truth
handleCockpitAction in src/observability/dashboard/server.js (behind
POST /api/action) never trusts the projection it just sent the client. Per
request it:
- Runs the shared guarded-POST chain (token auth, CSRF-origin check, rate
limit, body-size cap — the same chain
/api/env-writeuses). - Validates
actionagainstisKnownCockpitAction, requires an operator identity (resolvedBy), and validatesidempotencyKeyagainstisValidIdempotencyKey(8–128 chars of[A-Za-z0-9._:-], no..). - Resolves the target run across known project roots and re-checks
cockpitControlsEnabledagainst that root’s own policy — a client can’t ride one project’s opt-in into another’s runs. - For
restore-checkpoint, requiresstageIdand checks it againstisCanonicalStageId— the client-sent stage id is never trusted as-is. - Claims the idempotency key (see below), then executes:
executeResumeRuncalls the real planner (runPipeline, capped atRESUME_MAX_STEPS) and returns409 not_eligibleif nothing advanceable was found (complete,no_actionable_work,pending_approval,blocked_retry_policy,ask_user,dry_run).executeRestoreCheckpointdeep-verifies the checkpoint (verifyStageCheckpoint(runDir, stageId, { deep: true })) before touching any approval, refusing with409 not_eligibleif it isn’t restorable. If restorable, it looks for a consumed one-shot approval oncheckpointRestoreArtifact(runId, stageId); if none exists it enqueues a pendingcheckpoint_restoreapproval and returns409 approval_required— the operator resolves it on the Approvals page, then resubmits the same request. Only once approved doesrollbackToCheckpointrun.
- Writes a
cockpit_resume_runorcockpit_checkpoint_restoredrun event on success and broadcasts a fresh snapshot to connected clients — no optimistic success: the response body is the real outcome, and the UI reconciles from the next real state.
The ledger: idempotency store and audit trail
Every action carries a client-supplied idempotency key.claimIdempotencyKey
appends a started line to the append-only
.rstack/cockpit-actions.jsonl ledger under the harness file lock
(withFileLock), then completeLedgerEntry appends the terminal
completed/failed line. summarizeLedgerForKey reads the ledger in
append order, not by timestamp — the last entry for a key is authoritative,
which avoids a same-millisecond tie misreading a finished action as still
in-flight. Re-submitting a completed key replays the stored result
without re-executing; re-submitting a key that’s still started returns
409 in_progress. Because the ledger is never rewritten or deleted, it
doubles as the immutable audit trail for every cockpit action ever taken,
including denials (auth failures, unknown actions, disabled-feature attempts)
recorded with phase: 'denied'.
Try it
Enable cockpit controls for one project via.rstack/policy.json:
approval_required — approve the checkpoint-restore:<runId>:<stageId>
request on the Approvals page, then retry the same action.
Related
- Business Hub overview & navigation — where cockpit controls appear in the Run Workspace
- Approvals & governance — the approval queue
restore-checkpointrequests land on - Governance model — the broader approval/guardrail model cockpit actions plug into
- Business Flex profiles —
.rstack/policy.jsonconventions
