ci: reconcile fleet Linux pools from live probe receipts

TE
5d ago
+865-3

A stalled label remains dead when probe health ages or an API read fails. The reconciler reads current repository state before writing, accepts a fleet start or recorded three-minute no-start, and keeps legacy secret-bearing jobs on a separate ephemeral pool. When only the fleet is available, it publishes that safe route and explicitly reports legacy capacity unavailable.

Leo: “FOR OUR NIGHTLIES ARE THEY END TO END ON OUR FLEET? WE CAN'T BE RELYING ON BLACKSMITH THAT CUTS OUT.”

The entry point runs from the trusted tooling checkout beside main's existing policy. The outage switch still evaluates Blacksmith alone; a fleet probe cannot count as Blacksmith recovery. The writer preserves promotion and unknown metadata and records bounded route changes for the existing queued-job rescuer. It changes no runner trust, secrets, workflow permissions or started-run cancellation policy. Main workflow/schema wiring lands separately before activation.

Validation: 20 targeted Python integration tests and 8 cloud-watch rescue tests pass against the current main policy and the feat runtime, covering actual-start/no-start evidence, stale health, unknown reads, partial capacity, mirror recovery, concurrent state changes and bounded move receipts. Independent Astra review: SHIP at efe47cc7722fa1c75c084b7e732be540394031e9. Existing compile caches are unaffected; this performs no compile or build.

PerformanceBeforeAfterEvidence
Blacksmith minutes/dayNot measuredNot measuredRouting policy only; no unsupported usage estimate
Minimum recorded fleet no-start before overflowNo fleet probe3 minutesTargeted probe-receipt tests

Agent session: agent session
🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Reconciles fleet Linux runner pools from live probe receipts so a stalled label no longer stays dead when probe health ages or an API read fails.

  • The reconciler reads current repository state before any write and accepts only a fleet start or a recorded three-minute no-start.
  • Legacy secret-bearing jobs stay on a separate ephemeral pool; when only the fleet is available, it publishes that route and explicitly reports legacy capacity unavailable.
  • Adds a bounded rescue for the cloud-overflow probe watcher that only touches an unstarted watcher job.