#28163 ci(owned-pool rescue): snapshot reads use the re-minted token, so the failover keeps working after an hour

Leo, 2026-10-28 (rulebook, CI and GitHub): "Failover is stall-aware: a job queued more than about 3 minutes on a label that has started nothing is cancelled and re-dispatched onto a live member of the set."

What was wrong

The rescue sweeper re-mints its cmux-bot token every 45 minutes: _fresh_token() swaps the new token into the header sets in place. pool_snapshot() did not use those headers. It kept sending the token the client started with, and that token expires after an hour. From then on, every janitor snapshot read was refused with 401. With no snapshot:

• unstarted_failover() keeps every open-head run where it is ("could not read the pool snapshot ... the open head keeps the run");
• hold_for_destination() waits the owned floor.

Sweeper run 41006853342 started at 05:10Z on 2026-10-30. From 06:12Z it logged 22 could not read the pool snapshot (HTTPError: HTTP Error 401: Unauthorized) lines, and it made no open-head failover in its remaining four hours. Each sweeper runs five hours, so for about four of every five hours the stall-aware failover was off.

What changed (scripts/ci/owned_pool_rescue.py)

• GitHub.snapshot_token(): the snapshot reads use the client's current read token, which is the re-minted one. If the read token belongs to another App, which this client never re-mints, the minted App's token is used instead once one exists.
• pool_snapshot(): one read now serves every watch in the process for 60 s (SNAPSHOT_CACHE_SECONDS). The janitor writes a snapshot every 10 minutes, and each read costs two requests, so a closer read only finds the same snapshot again. Each caller still checks the snapshot's age against its own now.

Not in this PR

This fix alone does not move runs 40993742146 (#27797) or 41021261503 (#27756). Both have a red checks, so assess() leaves their queued fast tier alone (leave_decided), and the failover is never consulted. Their finished Linux jobs also fail the rule that nothing has started. #28148 covers the finished-jobs part. The red-sibling part and a stale-running label check have been passed to #28148's owner.

Tests

python3 tests/test_ci_owned_pool_rescue.py: 372 tests OK. The new SnapshotToken tests fail on base (1 failure, 2 errors) and pass here:

• snapshot reads follow the minted token after TOKEN_REFRESH_SECONDS;
• one read serves the process for a minute;
• another App's read token gives way to the minted one.

No new timer, observer or loop. The cache is one in-memory entry that expires after 60 s.

No capture: CI script change with no UI.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Fixes the owned-pool rescue sweeper so its stall-aware failover keeps working after the first hour: snapshot reads now use the re-minted token instead of the token the client started with, which expired after an hour and turned every read into a 401.

• Snapshot reads now use the client's current read token; when the read token belongs to another App, the minted App's token is used once one exists.
• One snapshot read now serves every watch of the process for 60 seconds (the janitor writes one every 10 minutes), while each caller still checks the snapshot's age against its own now.
• Adds tests covering the re-minted token, the 60-second cache, and another App's read token giving way to the minted one.

Summary by CodeRabbit

• Improvements
  • CI pool snapshots are reused for up to 60 seconds, reducing repeated reads.
  • Snapshot requests use refreshed credentials when needed, while existing snapshot validity checks remain in place.