base autofix: remake held repairs on the moving tip instead of starving

Merged
TE
15d ago
+612-60

The base autofix on feat-cmux-next has landed nothing since 08:58 UTC (8ff63657c96), so the #21897 settings exports and the CI target graph stay stale. This makes the run land what it can on a moving branch.

Why it starved (runs 39565868911 to 39580605258 on 2026-10-21):

  • The graph was held on every run. Its input area is any Packages/ change, which nearly every push makes. The push held it, and the next run's push found the tip moved again.
  • The Mac exports' autofix could be replaced. Runs whose own Mac job a newer push replaced still run autofix, about 1.5 minutes every minute or two. When a Mac job finished, its autofix queued behind one of those, and the next push replaced it while it was pending. That threw away half an hour of Mac work.
  • Settings exports were held too broadly. They were held for any settings-area move, even when their real inputs were unchanged.

Already on base, not changed here: no in-progress cancel (#22004), push retries on a moved tip (#22004), and settings regenerated on the newest tip up to 3 times in the Mac job (#23249).

Fix

  • New scripts/cmux-next/base-autofix-land.sh. It pushes with base-autofix-push.sh. When the push holds a file this job can make itself because the tip moved its inputs, it checks out the new tip, makes the repair again there, commits it as chore(next): base autofix at <tip> and pushes. This covers the CI target graph, the SDK layers, the namespace ratchet, the cloud clients and cargo fmt. It runs at most three rounds.

    • A held Mac export, or a file built from one, is never made there. The run fails, so the next run's diff base still includes it and its Mac job makes it.
    • The repair runs without the push token.
  • New scripts/cmux-next/base-autofix-repair.sh. It holds the Repair step's logic, unchanged for the first round, plus the dump-package container call. A round on a moved tip applies no Mac export or webviews patch made at the older commit, and leaves the shortcuts, app-host files and defaults snapshot to the next run.

  • Concurrency. The autofix job of a run that carries Mac exports gets its own group, so it is never the pending run a newer push replaces. Its push races the others safely, because the push script reads the tip and retries. Groups are now per branch.

  • Settings judged by their stamp. defaults_guard.py source-stamp --ref R prints the stamp R's sources give. The push, and the Mac job's regenerate-on-tip loop, treat the settings inputs as unchanged when the tip gives the exports' own stamp and regenerate-settings-exports.sh did not change. Anything else still holds them, and every read that fails counts as moved.

  • Base red from #23249, fixed here: base-autofix-settings-exports.sh fetched with --depth=1 unconditionally, which tests/test_ci_no_shallow_dev_fetch.py flags in every PR's cmux-next CI guard group (job 119156226796). It now passes --depth only in a checkout that is already shallow.

Tests. e2cb7884933 is the red commit:

  • scripts/cmux-next/tests/base-autofix-land.test.sh (new, wired into cmux-next.yml) simulates a moving tip with a lane clone that pushes during the run. Cases:
    • unmoved branch;
    • inputs moved, so the graph is made again on the tip and lands;
    • inputs moving every round, so it stops after 3, lands nothing stale and fails;
    • tip already right;
    • held settings plus graph, so the graph lands and the settings wait;
    • only the settings held, so no extra round;
    • no token in the repair's environment;
    • the wrapper's first-round and later-round arguments.
  • base-autofix-push.test.sh 21dโ€“21f and 24b cover the stamp: a settings-area move that changes no stamp input lands the exports and makes no extra regeneration. A change to the regeneration script or a source still holds them.
  • tests/test_cmux_next_base_autofix_finishes.py covers the concurrency group, and tests/test_defaults_guard.py covers source-stamp.

Review fixes, red test 88e4e20564e first (base-autofix-land.test.sh 7b, 7c): a push that never got through (the branch moved under every attempt) no longer hands on a held list, so the job can't remake only the graph and pass without the settings exports; and the Chief inspector page, which only the webviews patch makes, waits instead of being counted as remade.

Red on e2cb7884933 (base code):

bash scripts/cmux-next/tests/base-autofix-land.test.sh   # FAIL: base-autofix-land.sh: No such file or directory
bash scripts/cmux-next/tests/base-autofix-push.test.sh   # FAIL: exports whose stamp inputs did not move were held
python3 tests/test_cmux_next_base_autofix_finishes.py    # FAILED (failures=2)
python3 tests/test_defaults_guard.py                     # FAIL: test_source_stamp_prints_the_stamp_a_ref_s_sources_give

Green on the head: all four pass, and actionlint .github/workflows/cmux-next-base-autofix.yml is clean.

Observed / Not run

  • Observed: the commands above, run red and then green, and an independent review subagent (SHIP at 891a9c4744d after a BLOCK at aa0c3fde1f5 for the two holes fixed above). The run history and logs cited above come from the Actions API: the held graph in 119150798106 and 39580605258, and Mac jobs replaced while pending.
  • Not run: a real feat-cmux-next run of the new workflow. The first one is the run after this merges, and I'll report the first base run that lands the exports. shellcheck is not installed on this host. tests/test_cmux_next_checks_workflow.py errors in this sparse checkout only because ci-target-graph.json is not checked out here; this PR doesn't touch it.

Changelog

none

Agent session: huge-blue

๐Ÿค– Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Stops the base autofix on feat-cmux-next from starving on a moving branch: a repair whose inputs a newer push moved is now remade on the new tip and landed, instead of waiting for the next run and being held again. Settings exports are judged by their stamp rather than any settings-area move, so harmless moves no longer hold them, and the autofix that carries Mac exports can no longer be replaced while pending.

  • New base-autofix-land.sh pushes with base-autofix-push.sh, and when the push holds a file the job can make itself (CI target graph, SDK layers, ratchet, cloud clients, cargo fmt), checks out the tip, remakes it there, and pushes โ€” at most 3 rounds. Held Mac exports still wait for the next run, and the run fails so the next run covers them.
  • New base-autofix-repair.sh holds the Repair step's logic; rounds on a moved tip apply no Mac export or webviews patch made at the older commit.
  • A run that carries Mac exports gets its own concurrency group, so a newer push cannot replace its autofix while it queues behind another run's; groups are now per branch.
  • defaults_guard.py source-stamp prints the stamp a ref's settings sources give; the push and the Mac job's regenerate-on-tip loop treat settings inputs as unchanged when the tip gives the exports' stamp and the regeneration script did not change.
  • Fixes a pre-existing red (#23249): the settings exports fetch passes --depth only when the checkout is already shallow, now guarded so an empty depth list also works on bash 3.2.