base autofix: regenerate the settings exports on the newest tip, so they land

Merged
TE
15d ago
+494-40

Summary

The base autofix never lands the settings exports. A run takes about 28 minutes and feat-cmux-next takes a push about once a minute, so by the time base-autofix-push.sh (#22004) runs, some push has nearly always moved a settings-export input since the run's SHA, and the push holds the exports. It does that on every run. The exports are stale on the tip right now: python3 scripts/ci/defaults_guard.py check-stamp lists 9 stale sources, #21897's sessions.promptSuggestions among them.

Now the exports are made again on the newest tip within the same run, and the push judges them from the commit they were made at:

  • Regenerate on the tip (Mac job). The action exports job's settings step runs the new scripts/cmux-next/base-autofix-settings-exports.sh. After each regeneration it fetches the newest tip. If the tip moved the settings inputs (autofix-areas.sh's settings area) since the commit it generated on, it checks that tip out and regenerates there, at most 3 times. It outputs settings_made_at, and hands on every export file whose made copy differs from the run's SHA, so the repair carries the whole export as made there. If the inputs still moved after the third try, it hands on no settings export, prints ::error and the job fails. A diff or area read that fails counts as moved. The step has timeout-minutes: 40 inside the job's 60, so a slow regeneration fails only that step (continue-on-error) and the action exports' upload still runs. Only the Mac job can regenerate and only the Linux job holds the write token, so the retry lives on the Mac side and the SHA crosses over as a job output.
  • Judge from where they were made (push). base-autofix-push.sh REMOTE BRANCH BASE_SHA [SETTINGS_MADE_AT] judges the settings exports by input moves and newer writes in SETTINGS_MADE_AT..tip instead of BASE_SHA..tip. When they land, they land whole: every listed export file whose made copy differs from the tip's (the list matches regenerate-settings-exports.sh, and a test checks that). A SETTINGS_MADE_AT that isn't a full SHA between BASE_SHA and the tip holds them and fails the run with ::error, even when the tip's stamp is current. The diff of SETTINGS_MADE_AT..tip and its input areas are read as separate steps, and the run fails closed if either fails. Everything else is still judged from BASE_SHA.
  • Superseded drop. If the settings exports would wait (and SETTINGS_MADE_AT is valid) but defaults_guard.py check-stamp --ref <tip> passes, someone else already regenerated them. They're dropped from the repair and don't fail the run.
  • Subject-only autofix match. newer_writes used git log --grep, which matches any line of a message. A lane commit whose body quoted chore(next): base autofix at ... (a squash merge, say) had its writes ignored, and the repair overwrote them. Now only the subject counts.
  • The Mac job's timeout goes from 40 to 60 minutes to fit up to three regenerations. Before each regeneration on a moved tip, it rebuilds stale web bundles (build-web-bundles.sh, which does nothing when they're current).

Invariant: an export generated from inputs that changed on the moved tip is never landed as-is. It is regenerated on the newest tip (and judged from there), or it is held. Case 19 below tests exactly this. Removing the SETTINGS_MADE_AT..tip move check makes it fail ("exports whose inputs moved after they were made passed the run").

Catalog narrowing: not implemented. The exports do read the string catalog. SettingsSchemaExport.json(catalog:) reads its key set, and every schema/MDM text comes from ModuleResourceBundle.text → String(localized:bundle:), the compiled table in the test process's preferred language. That language isn't pinned to English, so a non-English value change can change the exports on a runner whose language isn't English. "Only non-en values changed" doesn't prove the exports are unchanged, so the catalog stays an input. The bounded regeneration covers that case anyway.

Testing

Round 2 (review fixes)

Review found that, with pipefail, a failed git diff SETTINGS_MADE_AT tip (or areas script) made grep -x settings read nothing and exit 1, which read as "settings did not move", so stale exports could land. The tests came first (9ed66aee261e), then the fix (d443614f1927). Same commands as below.

  • New case 21b: a SETTINGS_MADE_AT that is a real commit off the branch (the repair itself), with the tip's stamp current, must fail with ::error and land nothing.
  • New case 21c: an areas script that fails on SETTINGS_MADE_AT..tip must fail the run and land nothing.
  • New Python test: the settings step has its own timeout-minutes, below the job's, and the upload comes after it.

Red on 9ed66aee261e (the bash test stops at its first failure, which is 21b; with fail made non-fatal, 21c also fails with "exports landed although what moved after they were made was unreadable"):

FAIL: exports made at a commit off the branch passed the run: The settings exports are already current on 20f7f82b6e85 (defaults_guard.py check-stamp); dropping them from this repair.
feat moved to 20f7f82b6e85 after 9ffe5309e502 (attempt 1 of 5); writing 1 repaired file(s) onto it.
Pushed 9f6eb55 to feat.
exit 1

FAIL: test_a_slow_settings_regeneration_never_costs_the_action_exports ... AssertionError: 'timeout-minutes' not found in {'name': 'Regenerate the settings exports', ...}
Ran 20 tests in 16.694s
FAILED (failures=1)

Green on d443614f1927:

PASS: base autofix lands its repair on a moved feat-cmux-next; files whose inputs moved or that a newer commit wrote wait; the settings exports are regenerated on the newest tip, or wait
exit 0
Ran 20 tests in 16.713s
OK

Round 1

Tests were committed first (97df8abda643), then the fix (05d332f45701). Same commands for both:

bash scripts/cmux-next/tests/base-autofix-push.test.sh
python3 tests/test_cmux_next_base_autofix_finishes.py

New bash cases: 16 (subject-only match), 17 and 18 (a) exports made at a moved tip land, including a file the tip had stale and the regeneration left as at BASE_SHA, 19 (b) the invariant, 20 (b) a made-at commit outside BASE..tip holds, 21 (d) superseded drop passes the run, 22–24 regeneration at the run's SHA / once more on a moved tip / not for an unrelated move, 25 (c) three tries with inputs always moving: holds, ::error, no export handed on. The fake defaults_guard.py in the test now answers check-stamp --ref. New Python cases cover the workflow shape (settings_made_at output, the push passes it) and the export list matching regenerate-settings-exports.sh.

Red on 97df8abda643 (the bash test stops at its first failure):

FAIL: a repair over a lane commit quoting an autofix subject was pushed: feat moved to 79f6cee3e1e5 after 9fdbf729fb56 (attempt 1 of 5); writing 1 repaired file(s) onto it.
Pushed 7eb547b to feat.
exit 1

The same file with fail made non-fatal, to show each new case on the base scripts: 16, 17, 18, 20, 21 and 22–25 fail. 17 and 18 fail with "changed the inputs or the files of everything this repair writes", 18 also with "docs/mdm as made at the tip did not replace the tip's stale copy: mdm stale", 21 with "a repair the tip already superseded failed the run", and 22–25 because base-autofix-settings-exports.sh doesn't exist. 19 already passes on the base, which held every export whose inputs moved since BASE_SHA. It's there to keep the invariant through this change.

ERROR: test_the_push_judges_the_settings_exports_from_where_they_were_made ... KeyError: 'SETTINGS_MADE_AT'
ERROR: test_the_push_script_knows_every_settings_export ... IndexError: list index out of range
FAIL: test_the_mac_job_regenerates_the_settings_exports_on_the_newest_tip ... 'scripts/cmux-next/base-autofix-settings-exports.sh' not found in ...
Ran 19 tests in 19.185s
FAILED (failures=1, errors=2)

Green on 05d332f45701:

PASS: base autofix lands its repair on a moved feat-cmux-next; files whose inputs moved or that a newer commit wrote wait; the settings exports are regenerated on the newest tip, or wait
exit 0
Ran 19 tests in 17.313s
OK

Observed / Not run

  • Ran locally on huge-blue, on both rounds: both commands above (red, then green), python3 -m unittest tests/test_cmux_next_checks_workflow.py (OK), python3 scripts/cmux-next/prepush-gate.py (no new violations), and actionlint on cmux-next-base-autofix.yml (clean; no local shellcheck, so its run-block shellcheck pass ran only in CI). I also checked by mutation that case 19 fails when the SETTINGS_MADE_AT..tip move check is removed.
  • Not tested: the Mac-side "a failed diff counts as moved" path in base-autofix-settings-exports.sh, and the step timeout firing. Both were only read.
  • If the step timeout kills the settings step while it copies its 7 files out, a partial set can reach the artifact with no settings_made_at. The push then judges them from BASE_SHA, where their inputs moved, so they're held, but the copy isn't atomic.
  • Not run: the real Mac job. The regeneration in the tests is a stand-in for regenerate-settings-exports.sh. Checking out a newer tip mid-job, the incremental swift build there, and the web-bundle rebuild have only been read, not run. The first real base run after merge is the first live check.

Known gaps (not fixed here)

  • CmuxNextDesign feeds the schema (ModuleResourceBundle lives there) but isn't a settings-area input and isn't in the stamp.
  • The export tests' language isn't pinned to English, so the exported texts depend on the runner's preferred language.
  • The defaults snapshot is still held whenever the action-export inputs moved since BASE_SHA, whether or not this run regenerated the action exports.

Changelog

none

Agent session: huge-blue

🤖 Generated with Claude Code