feat(rules): arm a ticking status watcher at launch and wait through it

Merged
BO
2d ago
+37-7

The periodic-updates modifier already tells an agent to post a status line every 5 minutes during a job over about 10 minutes. In one long timed run the agent armed a watcher that reported milestones only and waited on the job in a long blocking call. The user had to ask why it was taking so long. This closes the three places where the text allowed that.

What changes in rules/modifiers/periodic-updates.md (1473 to 1961 bytes; the modifier is opt-in, so the 30,200-byte fresh-install ceiling is unaffected):

  • The watcher is armed in the step that launches the job. It posts a line every interval even when nothing changed. A watcher that fires only on milestones does not satisfy the rule.
  • The agent waits on the job through the watcher, not in one long blocking call. Such a call shows the user nothing and a user message can interrupt it.
  • The launch reply gives the total ETA with its basis (or "unmeasured") and the interval. The ETA is given again whenever it changes, including when more rounds are queued.

I did not add a second threshold such as 30 minutes beside the existing 10, and I did not add a launch helper. Hosts differ in how they run a watcher, and the rule already covers arming one, so a helper would reuse nothing. The two sentences that without: patterns in the existing screening scenarios match are unchanged.

New scenario. evals/end-of-reply/screen/modifiers/periodic-launch.md gives a reply after a background launch of a 60 to 90 minute run whose runner prints a line only every 20 minutes. It passes when the reply names the 5-minute interval and either "unmeasured" or an estimate for this run (an estimate word such as ETA, expect or about, then 60 minutes or more or an hour figure) with a basis such as the last runs. It fails on a blocking wait of 10 minutes or more written as a sleep, a timeout, or a wait in seconds. A watcher's own timeout_ms is allowed.

Checks. npx --yes oxfmt@0.70.0 --check . and npx --yes oxlint@1.85.0 . exit 0, with the existing control-character warning in test/install-ps1.test.mjs. node --test test/*.test.mjs (179 pass, 0 fail, 7 not run) and npm run check:package pass, including the test that every screening scenario's without: matches the composed rules once. Canned replies grade as intended: a reply that arms a 5-minute watcher and gives an ETA passes, and a milestone-only watcher and a 900-second until loop each fail.

Not run: the before and after model eval. The Claude Code command line that the eval wrapper uses returned "weekly limit" (resets 2026-10-18 12:00 MDT) for both Opus and Sonnet. No model has replied to the new scenario under the old or new text. The scenario is therefore checked for structure and grading only, not for whether a model follows the new wording. Run node evals/end-of-reply/run.mjs --scenarios evals/end-of-reply/screen/modifiers --rules <composed with the modifiers config> --cmd evals/end-of-reply/models/claude.sh --runs 3 on this branch and on main to compare.

The change is under Unreleased in the changelog. Nothing is versioned, tagged or released.

Summary by cubic

The periodic-updates modifier now has the agent arm a ticking status watcher at launch, post a status line every interval (not only on milestones), and wait on the job through the watcher instead of one long blocking call. The launch reply states the total ETA with its basis (or "unmeasured") and the interval, and repeats it whenever the ETA changes.

Adds a periodic-launch screening scenario that grades the reply: the ETA must follow an estimate word within 40 characters and be a duration of 60 minutes or more with a basis, or "unmeasured"; waits or timeouts of 10 minutes or more fail when phrased in minutes, seconds, or hours—including with up to four filler words between the command and the duration—except a watcher's own timeout_ms. Sub-hour figures like "0.5 hours" do not count as 60 minutes or more.

Summary by CodeRabbit

  • Documentation
    • Clarified that launch replies should include the total ETA, its basis—or state that it is unmeasured—and the update interval. Repeat the ETA when it changes, including when more rounds are queued.
    • Clarified that status updates should be posted at every interval, even when nothing changes, and continue while the job runs. Arm the status watcher at launch and follow progress through it rather than using one long blocking wait.
  • Tests
    • Added a screening scenario checking launch-reply timing and ETA details, including safeguards against excessively long waits.