fix(evals): keep a reply's opening prose when only calls follow the leading reads

Merged
BO
1d ago
+103-4

With allowRead, afterLeadingReads in evals/end-of-reply/grade.mjs grades only the text after the leading read-only calls and treats text before the first call as narration. A reply that opens with a status line, then makes a read-only call (for example date +%Z), then an acting call (start a watcher), and has no prose after those calls, loses its status line. A required pattern on that line then fails a correct reply. I saw this on a Sonnet run of the periodic-updates launch scenario: the reply said "ETA is about 68-81 minutes ... based on the last two full runs" and was graded as if it said nothing.

The fix: when the only prose in the reply comes before the first call and nothing but tool calls follow the leading reads, afterLeadingReads returns the whole reply. Every other shape is unchanged: narration before reads is still dropped when prose follows them, a reads-only reply is still graded whole, and a reply with no prose before the reads still grades the acting call alone.

Changes:

  • evals/end-of-reply/grade.mjs: 5 lines in afterLeadingReads plus its header comment.
  • test/grade.test.mjs: two new afterLeadingReads cases (status line kept; no status line, acting call alone, as before) and a grade() check that a required pattern matches the status line and a non-matching one still fails.
  • CHANGELOG.md: an Unreleased entry marked repository only.

Proof I ran:

  • Before the fix, the new test fails (afterLeadingReads returns only the watcher call). After it, test/grade.test.mjs has 23 pass, 0 fail.
  • The saved Sonnet reply that exposed this, graded against the periodic launch scenario's patterns: fail with the old grader, pass with this one.
  • node --test test/*.test.mjs: 179 pass, 0 fail, 7 skipped (exit 0). bun test test/grade.test.mjs: 23 pass.
  • npx --yes oxfmt@0.70.0 --check . exit 0; npx --yes oxlint@1.85.0 . exit 0 (one existing warning in test/install-ps1.test.mjs).
  • Local hr-code-review of the final diff found no defects.

Not done: I did not rerun the model evals with this grader; no scenario file changes here.

  • The checks in the README's Development section pass (Node tests, Bun test of the changed file, pinned oxfmt and oxlint; check:package and bun test skills/ not run, no packaged or skill file changed).
  • A change users will notice has a CHANGELOG.md entry under Unreleased.
  • Nothing names a private project or repository, a local path, a home directory or a credential.

Summary by cubic

Fixes the end-of-reply eval grader so a reply's opening prose is kept when it is the only prose and only tool calls follow the leading reads. Previously a reply with a status line, a read-only call, then an acting call, and no prose after them, lost its status line and could fail a required pattern even when correct. Also lets claude.sh keep named CLAUDE*/ANTHROPIC* environment variables via EOR_KEEP_ENV (space-separated names) instead of unsetting them all, so a run can use a login that lives in the environment; the default behavior is unchanged.