The eval scratch is read part by part, the plan makes the heads folder new, and the canary's head is held apart (#626)

Merged
DO
1d ago
+95-7

Closes #626.

What changes

  • scratch checks containment part by part. scratch in harness/eval/__init__.py refuses a scratch that is the workspace, is inside it, or holds it, using os.path.commonpath on their real paths, as heads_place already does. It no longer compares prefixes with a separator. A heldout linked to the root was accepted before, because the root's own path ends in its separator.
  • The plan makes the heads folder new. plan creates heads beside the scratch with mode 0o700, as it creates the scratch. A heads folder left by an earlier job stops the plan with scratch_unreadable, as a leftover scratch does. Hosted runners are discarded after each job, so this guards only a reused runner.
  • A test holds the canary's head apart from its case. test_the_canarys_head_is_apart_from_its_case asserts that the head the CLI starts in and the case's folder share no folder below the system's temporary directory.
  • #623's test says why it fails. test_each_head_is_unpacked_apart_from_the_cases asserts that the CLI was asked, in a head, before reading where it was. Two of #623's mutants now fail on that assertion instead of a KeyError.

Mutants

MutantState
the scratch is held to the workspace by its separator (new)killed
the plan makes no heads folder (new)killed
a heads folder an earlier job left is taken (new)killed
the heads folder is made open to all (new)killed
the canary unpacks its head beside its case (new)killed
the scratch may be in the workspace; a workspace inside the scratch is allowed; a workspace that is the scratch is allowed (repointed to the new line)killed
the fetch unpacks each head in the scratch; the run step looks for each head in the scratch (#623's)killed, now by an assertion

Every mutant in tools/mutants.py still matches its file exactly once and parses.

Record NNNN, number to be given. No change to eval.yml. No protected path is touched.

Verify

Pending.

Reviews

ReadState
Claude + securitypending
Argonpending
Musenot run (quota); owed after 2026-11-09
Astranot run (quota)
Codexautomatic on push

šŸ¤– Generated with Claude Code