The gate pools every dispatch of a gate set on the commit, under the set's own bars (#613, record 0169)

Merged
DO
12h ago
+990-68

Issue #613, "The gate pools k dispatches on one commit and reads all of them", PR 1 of 2. It follows the Builder's ruling of 2026-11-08 (defaults 1 to 7). The preflight is PR 2.

Dormant until a gate set is named. Until config/harness.json names a gate set, the Release's gate reads pr-review as before, by its newest successful dispatch. The record states the remaining gap as a residual: a development set, pr-review (which gates v0.1.x until v2 is seated), stays the best of several runs, because re-dispatching until one succeeds is still possible there.

What changes

  • Which set the gate reads.
    • adapters/candidate.py first reads config/harness.json at the commit, through the transport.
    • If it names gate_set, by one path part, the gate reads evals/<gate_set>/cases.json, which must be a gate set.
    • Otherwise it reads pr-review, which must not be a gate set.
    • Anything else gives eval_set_unreadable.
  • Every dispatch counts (pooled_gate).
    • It reads every completed dispatch of .github/workflows/eval.yml on the commit, oldest first, whatever it concluded.
    • One still running, or a run of another commit, workflow or event in the answer, gives eval_not_run.
    • Any of these, for a dispatch of any set, is eval_dispatch_faulted and closes the gate:
      • a conclusion other than success or failure;
      • a dispatch run again (run_attempt not 1);
      • a report not read whole, or one that names its set by more than its count and sha256.
    • A dispatch whose report names another set is not counted.
  • The pooled verdict (harness.eval.pooled_verdict).
    • It needs at least POOL_LEAST = 3 reports, all of them pooled; fewer gives eval_pool_short.
    • Each report is held whole by aggregate_verdict:
      • an Engine fault, a run fault or an unreadable report is eval_dispatch_faulted;
      • case_failed is pooled;
      • any other code closes the gate by its own code.
    • Each kind's cases and passes are summed over the reports and held to the set's bar.
    • The blocking word is true only if it is true in every report, and it closes nothing, as decision 0150 has it.
  • A gate set's own bars.
    • eval-set/1 gains bars, kind_least and record. A gate set must give all three; no other set may give any.
    • The job writes, and the gate holds, a gate set's report to its own bars and least of a kind.
    • Neither may be below BAR_OF_A_KIND / KIND_LEAST. A manifest under evals/ needs no person's approval, so it may raise the gate but not lower it.
    • tools/check_examples.py holds a gate set's manifest in LIVE to its record, which must contain bars_said's sentence word for word. It holds an eval-set example to held(). Nothing reads a record at run time.
  • Record 0169. For gate sets only, it amends decision 0150's rows that fix the bars in code or let the gate take the newest successful dispatch. It writes down four residuals:
    • a development set stays the best of several runs;
    • another set's faulted dispatch closes the pool;
    • a deleted run leaves no trace;
    • who may change a gate set: evals/ is not a protected path. Protecting it is a change to config/hooks.json, yours to decide.
  • Rules are in harness/eval/AGENTS.md. The cut skill says to dispatch a named gate set three times on the commit.

Checks

CheckResult
SIM_SEEDS=40 make verifyOK, 1709 tests (at cd431f04, main 46fea214 merged, record 0169)
Filtered mutants: the 47 new ones, the 13 brought up to date, and every one on adapters/candidate.py, tools/check_examples.py, the eval-set and harness schemas and the gate's shared tests165 of 165 killed (two survivors in the first run, now killed by added test cases)
gitleaksNo leaks
Argon (cc2ac91f), 17 parts1 major, 2 minor. Not changed: "_bars_stated runs only on LIVE", because check_examples refuses any file under evals/ that is not in LIVE. Already fixed in 9d0bfcfc: a set with extra keys. Fixed in 6d53380a: one mutant dropped two checks, now two mutants, each named for what it drops
Claude review (cc2ac91f)1 high, fixed: a re-run hid a faulted attempt. 4 low, fixed: deleted runs are named as a residual, the faulted-dispatch row is widened, the example is held by held(), and zero bars are refused through the floor
Claude security (cc2ac91f)1 medium, fixed: the floor on bars and least of a kind, and the record no longer calls evals/ protected. 2 low, fixed: re-runs, and a malformed set
Codex (502c73b6)1 P2, fixed in e8f11b59: least_passed read only the ends of the split, and rounding can make a split between them need fewer passes
MuseNot run (quota); owed after 2026-11-09
AstraNot run (quota); owed after 2026-11-09

šŸ¤– Generated with Claude Code