fix(flows): check the repository the way its CI does, and never throw the work away

Merged
KJ
2mo ago
+967-135

Why

The generated software-factory flow ran one hard-coded recipe — npm ci && npm test behind a lockfile detector — and treated its exit code as the whole verdict. f.run has no retry policy, so a failing test step ended the run as retries_exhausted with the agents' finished work unpushed and no test output anywhere a user could see. Two production runs today:

  • AgentWorkforce/cloud acbe30c1 — the test step failed on AssertionError: @cloud/core dist missing — run npm run -w @cloud/core build before this test. Cloud's CI builds core first (ci.yml:333); the recipe never did, so every run against cloud fails at this step whatever the ticket. Building core first in the same sandbox made the whole suite pass, agents' commit included.
  • AgentWorkforce/relay 139d1a46 — 18 tests failed inside the flow. In a clean shell of the same sandbox the agents' branch passed all 3,227 (base: 3,173, 0 failed). One cause is proven: Cloud's executor passes its own git hardening (protocol.allow=never, GIT_CONFIG_GLOBAL=/dev/null) to every step, which reproduces fatal: transport 'file' not allowed in sandbox-repo.test.ts. (The executor side is being fixed separately in cloud.)

Also: Codex committed summary.md into the branch in both runs, as in AgentWorkforce/flows#454.

What changes in the generated flow

Finding the check command, for any language. .relayflow/check.sh comes from, in order:

  1. the author's checkCommand constant in the flow;
  2. a .relayflow/check.sh the repository commits;
  3. a discovery agent that reads CI config, Makefile/justfile/Taskfile and AGENTS.md/CLAUDE.md/README (runs before the implementer, so the base commit is checked the same way);
  4. an ecosystem default written into the file: make/just test target, Node (unchanged pnpm/Yarn/Bun/npm logic), Cargo, Go, Python (uv/poetry/pip), Ruby, Maven, Gradle, .NET, Mix;
  5. otherwise none: nothing runs, and the PR says so.

Running it resiliently. Every check step prints one token (pass/fail/timeout/none) and exits 0. Full output goes to .relayflow/*.log, and its tail goes to stderr for the journal. The script runs without Cloud's GIT_CONFIG_* hardening, and stops itself at 14 minutes, before the 15-minute f.run lease would kill the run.

Repair, then compare. On failure, a repair agent (up to 2 attempts) reads the log and fixes missing setup by editing check.sh (as CI does) or fixes bugs in the change. It never skips or weakens tests. Whatever still fails is run on the base commit in a throwaway worktree:

basebranchoutcome
—pass / noneready PR, reviews, needs_human
failfaildraft PR with both outputs, reviews still run, needs_human
passfaildraft PR with the output, step_failed (the flow's verdict on its own work, like a failed review)
unknownfaildraft PR saying it could not compare, reviews, needs_human

The branch is always pushed; work is never thrown away. The PR body is summary.md plus a ## Checks section containing the verdict, the script that ran, the output tails and any repair notes. In the traditional preset, the fixer's revision is checked the same way. If it breaks checks that passed before it, it is still pushed, the PR goes back to draft with the report, and the run ends step_failed.

Working files. summary.md, the plans and reviews, .relayflow/ and the local kit files are excluded via .git/info/exclude, using root-anchored patterns so a docs/summary.md is still committed. If an agent committed them anyway, they're removed before every push by a plumbing commit built from a private index, so staged but uncommitted work is never swept in. Only files absent from the base commit are removed, so a repository's own summary.md survives.

Local kit: START-HERE, the run requirements and the preflight exemptions are updated to match.

Verification

  • npx vitest run (web): 19 files, 182 tests pass. The new real-shell tests use real git repositories for resolve (every ecosystem, none), run (pass/fail/none/timeout, Cloud's git config not reaching tests with a positive control), base comparison (regression vs pre-existing, worktree cleanup, unknown base), exclude, drop (keeps a pre-existing summary.md, never sweeps staged work, identity fallback), and the report. The generated flow is also executed with a mocked context for each branch.
  • tsc --noEmit (web): clean.
  • flows check: all 3 presets × cloud/local pass on 2.0.15 and 2.0.17. flows check also passes a flow referencing an undefined variable, so the generated flows were additionally checked with strict tsc against the 2.0.17 surface types: all 6 clean, and a deliberately broken control fails.

Follow-ups (not in this PR)

  • An optional "test command" field in onboarding. For now the author can set checkCommand in the flow or commit .relayflow/check.sh.
  • Cloud executor: run the flows CLI in a clean environment (separate cloud PR).
  • A saved check script per deployment, so later runs skip discovery.

🤖 Generated with Claude Code

[!NOTE]
High Risk
Changes core generated flow behavior (git, push/PR, check execution, and completion reasons) across all presets; mistakes could publish wrong PR state or mis-classify check failures.

Overview
Replaces the software-factory flow’s single hard-coded npm test step with repository-aware checks via .relayflow/check.sh, plus agents and shell helpers that resolve, run, repair, and compare results against the base commit.

Check pipeline. Before implementation, the flow settles a check script from checkCommand, a committed .relayflow/check.sh, a check-discovery agent (CI/Makefile/README), or ecosystem defaults (make, Node, Rust, Go, Python, etc.). Checks return one token (pass/fail/timeout/none) and always exit 0; logs live under .relayflow/. On failure, check-repair runs up to twice; if checks still fail, the same recipe runs on the starting commit to tell regressions from pre-existing breakage. Failed runs push draft PRs with a ## Checks report (not retries_exhausted with unpushed work). Regressions end step_failed; shared base failures still open drafts and can continue to review.

Git hygiene. Working artifacts (summary.md, plans, .relayflow/, kit files) are excluded via .git/info/exclude and stripped from the branch before push if agents committed them. PR bodies use .relayflow/pr-body.md. Fixer revisions get the same check/repair path; breaking previously passing checks drafts the PR again.

Product tweaks. New agent roles and onboarding labels; local/cloud wall-clock budget 1h → 2h; onboarding and START-HERE copy describe CI-aligned checks and checkCommand. Extensive vitest coverage for the new shell commands and generated flow behavior.

Reviewed by Cursor Bugbot for commit 38312f6cd8571bc1f1a204ba32b1ec3e670fa061. Bugbot is set up for automated code reviews on this repo. Configure here.