[spike] code review bot (do not merge)
[spike] Do not merge or enqueue. This is an overnight spike for Dean's "bots" roadmap item. The proposal is docs/proposals/code-review-bot.md, and the decisions for Dean are in its §8.
The idea
A code review bot is an automation preset, not a new subsystem.
- Trigger: a pull request is opened, marked ready, or pushed to in the chosen repositories.
- Each run: an ordinary coding agent with the repository checked out, and
ghreaching GitHub asluke[bot]. - Output: the agent posts one GitHub review (
COMMENT: a one-line body plus inline comments). - Noise control lives in the prompt alone:
- a concrete bar for findings, with a failure scenario required for each;
- at most 5 findings, P0/P1;
- the rules files read from the base branch;
- its own earlier threads found by a hidden marker: fixed ones are resolved and open ones are never reposted;
- silence on a clean re-review.
- Reused as is: the webhook, delivery dedupe, guards, run history, the creator's keys, archive-on-success, and messaging the agent to fix something.
What's in this branch
pull-request-updated(automation-github.ts): opened, reopened,ready_for_review, andsynchronize. Drafts and bot senders start nothing. One delivery can be two events: an opened PR also firespull-request-opened.- Workspace automations hear GitHub events:
- The listener now matches any of the automation's own repositories (
automation-store.ts), with the owner still a member of its workspace. - Create no longer refuses a GitHub trigger on a workspace automation (
automations-app.ts). - This also closes a bug on main the bot found: PATCH could give a planless automation a GitHub trigger that then never fired.
- The listener now matches any of the automation's own repositories (
- A run starts in the repository the event named, among the automation's repositories (
automation-runs.ts). - The preset (
packages/features/src/automation-presets.ts): name, trigger, and the final review prompt. The editor will offer it; for now its only consumer is the webhook test. - Docs:
docs/proposals/code-review-bot.md: the design, the research table, the options compared, the PR plan, and the decisions.code-review-bot/research.md: notes on 12 products, all URLs accessed 2026-10-15.code-review-bot/results.md: every run, and every finding judged.code-review-bot/harness/: the run harness.code-review-bot/index.html: the show.
Tests, each failing without its change:
- A delivery test (
github-webhook-app.test.ts): a workspace automation made from the preset queues a run for an opened, pushed, or ready PR in any of its repositories, and none for a draft, a bot's push, or a close. - A run test (
automation-runs.test.ts): the agent starts in the event's repository when it isn't the automation's first. automations-app.test.ts: the create refusal for a GitHub trigger on a workspace automation becomes a 201.
What's rough:
- The listener matches a JSON array with no index on repositories, which is fine at today's scale.
- The preset has no UI, because the automation editor doesn't exist yet.
- The prompt was tuned on this one repository.
./scripts/verify.shwas not run, and CI doesn't cover the Mac.
Demo
#2751 is a throwaway draft with three planted bugs and no test.
- First review (link): all 4 problems and nothing else.
- After a push fixing two: it resolved those two threads and posted nothing.
- After a push fixing a third and planting a regression: it resolved the third and missed the regression, which sat under a comment calling it deliberate.
The reviews were posted from Dean's gh (each says so), because this sandbox holds no App key. In Luke the same gh api call goes out as luke[bot], on the token the agent already has.
Show: https://stage-w2203.conductor.show/
Results (results.md)
All runs used headless Claude Code on Opus 5.5 high, handed the task text exactly as automation-runs.ts composes it, behind a gh shim that refuses writes.
- Precision: 36 comments posted across every run, 0 wrong, 0 nitpicks.
- Dry runs on 5 merged PRs: 4 clean, and 1 real bug that shipped (#2746's PATCH gap).
- Replays at the commit Devin Review first reviewed, on 7 PRs:
- The final prompt found 6 of Devin's 14 findings, plus 3 real ones Devin missed.
- One of those three is still on main: Home's Inbox lists teammates' items from #2741. It was traced end to end.
- v1 found 4 of the 14, plus 2.
- So: precision is high and recall is below Devin's. The misses are mostly state going stale, and stated intent respected too readily.
- Cost: $0.24–0.83 and 38–152 s per review. The spike's runs cost about $16 in all.
🤖 Generated with Claude Code
Summary by cubic
Overnight spike for a code review bot, implemented as an automation preset instead of a new subsystem. Do not merge: this exists to validate the design in docs/proposals/code-review-bot.md before turning the roadmap item into real work.
A pull request opened, marked ready, or pushed to now fires a new pull-request-updated trigger. Workspace automations hear GitHub events in any of their repositories, and each run starts in the repository the event named. Each run is an ordinary coding agent that posts one GitHub review as luke[bot]; all noise control (a findings bar, ≤5 P0/P1 items, no reposting earlier threads, silence on clean re-reviews) lives in the ~6k-character prompt.
In this branch
- The code review preset (
packages/features/src/automation-presets.ts) and thepull-request-updatedtrigger, reusing the webhook, delivery dedupe, guards, run history, and archive-on-success. - Fixes a main-branch bug the spike found: PATCH could give a planless automation a GitHub trigger that then never fired.
- Proposal, research on 12 products, measured results, a run harness, and a show page under
docs/proposals/code-review-bot; live demo at https://stage-w2203.conductor.show/.
Measured results
- 36 comments across all runs: 0 wrong, 0 nitpicks; dry runs on 5 merged PRs found one real bug that shipped.
- The final prompt found 6 of Devin Review's 14 findings plus 3 real ones Devin missed, one of which is still on main.
- $0.24–0.83 and 38–152 s per review; the whole spike cost about $16.