blog: Grading our own homework (RelayFlow vs raw agent benchmark)
Draft blog post about the Terminal-Bench 2.0 pilot comparing a raw cursor-cli agent against the same agent wrapped in a RelayFlow evaluator-optimizer loop.
Covers, honestly:
- The first tied result (9/12 both arms) and why it was wrong, not a real tie
- Two bugs found, both quietly penalizing RelayFlow: a shared agent-timeout clock, and exceptions bypassing our own cost/bookkeeping (
asyncio.CancelledErroris aBaseException, notException) - The fix, applied identically to both arms
- The real repeated-trial result: 33/36 (91.7%) vs 28/36 (77.8%), Wilson CIs still overlap, RelayFlow never lost on any of the 12 tasks
- What we're NOT claiming (a proven win) vs what we are (never behind, sometimes ahead, at 1.14x cost)
Needs review per repo rules (two reviews, one from a different agent/human) before merge. Not yet ready to publish -- flagging as draft for content/tone review first.
Summary by cubic
Adds a draft blog post documenting a Terminal-Bench 2.0 pilot that compares a raw cursor-cli agent against the same agent wrapped in a RelayFlow evaluator-optimizer loop.
The first run tied at 9/12 because two bugs biased against RelayFlow: a shared agent-timeout clock and asyncio.CancelledError bypassing its cost/bookkeeping. After fixing both identically in both arms, the repeated result was 33/36 (91.7%) vs 28/36 (77.8%) with overlapping Wilson CIs. The post claims only that RelayFlow was never behind and sometimes ahead at 1.14x cost, not a proven win.
This draft is not ready to publish; it needs content/tone review and two approvals per repo rules, one from a different agent or human.