feat(question): require the agent to recommend an option on every question

Merged
KE
1mo ago
+107-3

The agent now has to say which option it would pick on every question. The question tool requires a recommended field, so a recommendation can't be forgotten. Replo shows it to people and goes with it when nobody answers in Slack, Messages, or a task.

Context: step 3 of the auto-answer plan (Slack thread) · replaces #61 (an optional flag, closed) · upstream's tool only suggests a recommendation ("if you recommend…, add (Recommended) to the label"). Over the last 30 days only 29% of steps had one. When one existed, people picked it 70% of the time. Option 1 was picked 64% of the time.

What changed

  • Tool input: QuestionPrompt requires recommended, the exact label(s) the agent would pick. An empty list is the agent's explicit "the user decides" (payments, credentials, publishing). Labels must match an option, and a single-choice question takes at most one.
  • Tool text: the rule says to recommend on every question, pick what's easiest to change later, and keep "(Recommended)" out of labels.
  • Stored and served questions: QuestionInfo.recommended is optional.
  • Only the v1 tool, the one opencode serve runs. v2 (packages/core) stays as upstream wrote it, to keep the fork's diff small.

Backwards compatibility

  • Questions asked before the upgrade: stored and listed questions never required the field, so old rows still load, list, and answer. The existing question and route tests ask without it and pass.
  • Clients: the SDK and OpenAPI show recommended?: string[]. Older clients ignore it, and newer ones treat a missing field as "no recommendation".
  • A model mid-turn during the upgrade: its call fails validation once. It then retries with the new tool definition, which is re-sent on every request.

Worth a close look

  • Upstream removed strict validation from this tool once ("due to tool call errors", see the test file). Here the model only copies labels from options in the same call, and a miss returns an error that lists the valid labels. If tool-call errors show up in production, the fallback is case-insensitive matching.

Testing Done

  • Tool: a question keeps its recommendation through the stored row. Missing, unknown, and two-on-single-choice recommendations send the model back with the reason. The existing tests now pass the field. Mutation check: making the field optional fails the test. 4/4.
  • Question service: 17/17, unchanged. These tests ask without the field, which is the backwards-compat case.
  • Route harness: test:httpapi 213/213 in coverage, auth, and effect modes. Typecheck: 30/30.
  • SDK regenerated by hand. The diff is the new optional field only.

Next

  1. Review and merge. I won't merge it.
  2. I cut v1.18.32-5 with its SDK tarball (the #60 steps). Andytown then moves all three pins in #28547, shows the recommendation, and #28112 goes with it when nobody answers.

🤖 Generated with Claude Code

Summary by cubic

Requires the question tool to recommend an option on every question, so a recommendation can't be forgotten. Previously recommendations were optional and only 29% of steps had one; when one existed, people picked it 70% of the time.

  • recommended is now required on the tool's input; an empty list is the agent's explicit "the user decides" (payments, credentials, publishing). Labels must match an option, and a single-choice question takes at most one.
  • Stored and served questions keep the field optional, so questions asked before the upgrade still load and answer. plan_exit builds its question by hand and now recommends "No" since staying in plan mode is the reversible choice.
  • Only the v1 tool changes; v2 stays as upstream wrote it.