feat(feedback): send bugs without a repro to GitHub, refuse outdated reports

Merged
OK
12d ago
+290-42

What and why

e2e feedback reports are mostly agent-written. Most of the low-quality ones were sent from old versions, were flakes that need a repro, or repeated bugs we already knew about. 61 of 90 recent reports came from an older version than the latest release. Reports that need a repro die in Linear, because PostHog feedback is anonymous and we can't ask the reporter.

  • A --type bug report needs --repro, steps that reproduce it every time. Without it, the command sends nothing and prints a prefilled GitHub bug-form link (what and versions filled in, redacted, under 8000 chars). The user adds a minimal repro and submits it themselves. The agent never files it.
  • Before sending, it asks the registry for e2e@latest. If the installed version is older, it exits 2 and sends nothing. --allow-outdated overrides. A failed or slow registry check (3s timeout) is skipped. Under E2E_TELEMETRY_DISABLED/DO_NOT_TRACK no request is made.
  • New --error-code <CODE> field (error_code on the event) so reports group by code. --task/--expected/--actual/--approach/--repro are capped at 1500 characters.
  • Skill Feedback section:
    • never report app behaviour, one-off flakes, or outdated versions;
    • search GitHub issues first and comment on a match;
    • a bug that needs the app goes to GitHub.
  • Docs: CLI reference, telemetry, security outbound connections (registry.npmjs.org).

Verified

Ran it locally: yes

Built CLI against the live registry (latest 0.20.0):

$ e2e feedback --type bug -m "tap lands on the wrong tab" --actual "TARGET_NOT_FOUND on second run" --error-code TARGET_NOT_FOUND --dry-run
Not sent: a bug without --repro goes to GitHub, where the e2e team can ask for a reproduction. Open this prefilled issue, add a minimal reproduction, and submit it:
https://github.com/tester-army/e2e/issues/new?template=bug_report.yml&labels=bug&title=tap+lands+on+the+wrong+tab&what=tap+lands+on+the+wrong+tab%0A%0AActual%3A+TARGET_NOT_FOUND+on+second+run%0AError+code%3A+TARGET_NOT_FOUND&versions=e2e+0.20.0%2C+node+24.19.0%2C+darwin+27.0.0
exit=0

# package version set to 0.19.0
$ e2e feedback -m x --dry-run
feedback not sent: e2e 0.19.0 is older than the latest release 0.20.0, which may already fix this. Upgrade e2e, check the problem still happens, and send it again; --allow-outdated sends it anyway.
exit=2

$ e2e feedback -m x --error-code nope
error: option '--error-code <code>' argument 'nope' is invalid. must be an e2e error code such as LOCATOR_AMBIGUOUS, at most 64 characters
exit=2
  • packages/e2e unit tests: 2609 passed. New cases cover GitHub routing and redaction, the URL length cap, the outdated refusal and --allow-outdated, a registry failure, --error-code validation, and olderThan.
  • lint, typecheck and docs:check pass.
  • check:dead-code fails on 20 unused class members, and plugins:check fails because plugins/e2e/skills/migrate-to-e2e is missing. Neither comes from this PR.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.