feat(testing): add native Flakiness reporters (stage 1)
Summary
Implements the additive reporter foundation (stage 1 / Tasks 1–4) from docs/plans/2026-11-05-flakiness-testing-dashboard.md (#1165). This is not the hosted cutover or dashboard-retirement PR.
- Pin official
@flakiness/playwright@2.0.3and@flakiness/vitest@1.12.0, without upgrading Playwright or Vitest. - Wire every inventoried runner configuration: CMS, smoke, WWW, infrastructure, inherited no-server mode, core, SDK, root, sample app, reporting contracts, and the existing create-app CLI verifier.
- Hard-disable Flakiness uploads in CI. Local uploads require explicit opt-in, a locally supplied token, and a project. No new credentials, OIDC permissions, uploader workflow, or hosted service setup.
- Preserve native terminal/JSON/JUnit/HTML/coverage outputs, discovery, exclusions, retries, concurrency, disposable-target guards, and the existing stats reporter/dashboard.
- Keep each invocation's manifest and vendor report together outside native output cleanup; preserve watch reruns and preflight duplicate Vitest names before upload.
- Inventory commands and baseline identity multiplicities; exercise real pass/failure/retry/skip/setup/import/interruption/attachment/output-error paths with network-denied subprocess fixtures.
- Repair pre-existing missing esbuild lock entries and the CLI's incomplete execa dependency closure, preserving all existing dependency versions/ranges. Supply the CLI verifier's supported disposable-admin flags so its existing assertions run noninteractively; disable fixture telemetry.
Verification
All local results below are from frozen head 081ad676d3ffacb2f92da09ff0e3b5145a131810.
| Check | Verified result |
|---|---|
| Fresh Node 22 / npm 10 install | npm ci passed |
| Reporting contracts | 102 passed |
| Core tests with coverage | 2,068 passed, 247 existing skips |
| SDK | 68 passed |
| CLI native wrapper | 1 passed; direct verifier also passed |
| Full CMS E2E | 748 passed, 8 conditional skips, 0 failed/flaky, 2 workers, retries 0 |
| Smoke | 28 passed, 0 failed/flaky |
| Browser infrastructure | 2 passed |
| No-server mode | 4 selected health/runner contracts passed against an owned, prepared disposable server; server stopped afterward |
| Core/reporting/stats TypeScript, core build, skip/tag lint | Passed |
| Native/vendor reconciliation | 11 actual invocation reports reconciled; all vendor and manifest commits match the frozen head, including honest failed reports for the pre-existing collection/empty-suite failures below |
| Full-CMS source/identity freeze | No executable source changes; git HEAD, vendor commit and manifest commit identical |
| Baseline collection preservation | Successful collections retain identical identity multisets; WWW/sample remain failed collections, and root remains empty. No unavailable identities are treated as a pass. |
| CI | Exact-head PR Tests run 42127930499 has a registered watcher; terminal conclusion/artifact closeout is not yet verified |
| Final review closeout | Blocked on execution consent for read-only PR review/check readback; consent question timed out. Independent final Opus review has not run. |
The 8 CMS skips are the 7 separate-stats-worker conditions and the missing GitHub OAuth provider condition; no new test skips were added. Local native JSON and vendor JSON agree on 756 tests and 756 attempts, with 748 passed and 8 skipped. Earlier exploratory runs are not used as final attribution proof.
Native outputs and vendor reports remain independent evidence. No hosted delivery is claimed; this PR is not being declared fully reviewed or ready to merge while review/CI closeout remains incomplete.
Existing failures and cutover blockers
- Direct root Vitest has no tests (actual run exits 1). This is not reported as a successful empty suite.
- Sample Vitest discovers an existing Playwright-only contact-form spec and fails collection. The failure remains visible; no tests were moved, excluded, or skipped to hide it.
- WWW discovery has an existing missing
tests/www/utils/test-helpersimport. Its original assertions and failure are preserved. - The pinned Playwright reporter omits interrupted attempts; the pinned Vitest reporter can omit its report after SIGINT. Native exit status/artifacts and completed interruption manifests remain authoritative. These are explicit hosted-cutover blockers, not fabricated successes.
Rollout boundary
The isolated artifact/provenance validator and trusted publishing workflow are the next staged PR. The new workflow_run consumer must be reviewed and activated on the default branch before hosted verification. Project slug/access, visibility, retention/capacity, real upload/readback evidence, and cutover approval remain required. Synthetic contract-fixture reports must never be blindly uploaded as production histories.
Legacy ingestion, the Playwright dashboard, shared D1 data, installation/demo telemetry, credentials, Worker, and domain remain intact. Nothing is merged, deployed, released, revoked, or deleted by this PR.