test(evals): exercise the workspace V4 CLI

Draft
SH
2mo ago
+1428-385

Summary

Make workspace evals and packaging consume the V4 Browse CLI proven in #2835.

  • Point Stagehand evals at the workspace Browse package instead of published Browse 0.9.5.
  • Update native and external-agent adapters to the current Browse command grammar.
  • Keep eval session names within Unix socket limits, restrict the wrapper to driver commands, and clean daemons up on abort/signals.
  • Add command-mapping and real wrapper-process coverage.

The Browse migration changeset now belongs to #2835, the first complete supported-parity head; this PR has no changeset in its own diff. The unrelated eval CLI --verbose changes from the old draft are intentionally excluded.

Stack (#2872)

  1. #2833 — exact Browse V3 baseline import
  2. #2834 — Stagehand V4 runtime and standard command parity
  3. #2869 — CLI-owned cursor overlay
  4. #2849 — CLI-private CDP sidecar; V3 network parity
  5. #2835 — remove --return-xpath; supported V3 parity/release checkpoint
  6. #2838 — eval and packaging integration
  7. #2839 — managed Context names (fast-follow)
  8. #2701 — shared Functions core consumer (fast-follow)

Review boundary

This is a consumer/integration layer, not another runtime-parity layer. Its diff is limited to eval adapters, their tests, workspace dependency wiring, and the lockfile.

E2E Test Matrix

The current remote PR head is f13b62d4c17afc55673b82a34ea03622e6eec570. The complete matrix below ran at f8e2579571b5c059753881f14b80abb718066a94; the only inherited change since then is #2849's test-helper timeout diagnostic, whose focused network tests and Browse lint/typecheck passed at sidecar head 9887732b6. The product diff is unchanged from proof commit 1dd7d0c8ef7ac65697f91780620a417a8ceb6bdd, where the packed-consumer and browser-backed wrapper flows below ran. Browser-backed runs used an isolated daemon directory; the pre-existing default daemon was not touched.

Command / flowObserved outputConfidence / sufficiency
Fully verified head f8e257957: pnpm install --frozen-lockfile; build browse; build @browserbasehq/stagehand-evalsFrozen install passed; Browse generated its real Oclif manifest; eval ESM and CLI bundles both built.Proves the regenerated lockfile and exact current workspace dependency graph build.
Fully verified head f8e257957: full @browserbasehq/stagehand-evals test suite81 test files / 633 tests passed in 102.97 seconds.Proves the propagated parent/lock changes did not regress the complete eval suite, including Browse integration coverage.
Product proof head 1dd7d0c8: pnpm --filter browse pack; install the tarball in an empty npm consumer; run node_modules/.bin/browse --version and browse open --helpTarball installed; publish rewrite changed workspace:* to Stagehand 4.1.0; installed binary reported browse/0.9.6; V4 --remote and --cdp flags were present.Proves the publish artifact is consumable outside the monorepo and does not leak an invalid workspace range. Product code is unchanged at the current head.
Product proof head 1dd7d0c8: built ESM prepareBrowseCliHarnessAdapter twiceMetadata resolved the workspace packages/cli/bin/run.js at version 0.9.6; generated sessions and workdirs were distinct.Proves evals consume the workspace CLI and isolate concurrent/per-row state.
Product proof head 1dd7d0c8: execute the generated pinned wrapper as a real subprocess: browse open https://example.com, then browse get titleWrapper launched a real Browserbase V4 session, returned selected-target/session identity fields, and read Example Domain.Proves the wrapper executes the pinned entrypoint and appends its environment/session flags correctly.
Product proof head 1dd7d0c8: invoke browse cloud contexts list through that wrapperRejected locally with exit 64.Proves non-driver commands cannot escape the eval harness boundary.
Product proof head 1dd7d0c8: register both live adapter cleanups; cooperative abort; aggressive abortCooperative abort preserved both workdirs; aggressive abort stopped cleanup paths and removed both workdirs.Proves the two-stage abort semantics and real adapter cleanup are wired together.

@browserbasehq/stagehand-evals is private and is intentionally tested as a built workspace consumer, while the public browse package is additionally tested as an isolated packed consumer.