feat(screenshot): add --crop-on to crop captures to a selector frame

Merged
TH
2mo ago
+1899-155

Summary

Add screenshot --crop-on <selector>: capture the screen, then crop the saved PNG to the frame the selector resolves to on the same screen. Runs daemon-side after the platform write and before --scale, against a fresh full-tree snapshot (no setSessionSnapshot). Accepted on iOS simulators and Android emulators; every other target is refused before device work. Cannot combine with --overlay-refs or --fullscreen.

agent-device screenshot out.png --crop-on 'role=cell label=General'
agent-device screenshot out.png --crop-on 'label="Network & internet"' --pixel-density 3

Success carries post-crop width/height and a warnings entry when the frame is clipped to the image. Failures use stable details.reason codes: CROP_TARGET_NOT_FOUND (+find hint), CROP_TARGET_AMBIGUOUS (+candidates), CROP_SELECTOR_INVALID, CROP_FRAME_MISMATCH, CROP_TARGET_NOT_ACCEPTED.

Validation

Head 62f39143ce (rebased onto 33084c7748; the review commits 7edcd2adec → c5be390a85 → 643c9b8a80 → 62f39143ce are behavior-preserving — pnpm check:affected --run green, same seam/geometry tests): pnpm check:affected --run — all runnable checks passed.

Live pixel-identity (crop region vs independent raw capture, mean-abs-diff, collected at the pre-refactor head 570da2c417; the refactor is behavior-preserving):

  • iOS sim (Settings "General" cell): 370×52 @1× / 1110×156 @3×; vs raw simctl 0.0000 at both densities; one-row-shifted control 8.44.
  • Android emulator (android-helper @ package version): 436×71 @1:1; vs raw screencap 0.0000; control 42.83.
  • Live error paths: NOT_FOUND, AMBIGUOUS (matches:11), SELECTOR_INVALID, FRAME_MISMATCH. EMPTY_INTERSECTION is unit-tested (off-screen rows are absent from the live tree). macOS cell stays rejected (terminal lacks Screen Recording).

Size review

41 files, +1899 −155: net +670 production / +971 tests (was ≈740 net / 2043 gross at the pre-review head; the review pass trimmed ~70 production lines by splitting the crop module, reusing kernel rect helpers, hoisting the duplicated meaningful-signal predicate into contracts, and dropping two redundant test cases).

Retained production growth, itemized:

  • screenshot-crop.ts (+144): the crop leaf — capture, sparse/empty refusal, selector resolution with typed ambiguity/missing reasons, project→intersect→crop, and the single warning-composition owner.
  • screenshot-crop-target.ts (+148): the target acceptance matrix (one cell per capture frame), the device→cell classifier, and the pre-device argument policy. Split from the leaf so both modules meet the 300-line target (the review's size item).
  • capture-kit/snapshot-rect-projection.ts (+138): space decision (device-pixels vs viewport-points), no-clamp projection, crop intersection, and the shared viewport-bounds resolver consumed by both the overlay and the crop. Reuses kernel isPositiveFiniteRect/rectArea and the shared isMeaningfulSignal (no local redefinitions).
  • capture-kit/png-crop.ts (+54): in-place PNG region crop on the worker.
  • screenshot-runtime.ts (+111 −15): the integration delta on the existing module — read flag, answer policy pre-device, admit the plan, run crop after the write/before scale, append the warning.
  • contracts/screenshot.ts (+66 −2): the machine-readable flag + 10-reason typed taxonomy, flag keys, --crop-on definition, options↔flags projection, script string-flag reader.
  • client/screenshot-result.ts (+55 −21): default-level normalization (incl. response-level warnings) + field picker, shared by client and CLI.
  • Smaller: screenshot-runtime-binding.ts (+26 −10, plan admits the crop's snapshot op), platform-runtime-operations.ts (+18 −7), contracts/snapshot-text.ts (+8, the hoisted isMeaningfulSignal shared by overlay + crop), selector-pipeline-policy.ts (+8), capture-kit/package.json (+8, subpaths), screenshot-overlay.ts (+7 −74, refactored to consume the shared projection and the shared predicate — net −67), cli/commands/screenshot.ts (+6), resolution-policy.ts (+5), commands/capture/screenshot.ts (+4), script-utils.ts (+4), agent-device-client.ts (+2 −12, inline reconstruction moved into the shared normalizer), contract facades (+2 each), response-views.ts (+1), snapshot-types.ts (+1).

Why a materially smaller design was rejected:

  • The acceptance matrix is explicit (one cell per target) because the completeness gate requires every target that admits the plan to have exactly one cell; a default-reject form would let a target slip through unclassified.
  • The typed reason taxonomy is mandated by "key behavior on typed reasons and details, never error text"; collapsing reasons would push callers back onto message sniffing.
  • The crop runs daemon-side (not client-side) so it reads the exact platform-written file and projects against the same admitted snapshot binding as the capture; a client-side crop would re-read the published artifact and could diverge from the snapshot frame.
  • The request-scoped crop snapshot (never setSessionSnapshot) is what keeps the authorized-ref frame a following interaction resolves untouched; reusing the session snapshot would corrupt that invariant.

Tests (+915): geometry fixtures, the matrix one-to-one classifier/policy, the crop orchestration seam across 8 outcome paths, runtime ordering/warning wiring, the contract flag round-trip, CLI warning rendering, and the focused screenshot-result normalization. Two redundant cases were dropped (the crop-on CLI forwarding case — covered by the flag-mapping contract tests plus the generic dispatcher — and the transport-based warnings case, now a focused unit test).