feat: record-video returns a frame sheet and change numbers an agent can read, and accepts a mark

Merged
HA
8d ago
+3501-25

Summary

  • Stopping a recording now returns things an agent can read without watching the video: a PNG sheet of 32 consecutive frames around the moment worth looking at, per-frame change numbers in a sidecar JSON, and their summary on the stop response.
  • New --action mark tells a running recording "the moment I want to see is now"; the burst sheet is centered on the last mark, or on the largest change when nothing was marked.

User Impact

  • Before: record-video returned only a video path. Agents cannot watch video, so they fell back to single screenshots even for flicker, stutter, or one-frame glitches.
  • After: the stop response (manual or after an auto-stop, reported by the next stop/status) carries BurstSheetPath (4 columns x 8 rows, top-left first), BurstFrames (frame number and seconds of each tile), BurstAnchor (mark or largest-change), MarkSeconds, AnalysisPath, and FrameChange (changed-cell counts against the previous frame and the one two before, the largest change, and the top 3 grid cells that flip back and forth, as top-left-origin rectangles in video pixels). The numbers are measurements, not verdicts. Without a mark, a recording where no frame changes in at least two grid cells has no burst sheet and no largest change, matching its zero changed frames.
  • Start and status responses keep their earlier shape; every new field is omitted when empty.

Changes

  • Each captured frame (not each encoded repeat a slow Editor adds to keep pace) is reduced to a 320-wide tile and a 64-column luminance grid on CPU arrays. Tile arrays that leave the look-back window are reused, so recording does not allocate per frame. Nothing at stop time creates a Texture2D, so stops triggered by an assembly reload or Editor quit still write the files.
  • The sheet and <stem>_frames.json are written beside the video. A failing analysis is logged and the video is kept without it.
  • An auto-stopped recording keeps only the sidecar path in SessionState; the analysis is read back from the file.
  • Sidecars in the default Videos folder follow their video: after the existing retention, sidecars whose .mp4/.webm is gone are deleted.
  • mark is added to the record-video Action enum (catalog enum edited by hand, descriptions regenerated, release-input stamps refreshed, generated skill copies updated). Mark without a recording fails and, unlike stop, does not report the last auto-stopped recording.

Cost

Measured per captured frame at 1280x720, 30 fps, 10 s recordings:

  • Debug code optimization, final code (tile arrays reused): 0.564 / 0.592 / 0.546 ms average, maxima 0.9-2.7 ms, 0-1 gen-0 garbage collections per recording.
  • Before the tile arrays were reused (one 230 KB array per frame): 0.780 / 0.802 / 0.801 ms average, with occasional 1.5-5 ms maxima from garbage collection.
  • Release code optimization (measured before the reuse): 0.475 ms average.
  • 1.0 ms is about 3% of a 33 ms frame at 30 fps.

Verification

  • uloop compile: 0 errors.
  • uloop run-tests (regex over the new and touched record-video test classes plus SkillParameterTableCoverageTests and DefaultToolsCatalogDriftTests): all passed (61 in the final run over the burst, contact sheet, frame change, session, and analyzer tests).
  • scripts/sync-tool-docs.sh --check passes; check-release-triggers --base origin/feature/record-video-readable-evidence --head HEAD passes.
  • Live Editor checks:
    • Game View with a mark: BurstAnchor: mark, the sheet's orientation matches the scene, an object blinking every frame at the bottom-right shows up as bottom-right TopAlternatingCells, and the sheet's dominant colors equal the screenshot's.
    • Game View without a mark: BurstAnchor: largest-change.
    • Editor window (Console, outside Play Mode): orientation and colors match screenshot --window-name Console.
  • A manual --action mark sent from a terminal can take Editor focus and briefly pause capture (about 0.7 s observed); the one-shot in the follow-up change runs focus-window first.
  • "Mark while recording succeeds" is pinned at the session host level: the use case goes through the static recording service, which tests cannot replace.