feat: record-video returns a frame sheet and change numbers an agent can read, and accepts a mark
Merged
HA
8d ago+3501+3501+0123456789012345678901234567890123456789-25-25-01234567890123456789
Summary
- Stopping a recording now returns things an agent can read without watching the video: a PNG sheet of 32 consecutive frames around the moment worth looking at, per-frame change numbers in a sidecar JSON, and their summary on the stop response.
- New
--action marktells a running recording "the moment I want to see is now"; the burst sheet is centered on the last mark, or on the largest change when nothing was marked.
User Impact
- Before:
record-videoreturned only a video path. Agents cannot watch video, so they fell back to single screenshots even for flicker, stutter, or one-frame glitches. - After: the stop response (manual or after an auto-stop, reported by the next
stop/status) carriesBurstSheetPath(4 columns x 8 rows, top-left first),BurstFrames(frame number and seconds of each tile),BurstAnchor(markorlargest-change),MarkSeconds,AnalysisPath, andFrameChange(changed-cell counts against the previous frame and the one two before, the largest change, and the top 3 grid cells that flip back and forth, as top-left-origin rectangles in video pixels). The numbers are measurements, not verdicts. Without a mark, a recording where no frame changes in at least two grid cells has no burst sheet and no largest change, matching its zero changed frames. - Start and status responses keep their earlier shape; every new field is omitted when empty.
Changes
- Each captured frame (not each encoded repeat a slow Editor adds to keep pace) is reduced to a 320-wide tile and a 64-column luminance grid on CPU arrays. Tile arrays that leave the look-back window are reused, so recording does not allocate per frame. Nothing at stop time creates a Texture2D, so stops triggered by an assembly reload or Editor quit still write the files.
- The sheet and
<stem>_frames.jsonare written beside the video. A failing analysis is logged and the video is kept without it. - An auto-stopped recording keeps only the sidecar path in SessionState; the analysis is read back from the file.
- Sidecars in the default Videos folder follow their video: after the existing retention, sidecars whose
.mp4/.webmis gone are deleted. markis added to the record-videoActionenum (catalog enum edited by hand, descriptions regenerated, release-input stamps refreshed, generated skill copies updated). Mark without a recording fails and, unlikestop, does not report the last auto-stopped recording.
Cost
Measured per captured frame at 1280x720, 30 fps, 10 s recordings:
- Debug code optimization, final code (tile arrays reused): 0.564 / 0.592 / 0.546 ms average, maxima 0.9-2.7 ms, 0-1 gen-0 garbage collections per recording.
- Before the tile arrays were reused (one 230 KB array per frame): 0.780 / 0.802 / 0.801 ms average, with occasional 1.5-5 ms maxima from garbage collection.
- Release code optimization (measured before the reuse): 0.475 ms average.
- 1.0 ms is about 3% of a 33 ms frame at 30 fps.
Verification
uloop compile: 0 errors.uloop run-tests(regex over the new and touched record-video test classes plusSkillParameterTableCoverageTestsandDefaultToolsCatalogDriftTests): all passed (61 in the final run over the burst, contact sheet, frame change, session, and analyzer tests).scripts/sync-tool-docs.sh --checkpasses;check-release-triggers --base origin/feature/record-video-readable-evidence --head HEADpasses.- Live Editor checks:
- Game View with a mark:
BurstAnchor: mark, the sheet's orientation matches the scene, an object blinking every frame at the bottom-right shows up as bottom-rightTopAlternatingCells, and the sheet's dominant colors equal the screenshot's. - Game View without a mark:
BurstAnchor: largest-change. - Editor window (Console, outside Play Mode): orientation and colors match
screenshot --window-name Console.
- Game View with a mark:
- A manual
--action marksent from a terminal can take Editor focus and briefly pause capture (about 0.7 s observed); the one-shot in the follow-up change runsfocus-windowfirst. - "Mark while recording succeeds" is pinned at the session host level: the use case goes through the static recording service, which tests cannot replace.