fix(cache): replay steps on nodes known only by a generated test id

Merged
PI
11d ago
+445-72

Part of ENG-845.

A flow editor names each node rf__node-<uuid> and mints a new id on every run. When that test id was all a node had, the replay cache recorded it verbatim, so the next run could never find it. A step whose effect included such a node ended in end-mismatch, and a click on one in target-not-found, on every run (REPLAY_STALE under --strict-cache).

  • testIdShapeTier (packages/e2e/src/cache/relocate.ts) reads a test id that is a node's only identity as a shape: each uuid, hex id, cuid, or recent epoch time in it becomes # (rf__node-#, row-#-delete). Indexed ids such as todo-item-12 or rf__node-dndnode_0 stay exact.
  • End check (packages/e2e/src/cache/anchors.ts): such a node is skipped while a stable anchor exists, like volatile text. When it is the only change, any node of its shape that was not on screen before the replay counts. A vanished one is still checked exactly.
  • Click targets: the recording counts the nodes of that shape (position). After an exact miss, replay matches by shape only where a place means something, the same rule as an unnamed control: the node was alone, or it sits in a named row or group.
  • No REPLAY_POLICY_VERSION bump. Exact matching runs first and is unchanged, the shape paths only run where main always missed, and none of the 389 test id descriptors in the committed benchmark recordings has a minted part.
flowchart LR
  T[recorded target] --> E{exact match}
  E -- found --> A[act]
  E -- none --> M{only a minted test id?}
  M -- no --> H[hand off]
  M -- yes --> P{alone when recorded, or in a named container?}
  P -- no --> H
  P -- yes --> C{same count of that shape?}
  C -- yes --> A
  C -- no --> H

What the "add a node" step records:

// main
"endAnchors": [{ "role": "status", "name": "Node added" }, { "role": "box", "testId": "rf__node-712f7aa3-f7cc-4bbf-bf52-1ddf74…" }]
// this branch
"endAnchors": [{ "role": "status", "name": "Node added" }]

Known limits:

  • nanoid's alphabet includes - and _, so its pieces often look like words. 79% of nanoids are recognized and skipped as end anchors, but two nanoids share a shape only about 20% of the time; the rest hand off as on main.
  • Two nodes of one shape with no named container still hand off: their order is not an identity.
  • A click recorded before this release has no count, so it hands off once and re-records.

Verified

Ran it locally: yes

  • Scratch testbed project: "Add node" appends <div data-testid="rf__node-${crypto.randomUUID()}"> and sets a status; clicking a node sets Node <n> selected. Built-in agent (openai/gpt-6-luna-fast), one recording run, then e2e run --strict-cache, the same commands on main (ea7cdb51) and this branch:
Stepmainthis branch
add a nodeREPLAY_STALE (end-mismatch): missing box rf__node-712f7aa3-…replayed in 172 ms, no model call
click the only nodeREPLAY_STALE (target-not-found)replayed in 152 ms, no model call
click the second of two nodes, no containerREPLAY_STALE (target-not-found)REPLAY_STALE (target-not-found), on purpose
  • The same recordings replayed against a broken app on this branch: an "Add node" that does nothing fails with end-mismatch, and an extra node on the canvas with target-ambiguous.
  • Web benchmark tests-agent with CI=1 --strict-cache: 26 replayed · 2 handed off · 4 missed on main and on this branch, the same six steps (two never recorded, four with a run-time value).