feat(engine): bind strict spawn replay to exact nodes

Merged
KH
1d ago
+34046-51

Current exact head

Head: cc025be4d6bcc7ec58617a561c43dde86ac8a6bb

This successor closes the current review findings without weakening strict-spawn lifecycle guarantees: replay documentation is lookup-only; every unready unsent placement releases only its owned capacity reservation while preserving the exact-node token for same-key retry; stale dispatchers cannot decrement a newer claim; and an unknown send outcome reports the committed dispatched snapshot. The latest false-ready reservation regression is mutation-verified, the complete crash-recovery file passes 32/32, and engine lint/build are clean: https://github.com/AgentWorkforce/relaycast/pull/644#issuecomment-6771930470. Exact-head canonical CI, binary smoke, self-host, CodeRabbit, Greptile, Codex code review, and Codex security review are green at this head; zero review threads remain unresolved. Snow Lunch independent source review is requested and pending. No merge, deployment, release, or publication has occurred.

Outcome

  • Add POST /v1/actions/spawn/strict-invoke, a dedicated mixed-version-safe route for builtin spawn using relay.spawn.invoke.logical/1.
  • Bind the durable claim to workspace, caller, action, idempotency key, exact node ID/name, agent name, harness, task, requested directory, and readiness requirement.
  • Allow only server-materialized worker_cwd to differ when replaying the same logical request.
  • Add replay-only POST /v1/actions/spawn/replay, which never creates missing work and may resume an existing, validated, definitely-unsent claim on its exact node. Dispatched, terminal, and unknown-send outcomes remain immutable replays.
  • Persist spawn_contract_applied server-side and return it only for a builtin invocation with null action ID and a valid stored strict contract. Client-supplied proof is rejected.
  • Make the strict contract exclusive to the dedicated route. Generic and node-addressed legacy routes reject it instead of exposing a second proof surface.
  • Bypass registered spawn actions and node shadows for strict invokes; exact-node placement never falls back by name, capability, capacity, or replacement node.
  • Keep generic legacy spawn behavior unchanged when the strict contract is absent.

This is the server half of a paired Relay/Flows migration. Its client is Relay PR #2119. It remains a draft and does not merge, release, publish, or deploy anything.

Historical verification at c74c34065c3fa7bcc95d98a3ea4038e44dee9340

Two earlier broad attempts are not claimed: one failed with Linux EDQUOT while tests wrote under the shared /tmp; another, run concurrently with lint/build, reached 2,668 passes and one timeout. The clean isolated rerun moved test temporary files onto the home filesystem and bounded workers:

$ TMPDIR=/home/khaliqgant/.cache/relaycast-707-tmp npm test --workspace @relaycast/engine -- --reporter=dot --maxWorkers=4

 Test Files  189 passed | 2 skipped (191)
      Tests  2669 passed | 42 skipped (2711)
   Duration  115.76s (transform 8.69s, setup 113.21s, import 67.36s, tests 238.93s, environment 24ms)

$ npm run lint --workspace @relaycast/engine

> @relaycast/engine@12.8.0 lint
> eslint src/

$ npm run build --workspace @relaycast/engine

> @relaycast/engine@12.8.0 build
> tsc && node -e "const fs=require('fs');fs.cpSync('src/db/migrations','dist/db/migrations',{recursive:true});fs.cpSync('src/db/migrations-pg','dist/db/migrations-pg',{recursive:true})"

$ node -e "const fs=require('fs');const YAML=require('yaml');YAML.parse(fs.readFileSync('openapi.yaml','utf8'));console.log('openapi.yaml parsed')"
openapi.yaml parsed

$ npx vitest run packages/types/src/__tests__/sdk-openapi-sync.test.ts --reporter=dot

 Test Files  1 passed (1)
      Tests  3 passed (3)
   Duration  194ms

$ git diff --check
# exit 0, no output

Mutation: the strict contract is accepted only on the dedicated route

Production mutation: remove the generic-route refusal while leaving the conformance assertion intact.

$ TMPDIR=/home/khaliqgant/.cache/relaycast-707-tmp npx vitest run packages/engine/src/__tests__/conformance/node.test.ts -t 'binds logical spawn replay' --reporter=verbose

 FAIL  packages/engine/src/__tests__/conformance/node.test.ts > node adapter conformance > fleet node control > binds logical spawn replay to the exact node while ignoring only rematerialized cwd
AssertionError: expected 201 to be 400 // Object.is equality

- Expected
+ Received

- 400
+ 201

 Test Files  1 failed (1)
      Tests  1 failed | 82 skipped (83)

After byte-for-byte restoration:

$ sha256sum packages/engine/src/engine/action.ts packages/engine/src/__tests__/conformance/node.test.ts
9abd966bb1668f4ec1cbf20ba73974ca70baa1d5b0c28d1e6df2532d7ef35ed0  packages/engine/src/engine/action.ts
d0ff15bd3b8e3b340211ec3ca6e23f41e4fb02c50e55c438e75b62c0bb368bc0  packages/engine/src/__tests__/conformance/node.test.ts

$ TMPDIR=/home/khaliqgant/.cache/relaycast-707-tmp npx vitest run packages/engine/src/__tests__/conformance/node.test.ts -t 'binds logical spawn replay' --reporter=verbose

 Test Files  1 passed (1)
      Tests  1 passed | 82 skipped (83)
   Duration  3.47s

The focused conformance case also pins: missing strict key, unknown top-level field, workspace-key refusal, client proof spoof refusal, registered spawn shadow bypass, builtin/null-action provenance, exact ID/name/capability failures, offline refusal, cwd-only rematerialization, caller isolation, name reuse, replay-only behavior, refusal to upgrade an old inert row into a strict claim, and rejection of strict bodies on generic and node-addressed routes.

Independent review

Independent review of the complete exact-head diff found no remaining issue at
c74c34065c3fa7bcc95d98a3ea4038e44dee9340 (binary diff SHA-256
bf46cd47c1443b1f79ef8e08413f138435d86ef66122f07c835967d83feb8049).
The reviewer confirmed the dedicated-route restriction, generic-route OpenAPI
responses, node/core-bridge transport classification, route ordering,
authentication, replay/idempotency behavior, and legacy compatibility. Their
focused reruns passed the SDK/OpenAPI sync test 3/3 and node conformance 2/2
(81 skipped); git diff --check origin/main...HEAD was clean.

Delivery boundary

The companion Relay client must ship only after this server route is available. Flows #707 stays draft until the paired Relay client and production Desktop service-caller implementation are released and an actual local direct-node flow proves spawn, task delivery/output, restart/retry reconciliation, release, and no broker/PTY use.