fix(workspace): keep tool order stable across steps so the prompt cache hits

Merged
AN
2mo ago
+75-5

Issue for this PR

Closes #1402

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Why. In workspace mode, the second model call of every user turn re-sent the whole prompt uncached. On the RSI harness that is a full ~35k-token cache write instead of a read on every turn: more latency (uncached prefill) and roughly 12× the input cost for that call (Anthropic: write 1.25× vs read 0.1×).

Evidence. Across 222 saved harness sessions, the second call missed the cache in 49/49 workspace-mode sessions and 0/169 non-workspace sessions. Capturing the raw provider requests showed the system prompt and every tool definition were byte-identical between calls; only the order of the tools array changed: datamate_manager moved from index 79 to 92.

Root cause. pinTurnTools (engine-overlay.ts) keeps a turn on the engine tools it catalogued first. On later steps it deleted every datamate_-prefixed key and re-added the pinned ones. JS re-inserts deleted keys at the end of the record, and the AI SDK / Anthropic adapter send tools in record order. The prefix also matches the native datamate_manager, which sits mid-record, so it was pushed behind dbt_pr_review. catalogued resets per loop(), so this repeated on every user turn, not just the first.

Fix. The first catalog's key order is stored with the pinned engine tools, and later catalogs are rebuilt in that order (keys new since then go last). Pinning semantics are unchanged: pinned values win, engine tools that appeared later are dropped, ones that vanished are restored. Membership checks use own keys, so a tool named after an Object.prototype property (constructor, toString) is kept (review feedback).

Not in this PR (follow-ups, no observed misses from them in the data above): training entries render (applied Nx) in the system prompt and bump it on injection; workspace memory/skills/identity can arrive after their bounded waits and change later calls.

How did you verify your code works?

  • Unit: three new tests in engine-overlay.test.ts assert key order (the existing ones use toEqual, which ignores order): identical catalogs with datamate_manager mid-record, a mid-turn catalog change (a pinned tool vanished, survivors reversed, a new engine tool appeared), and tools named constructor/toString. Each fails on the code before its fix and passes here. bun test test/altimate/workspace/: 872 pass, 1 fail — skill-sync "flushPendingSyncs waits…" times out at 5s in the full suite and fails identically on main; passes alone. Unrelated.

  • Typecheck: bun run typecheck clean.

  • End to end (real SaaS workspace 17, Vertex Claude Haiku 4.5, run --yolo, harness task control-customers-vip as the workspace user), raw requests captured:

    call 2 cache readcall 2 cache writedatamate_manager index
    main035,52979 → 92
    this branch35,18051579 → 79
  • Not run: TUI / serve sessions (the pin is a no-op under serve), non-Anthropic providers (OpenAI-style prefix caching benefits from the same stable order but was not measured).

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

🤖 Generated with Claude Code


[!NOTE]
Medium Risk
Touches multi-step tool cataloguing sent to model providers in workspace mode; behavior change is narrow (ordering + edge-case key handling) but affects every turn’s second+ model calls and caching/cost.

Overview
pinTurnTools now records the first step’s tool key order alongside pinned engine tools and rebuilds later catalogs in that order (new keys append last), instead of deleting and re-inserting datamate_-prefixed entries—which had shuffled record order and broke provider prompt cache on the second model call in workspace mode.

Pinning behavior is unchanged (pinned engine values win; tools that appear only on later steps are dropped; vanished pinned tools are restored). Own-property checks (Object.hasOwn) replace naive key iteration so tools named like constructor or toString are handled correctly.

Three unit tests assert stable key order (including datamate_manager mid-catalog), order when the engine set changes mid-turn, and prototype-named tools.

Reviewed by Cursor Bugbot for commit f0c40522bea9b3d232fdefbaf26907dc3f00d96e. Bugbot is set up for automated code reviews on this repo. Configure here.


Summary by cubic

Fixes the workspace prompt-cache miss: in workspace mode, the second model call of every user turn rewrote the full ~35k-token prompt as a cache write instead of reading it, because pinTurnTools re-added pinned engine tools at the end of the tools record and the provider sends tools in record order. Deleted and re-added keys moved datamate_manager from index 79 to 92, missing the cache entirely — roughly 12× the input cost plus uncached prefill latency per turn.

The first catalog's key order is now stored with the pinned tools, and later catalogs are rebuilt in that order with new keys appended last; pinning semantics are unchanged. The rebuild uses own-key checks so tools named after Object.prototype properties like constructor or toString aren't dropped.

Adds three tests asserting key order and prototype-named tool retention (the existing ones use toEqual, which ignores order); all fail on main and pass here. E2E on a real workspace shows the second call reading 35,180 cached tokens instead of writing 35,529, with datamate_manager staying at index 79.

Summary by CodeRabbit

  • Bug Fixes
    • Workspace tool catalogs retain their original ordering throughout a turn, with newly appearing entries appended in arrival order. Pinned engine tools remain consistent with the first catalog, even if later catalogs remove or change them. Current non-engine tools are preserved while the catalog updates.