fix(workspace): keep tool order stable across steps so the prompt cache hits
Issue for this PR
Closes #1402
Type of change
- Bug fix
- New feature
- Refactor / code improvement
- Documentation
What does this PR do?
Why. In workspace mode, the second model call of every user turn re-sent the whole prompt uncached. On the RSI harness that is a full ~35k-token cache write instead of a read on every turn: more latency (uncached prefill) and roughly 12× the input cost for that call (Anthropic: write 1.25× vs read 0.1×).
Evidence. Across 222 saved harness sessions, the second call missed the cache in 49/49 workspace-mode sessions and 0/169 non-workspace sessions. Capturing the raw provider requests showed the system prompt and every tool definition were byte-identical between calls; only the order of the tools array changed: datamate_manager moved from index 79 to 92.
Root cause. pinTurnTools (engine-overlay.ts) keeps a turn on the engine tools it catalogued first. On later steps it deleted every datamate_-prefixed key and re-added the pinned ones. JS re-inserts deleted keys at the end of the record, and the AI SDK / Anthropic adapter send tools in record order. The prefix also matches the native datamate_manager, which sits mid-record, so it was pushed behind dbt_pr_review. catalogued resets per loop(), so this repeated on every user turn, not just the first.
Fix. The first catalog's key order is stored with the pinned engine tools, and later catalogs are rebuilt in that order (keys new since then go last). Pinning semantics are unchanged: pinned values win, engine tools that appeared later are dropped, ones that vanished are restored. Membership checks use own keys, so a tool named after an Object.prototype property (constructor, toString) is kept (review feedback).
Not in this PR (follow-ups, no observed misses from them in the data above): training entries render (applied Nx) in the system prompt and bump it on injection; workspace memory/skills/identity can arrive after their bounded waits and change later calls.
How did you verify your code works?
-
Unit: three new tests in
engine-overlay.test.tsassert key order (the existing ones usetoEqual, which ignores order): identical catalogs withdatamate_managermid-record, a mid-turn catalog change (a pinned tool vanished, survivors reversed, a new engine tool appeared), and tools namedconstructor/toString. Each fails on the code before its fix and passes here.bun test test/altimate/workspace/: 872 pass, 1 fail —skill-sync"flushPendingSyncs waits…" times out at 5s in the full suite and fails identically onmain; passes alone. Unrelated. -
Typecheck:
bun run typecheckclean. -
End to end (real SaaS workspace 17, Vertex Claude Haiku 4.5,
run --yolo, harness taskcontrol-customers-vipas the workspace user), raw requests captured:call 2 cache read call 2 cache write datamate_managerindexmain0 35,529 79 → 92 this branch 35,180 515 79 → 79 -
Not run: TUI /
servesessions (the pin is a no-op underserve), non-Anthropic providers (OpenAI-style prefix caching benefits from the same stable order but was not measured).
Screenshots / recordings
Not a UI change.
Checklist
- I have tested my changes locally
- I have not included unrelated changes in this PR
🤖 Generated with Claude Code
[!NOTE]
Medium Risk
Touches multi-step tool cataloguing sent to model providers in workspace mode; behavior change is narrow (ordering + edge-case key handling) but affects every turn’s second+ model calls and caching/cost.
Overview
pinTurnTools now records the first step’s tool key order alongside pinned engine tools and rebuilds later catalogs in that order (new keys append last), instead of deleting and re-inserting datamate_-prefixed entries—which had shuffled record order and broke provider prompt cache on the second model call in workspace mode.
Pinning behavior is unchanged (pinned engine values win; tools that appear only on later steps are dropped; vanished pinned tools are restored). Own-property checks (Object.hasOwn) replace naive key iteration so tools named like constructor or toString are handled correctly.
Three unit tests assert stable key order (including datamate_manager mid-catalog), order when the engine set changes mid-turn, and prototype-named tools.
Reviewed by Cursor Bugbot for commit f0c40522bea9b3d232fdefbaf26907dc3f00d96e. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Fixes the workspace prompt-cache miss: in workspace mode, the second model call of every user turn rewrote the full ~35k-token prompt as a cache write instead of reading it, because pinTurnTools re-added pinned engine tools at the end of the tools record and the provider sends tools in record order. Deleted and re-added keys moved datamate_manager from index 79 to 92, missing the cache entirely — roughly 12× the input cost plus uncached prefill latency per turn.
The first catalog's key order is now stored with the pinned tools, and later catalogs are rebuilt in that order with new keys appended last; pinning semantics are unchanged. The rebuild uses own-key checks so tools named after Object.prototype properties like constructor or toString aren't dropped.
Adds three tests asserting key order and prototype-named tool retention (the existing ones use toEqual, which ignores order); all fail on main and pass here. E2E on a real workspace shows the second call reading 35,180 cached tokens instead of writing 35,529, with datamate_manager staying at index 79.
Summary by CodeRabbit
- Bug Fixes
- Workspace tool catalogs retain their original ordering throughout a turn, with newly appearing entries appended in arrival order. Pinned engine tools remain consistent with the first catalog, even if later catalogs remove or change them. Current non-engine tools are preserved while the catalog updates.