fix(session): keep the cached system prompt prefix stable across steps

Draft
RA
4h ago
+435-72

Issue for this PR

Closes #1633

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The system prompt is the cached prefix of every request, so any change to it between two requests of one session is a full cache miss: the gateway bills the whole prefix (system prompt plus tool schemas, about 75K tokens on a typical repo) again as cache writes at the full input rate, instead of as cache reads at 0.1×. Two parts of the prompt changed for reasons unrelated to the task.

1. The Altimate Workspace section flipped on a timer (altimate/workspace/identity.ts).

  • It rendered one sentence for a fresh link check and another for a memoised one. For an unlinked repo that was "No Altimate Workspace is linked to this project." vs "…was linked … as of the last check, up to five minutes ago…". The outcome memo is 30 s and the miss memo 5 minutes, so a long session alternated between the two.

  • Separately, when a re-check outlasted the 1.5 s deadline, lastKnown marked the previous (fresh) outcome stale. For a linked repo that removed the workspace contents and the team-memory line from the prompt for that step.

  • The "Team memory:" line was read from the 60 s positive enablement cache, so on a linked project it disappeared a minute after each enablement check and came back on the next one.

Fix: one hedged sentence per verified link state, true whether or not the last check is fresh ("…as of the last check. The link is re-checked every few minutes, so a change made elsewhere may not show yet."), and a slow re-check reuses the previous outcome as it was, bound or unbound. The team-memory line reads the last answer any enablement check gave (memoryEnabledLastKnown, keyed by tenant, API URL and workspace id, no expiry on read; a failed check keeps it). Only the resolver marks a link stale (the server could not confirm it); that case keeps its own "was last known to be linked … could not be re-verified just now" sentence and still withholds contents and the memory line, as before. A summary refresh that fails at its 5-minute TTL keeps the previous contents instead of replacing them with "not known".

2. Hoisted reminders went into the cached system message (session/prompt.ts).
After a compaction, the original-task pin (and the plan-switch reminders) are hoisted out of the user turn for models that are not Anthropic-like (#887/#888). The altimate-backend/altimate-default model is registered as OpenAI-compatible, so it takes that path, and the hoisted text was appended to the system array, which is the message carrying the cache breakpoint. The first request after every compaction therefore missed on the whole prefix, not just on the replaced history.

Fix: the hoisted text goes with the trailing environment system message, after the breakpoint (llm.ts already splits that message out for providers with explicit breakpoints). It is still sent as system text, so the #887 behaviour is unchanged.

Behaviour change to review: a verified link reads the same whether its memo is fresh or memoised ("as of the last check"); an unconfirmed (resolver-stale) link still says "last known". The team-memory line now stays on after the last enablement check said "enabled", until a later check says otherwise.

Not in this PR (limits):

  • After a compaction the prefix can still change for other, deliberate reasons: lessons.compact re-renders teamRules once a request- or file-tier lesson has been shown, and addHistoricalToolStubs changes the tool list when a compaction removes the last reference to a gone tool. The mock capture below has neither.
  • Contents timing on a linked project: a cold workspace summary slower than 300 ms is missing on the first step only, and the skills read can render "could not be read just now" under heavy load.
  • Skills synced from the workspace, memory injection and team rules can change the prompt mid-session on events. On providers without explicit breakpoints the date sits in the first system message, so a session crossing midnight changes it once.
  • A per-request prefix-hash log/counter.

How did you verify your code works?

  • test/session/system-prefix-stability.test.ts (rewritten after review): through the real prompt loop against a mock OpenAI-compatible server on an altimate-backend model, a builder turn after a plan turn carries the hoisted plan-to-build reminder as system text, and its first system message is byte-identical to the earlier builder turn's. Fails with the previous prompt.ts. Replaces the hand-built LLM.stream test and source-text guard, which passed on main.

  • identity-section.test.ts, "prefix hash over steps with a stepped clock" (setSystemTime moves Date.now for every module): an unlinked project over six steps including a hanging re-check past the 5-minute miss memo, and a linked project with memory on over six steps (T+30 s, T+61 s, T+2 min, two hanging re-checks past 5 min). One hash each, and the team-memory line on every step. Both failed before the fix (2 hashes; line gone at T+61).

  • contents.test.ts: a failed refresh past the TTL keeps the cached summary and retries next step (failed before the fix).

  • identity.test.ts: memoised and fresh misses render identically; an unconfirmed link is hedged as "last known" (pinned and unpinned).

  • The earlier stale/fresh render tests and the v0.12.1 release-contract tests are updated to the wording above.

  • No-regression run, bun test test/altimate/workspace test/session test/skill (packages/opencode), on the merged head before the review fixes (374bfb80e9) and after (3d2a473812):

    • before: 3,962 pass, 0 fail (4,019 tests). Failures: none.
    • after: 3,961 pass, 0 fail (4,018 tests; +4 new, −3 from the replaced stability file, −2 from collapsing the stale-render loop). Failures: none.
    • also after: bun test test/altimate test/memory: 10,975 pass, 1 fail, tool CLI > the daemon releases a DuckDB file between calls, which fails the same way with these changes stashed (DuckDB native binding).

  • Typecheck clean; script/check-tracker-leaks.ts and analyze.ts --markers --strict pass.

  • Request-body capture of both builds against a local mock gateway (comment): on main the first system message changed twice in a 7-minute session on an unlinked project, and once after a compaction; on this branch it was byte-identical across every request in both scenarios.

Still to add: a short gateway A/B (cache_read on the request after a timer flip and after a compaction, including a linked project and one with lessons).

Cost side effects

Reduces cost; adds no model calls, queries or resources. In a headless evaluation of about 5,400 requests, the system-prompt hash changed 785 times; 315 of those were full mid-session cache misses, and prefix re-writes were about 43% of billed tokens. This PR removes the two causes identified there; the remaining misses need the per-request hash log to attribute.

Screenshots / recordings

N/A

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR
  • I checked for cost side effects (model calls, queries or cloud resources that grow with user data) and described them, or wrote "none"

🤖 Generated with Claude Code