Index-backed adaptive context signals and token budgets

Merged
UN
3mo ago
+975-87

Summary

  • route pm context and pm next through the public SDK signal store with metadata-index cursor provenance and isolated projection namespaces
  • persist caller-independent activity/graph/priority/risk/knowledge signals while recomputing author, clock, semantic, and usage overlays per read
  • expose positive --token-budget controls and explained feature-store metadata across CLI, SDK, MCP schemas, runtime contracts, and generated fixtures
  • expose the effective transactional metadata-index cursor without leaking runtime file layout; harden corrupt-state fallback and SDK namespace validation
  • add focused public documentation and package-generated changelog output

pm lineage

GitHub intake discovered during this implementation is documented without expanding PR scope: pm-ypuc39 / GH-641, pm-lcnk2n / GH-642, and pm-tz2ikr / GH-638.

Verification

  • exact all-source coverage: 335 files, 6,079 tests, 100/100/100/100
  • typecheck, ESLint, zero-clone duplication, 100% docstrings/static quality, SDK package boundary, token-budget, contract, docs-link, context-evaluation, secret, and npm vulnerability gates pass
  • packed npm smoke: 13 packages; package-first dogfood: 88 commands
  • fresh packed consumer proved CLI context rebuilt -> fresh reuse, independent next storage, SDK cursor access, derived-index provenance, and a 192-token bound
  • fresh 10k context p95: 583 ms CLI / 207 ms SDK, 1,300 / 892 estimated tokens
  • Sentry: 0 critical/high issues in 14 days; telemetry: 0.46% finish error rate and 0 missing error-code rows

The aggregate scale check still reports the pre-existing unrelated get token baselines (CLI 1,042 > 322; SDK 701 > 288), already documented on pm-801d. Context itself remains within the product latency and token targets.

Tracker state

pm-3hps is closed with full evidence. pm-4k6b and pm-801d are open/unclaimed because their canonical 100k/1M acceptance remains intentionally unmet. No pm items are in progress.

Summary by cubic

Routes pm context and pm next through a shared SDK signal store with index-backed snapshots, explainable ranking, and explicit token budgets, now with normalized validation and sensible defaults. Also surfaces ranking controls on pm claim --next.

  • New Features

    • Shared pipeline via readWorkspaceContextSignals/ContextSignalStore; isolates context/next namespaces and persists caller-independent signals with dynamic overlays per read.
    • Token budgets and ranking controls: add --token-budget (alias --token_budget) and --explain-ranking to context, next, and claim --next; wired across CLI flags, commander contracts, completions, tool option contracts, parameter schema/tables, and generated fixtures.
    • API/docs: expose readItemMetadataDerivedIndexState for the effective cursor with safe fallback; add “Context Relevance and Packing” guide and performance notes; changelog entry kept in Unreleased and pm-changelog upgraded to 2026.7.22. Tracks progress: closes pm-3hps; advances pm-4k6b and pm-801d. Records GitHub intake: pm-ypuc39, pm-lcnk2n, pm-tz2ikr.

  • Bug Fixes

    • Normalized --token-budget validation and bound default packing budgets to the scale-adjusted context limit.
    • Removed stale context-signal imports flagged by CI.
    • Split complex command integration tests around a shared fixture to satisfy strict complexity gates.