fix(dreamer): gate and scan the retrospective on real message activity

Closed
TR
2mo ago
+495-54

The retrospective gate tested session_projects.updated_at > watermark, but updated_at is written only at first project binding (and by the backfill at scan time) — it is not an activity timestamp:

  • under-scan: once the watermark passed a session's registration time, new messages in that session could never re-trigger a run, and a session truncated at the per-session cap never had its tail read;
  • over-scan: a backfilled old session is stamped with the backfill time, newer than every message it contains, so it stayed eligible forever and re-ran nightly to no effect.

Derive the activity signal from the message table at query time instead:

  • a new MessageActivityProvider counts root sessions with a message newer than the watermark (indexed message table, sub-ms); a missing provider or an unavailable store is treated conservatively as "run" (the executor bails before any child session);
  • the retrospective scanner makes message activity the eligibility driver, demoting the updated_at filter to the indexless-provider fallback.

I'm not a huge fan of the "read the opencode DB everywhere" pattern and this fix does not address the issue on Pi, but it is one option for a fix, so I thought I'd offer it as a PR. An alternative might be to put an actually useful activity timestamp in the DB. The downside of that is that one has to always keep that timestamp in sync. For that option, three requirements would have to be met for the timestamp:

(a) Same time base as the watermark — the session's max message time_created, not wall-clock at registration.
(b) Advanced on every activity, not once at binding.
(c) Existing rows repaired — otherwise every already-registered stale session keeps its wrong value forever.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes the retrospective gate and scanner so eligibility is driven by real message activity instead of session_projects.updated_at, which records first project binding/backfill time, not activity. That caused under-scan (sessions with new messages never re-triggered once the watermark passed their registration time) and over-scan (backfilled old sessions re-ran nightly). A new MessageActivityProvider counts root sessions with messages newer than the content watermark directly from the message table; a missing provider or unavailable store is conservatively treated as "run", and providers without the indexed frontier keep the updated_at filter. This fix does not cover the Pi store.

Bug Fixes

  • Sessions truncated at the per-session cap now get their tail read when new activity arrives.
  • Backfilled sessions with no messages past the watermark no longer stay eligible forever.
  • Subagent messages no longer count as project activity for the retrospective gate.

RetriggerConfidence Score: 3/5

This PR is not yet safe to merge because it can suppress Pi/OMP retrospectives and can permanently miss a message inserted during an OpenCode scan.

Findings

  1. P1 Pi retrospectives are skipped ▶
  2. P1 Concurrent messages can be lost ▶
Summary

This PR replaces registration-time retrospective eligibility with activity derived from OpenCode message timestamps and threads the new activity provider through scheduling paths.

  • Extracts canonical root-session selection and adds an indexed per-session message frontier.
  • Uses that frontier to select and order bounded retrospective scans.
  • Adds conservative handling for a missing message store and tests the new gate behavior.
  • The timer path currently applies the OpenCode-specific provider to Pi/OMP registrations, and the multi-statement scan still has a same-millisecond insertion race.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Retrospective becomes due] --> B[Message activity gate]
    B --> C[Select project root sessions]
    C --> D[Query message timestamps newer than watermark]
    D -->|No activity| E[Skip until next schedule]
    D -->|Activity| F[Acquire retrospective lease]
    F --> G[Query oldest frontier again]
    G --> H[Read bounded eligible sessions]
    H --> I[Normalize and globally cap messages]
    I --> J[Persist maximum safely scanned timestamp]
    K[Pi/OMP registration] -. currently receives OpenCode provider .-> B
    L[Concurrent message insertion] -. can occur between G and H .-> J

Reviews (1) · Last reviewed commit: "fix(dreamer): gate and scan the retrospe..."