Catch up on mentions made while the connector was down, and restart it after a memory stop

RO
3d ago
+133-21

Summary

When Claude Code stops basecamp connect because memory is low, the agent stops answering. Mentions made while it's down are also lost when it starts again. This PR makes two changes:

  • A restarted connector catches up. It hands off requests made up to 4 hours before it started, as long as this run admitted them. Before, the limit was one minute.
  • The skill starts the connector again at once after a memory stop, and the watcher too.

Stacked on #902 (connect-one-shot-watcher). The skill text changes #902's watcher steps, and #902's handled-ids file is what stops a request from being handled twice across restarts. Merge #902 first.

Card: Claude Code kills the connector for low memory, and the agent stops answering. These are options 2 and 3 of the proposal on the card.

Why

Claude Code 2.1.303 stops an idle session's background Bash tasks under critical memory pressure. All of these have to hold:

  • the system reports critical pressure
  • on Linux, less than 10% and less than 4 GiB of memory is available
  • nobody has typed for 30 minutes
  • no turn or delegated agent is running

Its notice says "stopped because the system is running low on memory … Do not start it again on your own". The session then waits for a person, and the agent stays deaf. Claude Code ends the task's process group and descendants, and it doesn't look at the task's own memory, so a small connector gets stopped too. A task started again within 30 minutes of such a stop isn't stopped again until someone types in the session.

On restart, handOff closed every admitted request made more than HandoffGrace (one minute) before the run started as before_this_run. The feed resumes from its checkpoint, so intake does see the mentions made during the outage. Handoff then dropped them.

Change

internal/connector/handoff.go

  • New rule in handoffCutoff. If this run wrote a record's admission decision (Decision.DecidedAt at or after Started), the record may have been made up to handoffCatchUp (4 hours) before the run.
  • A record an earlier run admitted keeps HandoffGrace (one minute). An earlier run's decision was made under whatever trust held then. Replaying it hours later could hand off a request from someone who was removed from --allow while the connector was down. Requests made during an outage are admitted by the new run, under current trust.
  • The before_this_run log line now carries the cutoff.

internal/commands/connect_run.go: the run's start is taken before intake and admission start, not after. Otherwise something this run admits could be stamped before Started and get the one-minute rule.

skills/basecamp-connect/SKILL.md

  • When the connector stops: on a low-memory notice, start the connector again at once as in step 1. Then drain the old output file and start the watcher on the new one, and tell the person. The person asked for it to run when they started it.
  • A watcher-only notice restarts the watcher on the running connector's current file, never on an old one.
  • When the connector stops, Checking on it and Stopping now state the 4-hour catch-up instead of "not picked up later" and "starts from the present".
  • The per-request steps are unchanged, so the compaction budget isn't touched.

Tests

  • handoff_test.go:
    • A request this run admitted is handed off when made 2 minutes, 1 hour or exactly 4 hours before the run.
    • It is closed as before_this_run at 4 hours and 1 second.
    • A request an earlier run admitted is handed off at exactly one minute, and closed at one minute and one second and at one hour.
  • skill_layout_test.go: the memory-stop steps exist, including the watcher rule. The skill states the catch-up in the same number of hours as handoffCatchUp, so changing one without the other fails.
  • Mutations. Each of these fails tests:
    • using the one-minute rule for every record (the this-run test, at 2m, 1h and 4h)
    • using the 4-hour rule for every record (the earlier-run test, at 1m1s and 1h)
    • reverting the skill (both skill tests)
  • Not covered by a test: moving started earlier in runConnect. It removes an ordering race between the handoff's start time and admission's first decision. I found no seam to test that without a real run.
  • bin/ci is green locally (MISE_BATS_VERSION=1.13.0, as in #902).

Trade-offs

  • Late answers. After a stop of up to 4 hours, people get answers to what they asked while the agent was down, possibly after someone else has dealt with it. That's the point of the change. The 4-hour window is a judgment call, set in one constant.
  • Overlap with open PRs.
    • #825 adds HandoffOptions.Grace and changes the same handOff lines. It keeps HandoffGrace as the earlier-run rule, which this PR leaves alone, so its e2e test's expectation still holds. A textual conflict is likely, and it's easy to resolve.
    • #907's connect release help says a restart doesn't replay older requests. That stays true for an earlier run's records, but would read better if it mentioned the catch-up.

Summary by cubic

A restarted Basecamp connector now answers requests made while it was down (up to 4 hours before the run), instead of dropping anything older than a minute. The skill also tells the agent to restart the connector at once after Claude Code stops it for low memory.

  • Requests the current run admitted get the 4-hour catch-up window; requests an earlier run admitted keep the one-minute grace, so a restart never replays a decision made under trust that may have changed while the connector was down.
  • The run's start time is recorded before intake and admission begin, so requests admitted by this run are never stamped before the run.
  • The memory-stop steps restart the connector and the watcher. If a notice names only the watcher, check the connector first: if it was stopped too, restart it (which starts a new watcher) and never start one on the old file; otherwise start the watcher on its current output file.