Test how basecamp connect fails and recovers, end to end

Closed
RO
15h ago
+976-32

PR 3 of a stack of 8 that tests basecamp connect end to end. Stacked on #823 (the harness). It adds the failure and recovery scenarios, and fixes one real bug they found.

The first commit is the same change as #820 (the connector reports its backlog edges on stderr). The backlog scenario needs it. When #820 merges, that commit drops out on rebase.

Why

#823 holds the connector to the happy path. A connector has to live through more than that: a Ctrl-C, a second copy started from another repo, a disconnect in Basecamp, a rate-limited token endpoint, a dropped or silent cable, a flaky read, a crash in the middle of a handoff, and a backlog. Each of these is a fault that Basecamp or the operator can really cause. Unit tests prove the parts. They do not prove what a person sees when the real binary meets the fault.

What

A product fix (its own commit, with unit tests): the connector now stops when Basecamp no longer takes its agent's credential.

The bug: a running connector learned of a disconnect only from the event feed. The feed sends a request to Basecamp only when it connects. While its live connection stays up, it sends nothing. Admission and intake send reads all the time, but to them a refusal was only one failed read. Admission blocked the mention as read_failed, intake kept its last project list, and the connector ran on. Every mention to the agent was blocked, with no explanation, for as long as the socket stayed up.

Reproduced end to end: while the connector streams, disconnect the agent, or connect it on another computer and let its token come due. Then the operator mentions the agent. The mention is blocked read_failed and the connector does not exit.

The fix is a credential watch (internal/commands/connect_credential.go) on the reads that admission and intake make:

  • When the token endpoint refuses a renewal (invalid_client), that is already Basecamp's answer. The run stops at once.
  • A read answered 401 only causes a check, because something other than Basecamp can answer 401. The watch asks Basecamp itself (the same confirmAgentRefused the feed's stop uses). It stops the run only if Basecamp confirms.

In both cases the connector exits 3 with the message the feed's stop uses ("... was disconnected in Basecamp, or connected on another computer", and the connect setup hint), and status records it as disconnected.

The scenarios (e2e/connect/failures_test.go). No knobs, no sleeps used to sync. Every wait is on a line or on the fake's state, with a deadline that only bounds a failure.

Fake changes (only additions): Server.Disconnect (revokes the agent's tokens and clears its secret, so a read gets 401 and a mint gets invalid_client), and AgentClient.TokenLifetime (tokens that say they last as little as a second, so a renewal comes due during a test). Both have tests in the fake's own package.

Harness changes: Profile / SetupProfile / ConnectProfile to connect a second agent, AnotherComputer (a second home against the same Basecamp), and Connector.Pid.

Tests

  • SIGINT on a running connector: it logs connector: shutting down with signal=interrupt, exits 130, and logs no ERROR.
  • A second connector for the same agent, from another directory with the same XDG_STATE_HOME: it exits 5 at once, writes nothing to stdout, and its refusal names the pid of the first connector. The fake sees no request from it, not even for a token. The test checks only the exit code and the pid, so the new wording in #821 also passes.
  • Two agents (a second agent person and OAuth client in the world) run at the same time. Each gives one request for its own mention and not_addressed for the other agent's mention.
  • Disconnected while running: the next read gets 401 and the connector exits 3 with the disconnect message. No request is printed. A second variant: the cable drops and the reconnect's stream ticket is refused. That also exits 3.
  • Connected on another computer while running (the invalid_client trigger): the world's tokens last one second. Another home runs auth agent connect, which rotates the secret, as Basecamp does, and leaves this computer's token alive. When the token comes due, the next mention's read renews it, the renewal gets invalid_client, and the connector exits 3. The one wait on the clock is the token's own expiry, which auth status --json reports. That expiry is the premise of the scenario, not a guess about timing.
  • Token 429 at start (Retry-After: 1, once, with the stored token due): it logs connector: could not get the agent's token yet, comes up, and hands off a mention.
  • Cable drop: the reconnect is held at its stream ticket, an event is put on the poll lane only, and then the reconnect is released. The event arrives by catch-up (pointer lane poll) and gives exactly one request.
  • Stale cable (StopPings): the connector reconnects and hands off a later live mention. This runs once, because staleness is 7.5s and nothing can make it shorter.
  • A recording read fails twice, then serves: one admitted event line, one request, and three reads of the comment.
  • Crash during handoff: connect --hold, mention, wait for its admitted event line (no request), kill -9, basecamp connect release -P agent, restart. The request for that event_id is printed once (within the grace period).
  • Backlog: recording reads are held open and DefaultBacklogWarn + admission.DefaultWorkers events are emitted live (each worker holds one event at its read). The warn line arrives with warn_at=1000. The reads are released and the recover line arrives.

The 11 scenarios and the 5 from #823 run in parallel in about 13s (the stale cable sets the floor). go test -race -count=10 ./e2e/connect passed in 154s, and go test -count=20 ./e2e/connect passed in 190s, with no failures. bin/ci is green.

Summary by cubic

Fixes the connector running on after its agent is disconnected in Basecamp or connected on another computer, and tests how basecamp connect fails and recovers, end to end.

The fix

  • A running connector learned of a disconnect only from the event feed, which asks Basecamp for nothing while its live connection stays up; admission and intake saw the refusal as one failed read, so every mention was blocked read_failed and nothing said why.
  • A credential watch over admission's and intake's reads stops the run: a refused token renewal ends it at once, and a 401 read ends it only when Basecamp confirms the agent is gone.
  • The connector exits 3 with the disconnect message, and status records it as disconnected.

The tests

  • 11 scenarios cover SIGINT, a second connector, two agents, disconnects, a rate-limited token, cable drops and staleness, flaky reads, a crash during handoff, and a backlog; each waits on a log line or fake state, never on a sleep.
  • The fake can disconnect an agent (tokens revoked, secret cleared) and mint short-lived tokens so a renewal comes due within a test; the harness handles a second agent and a second home.
  • The backlog queue now reports its warning, pause, resume and recovery edges on the logger, which production never wired to anything before.