Keep basecamp connect running, and starting, through a rate-limited or failing token endpoint

Merged
JE
17d ago
+1560-57

The failure

When the agent token endpoint answers a renewal with a 429 or a 5xx, or doesn't answer at all, basecamp connect stops.

  • Running: the event feed classifies a failure only by the SDK's error type. managerTokens handed it the CLI's own error, so the feed read every non-transport failure as unrecoverable and ended the run.
  • Starting: the first read of who the agent is needs a token. A failed renewal ended the start.

Since #814 the connector is started from the person's Claude session by /basecamp-connect, and nothing restarts it. A connector that exits on a brief rate limit stays down until someone notices, and requests made while it's down are not picked up later.

The fix

One rule for the feed and the start. tokenRetry is the only classifier.

  • The feed reads through feedTokens, which hands it every token failure already classified as the SDK's error type.
  • A starting connector waits by the same rule in awaitConnectToken.
  • The two can't drift apart.

The status decides first. A response that arrived is classified by its status, even if its body was cut off or too large to read.

  • A 401/403 or invalid_client/invalid_grant is a refusal.
  • A 429 or 5xx is retryable.
  • Only a 200 cut short counts as a network failure.

A request that got no response is a network failure, and it's retried. That covers a dropped or refused connection, DNS, a timeout, and TLS or proxy failures. Retrying a misconfigured TLS or proxy setup forever is accepted on purpose: the process stays visibly alive and logs why.

Never retried: an egress-policy block (surfguard.ErrBlocked), or a URL the client can't send.

How long it waits: exactly the wait the server named, never longer than one cap. Otherwise it uses the feed's backoff. Nothing else sets it.

  • If the server named a wait, in seconds or as a date, on any retryable status (a 429, or a 5xx such as a 503), it waits exactly that.
  • One cap applies to every token path: an agent mint, a 5xx, and a bot user's OAuth refresh, at start and in the feed. The cap is auth.MaxServerWait, 30 minutes: the bound #787 already writes a rate-limit hold under, so a held rate limit and every other named wait agree.
  • When the server asks for longer than the cap, the start and the feed log once what it asked for and what they will wait.
  • A held rate limit is a named wait too. #787's hold error, and the 429 that sets it, carry what's left of the hold, read from its stored deadline (#787's Rule 1). That includes a 429 with no Retry-After, which #787 holds for its default.
  • Otherwise it uses the feed's backoff: random, starting at 1s, capped at 60s.
  • It never falls back to a default wait from the SDK.

The start waits instead of exiting. It logs each wait once and keeps trying. A SIGTERM, an interrupt or a canceled context ends the wait promptly, even mid-renewal, with the connector's usual exit codes, and a second signal forces the exit. Any other failure at start still reads "Could not read who profile … is". Before waiting, the start checks the instance lock under the agent that connect.json names, taking it and releasing it at once. A second connector for a running agent is therefore refused immediately instead of waiting out a rate limit first. The lock is held for the run only once the credential is proven to be that agent, so a start holding some other agent's credential doesn't block the real connector while it waits.

The skill knows. /basecamp-connect treats the "could not get the agent's token yet" line as a connector that is waiting, not one that failed, and leaves it running.

What stays fatal

A credential refusal exits as a disconnect, in the running feed and at start alike. That means invalid_client, invalid_grant, or a bare 401/403, including one whose body was cut off. Other non-retryable answers keep their current behaviour.

Dropped since earlier revisions

  • The systemd start-limit rationale. #815 removed connect service, so the CLI no longer writes a systemd unit. The start waiting still matters, more so now: nothing restarts the connector at all.
  • Nothing on the worker side. This PR never touched the dispatcher, the drivers or the outbox, so #815's removal didn't take anything from it.

Tests

  • connect_token_retry_test.go: the live feed through feedTokens, against a fake token endpoint:
    • 429 with Retry-After in seconds and as a date → throttled
    • 429 without one → throttled for the default hold
    • 5xx, a truncated 503, or a dropped connection → transient
    • invalid_client, a bare 401, or a truncated 401 → unrecoverable, and recognised as a disconnect
  • connect_start_wait_test.go: the start:
    • 429 then 200, and 503 three times then 200 → waits, then comes up
    • a stored 429 hold due 90s out → waits 90s, not the backoff
    • a truncated 401 → exits as a disconnect
    • a truncated 503, a truncated 200, or an oversized 503 page → waited out
    • an untrusted TLS certificate → keeps waiting, logged
    • refusals → no wait
    • cancel, SIGTERM or interrupt during a wait or mid-renewal → ends promptly
    • a second signal → forces the exit
    • a second connector while the first holds the lock → refused at once, with nothing minted
    • a start waiting for its token → holds no lock
    • a 503 or 502 that names a Retry-After → waited exactly that, by the start and the feed
    • a 503 naming Retry-After 99999999 → waited for the cap, by the start and the feed, with the capped wait logged once
    • a closed signal channel → not taken as a stop; no log set → still waits
    • which no-response failures are retried and which are not
  • agent_hold_test: pins the wait a held mint carries.

go test ./internal/auth/... ./internal/commands/... ./internal/connector/... -race -count=2 and lint, compared with a clean origin/main on the same machine, show no new failures.

Side effects

  • The SDK's raw GET path now retries a rate-limited or failing token renewal within one call, the same way it retries a 429 from the API.
  • setup.ErrorText now renders such a failure as HTTP 429 / HTTP 503.

Not addressed

A credential refusal that arrives while the feed sits idle on its websocket. The feed makes no token requests while idle, so nothing notices a refusal until it next needs a token.