PR #1006 - basecamp/basecamp-cli

On Linux, one keyring read that goes over the 10-second limit leaves basecamp connect alive but deaf. The process stays up, delivers nothing, and never exits. Only a restart brings it back. This happened three times on one machine on 2026-11-06, while dozens of agents were running (card).

What happened

credstore refuses every keyring call for the rest of the store's life after one call times out (basecamp/cli#79). This is by design: go-keyring can't cancel the call, so an unknown write must never be overlapped. Its documented way out is a new store, which for the connector meant a restart.
Every API request reads the token from the keyring, so from then on every token request failed.
The error wraps context.DeadlineExceeded, so the SDK's feed classifies it as transient. A catch-up walk then retried its poll forever, and the feed has no log line for a poll retry. The walk never reached its head, so the stream was never renewed in band. The server closed the socket at its advertised deadline: 19:00:07 against a deadline of 18:00:06Z, and 21:45:10 against 20:45:08Z. After that the reconnect retried its ticket mint forever, also with no log line.

The server did nothing wrong. No HTTP errors were logged, and both disconnects land on the stream deadline the server advertised. A keyring read also never waits on the network: a refresh holds a file lock across the token request, not the keyring.

Why the keyring was slow: gnome-keyring-daemon 51.1 aborted 25 times that day, and systemd restarted it each time. Each abort came from an assertion in its OpenSession handler when a client hung up mid-call. Each restart drops every in-flight Secret Service call. This upstream bug is long-standing, so the fix must not depend on it being fixed.

Load alone is enough, though. The third stall, at 23:24:47, came with one daemon running and no crash or restart logged: the same daemon process ran throughout, and the load average was about 34 on 32 cores. The log reads could not refresh the agent's projects: keyring unavailable after an earlier timeout: keyring read timed out after 10s. A catch-up walk started at 23:25:16 and never drained. So one slow read on a busy machine is enough to latch the store, and what has to change is the latch, not the daemon.

What this changes

Everything works by default. There are no new environment variables or flags.

A read the keyring didn't answer is labeled as such. It is wrapped as auth.ErrKeyringUnavailable. That covers a timeout, credstore's refusal after one, a D-Bus failure (the Secret Service gone or restarting), or a bus connection that hangs up, resets, refuses or breaks. A permission refusal (fs.ErrPermission) is an answer and is not labeled. The label is applied only while the keyring is the store in use, so a file-backend miss that carries an old probe timeout isn't mistaken for an outage. unreadable passes it through as itself rather than as "not authenticated".
The token stays usable from memory. auth.Store keeps the credential it last read or wrote for each key. It forgets it on a missing or invalid read, a delete, or a write of unknown outcome. Each key carries a generation that every write and invalidation bumps, so a read that was in flight across a save or a delete can't put back what it replaced. While the keyring isn't answering, Manager.storedAccessToken serves the held credential if it isn't due for renewal. That includes the read-back right after a renewal was saved. Nothing is written.
A timed-out read is retried. After a backoff of 15s, doubling up to 5 minutes, the Store builds a fresh credstore, outside its lock. The fresh store is kept only if its probe reached the keyring, so a retry never falls back to credentials.json. Results from a replaced store are ignored. A timed-out write or delete keeps credstore's latch, because its outcome is unknown.
A restarting keyring is asked again. A restarting Secret Service can report a credential it handed out a moment ago as missing. If this process read that credential before, the read asks once more after 500ms. A caller that gives up meanwhile ends the wait, and no second read starts. A credential that is really gone is still reported as gone.
The feed can't wait forever. In the connector's feed token provider, an outage is logged once and retried. Any answer from the keyring resets the clock, even a failure from something else. After 15 minutes of silence, the error ends the feed, so the connector exits non-zero with the reason. A connector that starts while the keyring is silent waits it out on the start backoff, as it already did for a timeout, rather than exiting on a D-Bus error.

Earlier work this must not undo

PR / issue	What it fixed	How this keeps it
#578	Lazy store construction, so credential-free commands never probe	Construction is still `sync.Once` on first use. A rebuild happens only after a keyring read timed out.
#581, #805 / #800, basecamp/cli#79	Bounded probes and operations; no silent switch to file storage once the keyring is in use; a timeout is not a missing login	The bounds are unchanged. A fresh store that fell back to file is discarded. Writes and deletes keep the latch. Timeouts stay matchable and are never `auth_required`.
#664, basecamp/cli#70	No silent stale `credentials.json`; one probe item per probe	Never falls back to file. A rebuild probes under a new pid+sequence account.
#720, #711	Cross-process refresh lock; rotated refresh tokens are never lost	Nothing is written from memory. Renewal still re-reads under the key lock. A held credential is never served once it is due for renewal.
#811	The feed survives a rate-limited or failing token endpoint	Every non-keyring failure still goes through `tokenRetry` unchanged.
#866 (closed unmerged)	Proposed: a refused store is not "log in"	Nothing of it is on main. This PR only keeps its premise: `fs.ErrPermission` is the keyring answering, not an outage, so it is never served from memory or waited out.
#713, #787, #959	Revocation and refusal handling	A missing read forgets the held credential. A revoked token is refused by the server (401) and ends as before.

Tests

internal/auth/keyring_stall_test.go covers:
- the retry after the backoff, and exactly one retry per capped wait;
- no fallback to file storage;
- no retry after a write timeout;
- a late timeout from a replaced store being ignored;
- the held token being served for a timeout and for a missing Secret Service;
- a vanished credential being asked for again;
- the held token not being served when nothing is held, when it is due for renewal, for a canceled caller, after a missing read, or on a file backend;
- the held token being served for a dropped bus (EOF, unexpected EOF, reset, refused, broken pipe), and not for a permission refusal;
- a read racing a delete or a save not restoring the old credential;
- the re-ask wait ending with the caller;
- a renewal whose read-back stalls serving the token it saved.
internal/commands/connect_keyring_stall_test.go covers:
- a stall being retried and logged once;
- a silent keyring ending the run on both the poll and mint lanes;
- a token, or a non-keyring failure, resetting the outage clock;
- a start waiting out a silent keyring.

With each behavior switched off (and the tests kept), its tests fail. They pass with it.

For review

Where the retry lives. The retry could later move into credstore: clear the latch when the abandoned call returns. That keeps "one abandoned call per store" exactly. Here, while D-Bus stays hung, each retry can leave one abandoned read, at most about one every 5 minutes.
The limits. The 15-minute outage limit and the 500ms re-ask are judgment calls.
Per-process memory. The held credential is per process. A logout in another process during a stall isn't seen until the keyring answers again (at most the 5-minute backoff) or the token is due for renewal. Agent logout doesn't revoke the token anyway (#851).
Out of scope. A probe that fails at process start still picks the file backend for the life of the process (#805's behavior, unchanged). That start fails visibly on credentials.json; it doesn't stall.