Read a slow keyring instead of calling its credential missing

Merged
JO
9h ago
+1201-53

On a loaded Linux machine, basecamp connect exited 3 with Profile "chef" holds no credential and suggested basecamp auth login, while the credential was in the keyring the whole time. Following that hint would have replaced it. This closes a gap in #1006: that PR handles a keyring store already in use, but not the first construction of the store, when the startup probe times out.

What happened (v0.24.2, 2026-11-07)

Four connector startups between 15:00 and 15:19 all exited 3:

warning: system keyring unavailable (keyring probe timed out after 10s: context deadline exceeded), credentials stored in plaintext at ~/.config/basecamp/credentials.json
Error: Profile "chef" holds no credential
Run: basecamp auth login

The load average was about 135 on 32 cores, /proc/pressure/io showed full avg300=28%, gnome-keyring-daemon was in D state, and 52 GB of swap was in use. The startup probe writes a throwaway entry and deletes it. gnome-keyring rewrites and fsyncs its whole keyring file on each write, so a store plus clear took 3.7s, and the probe's round trips went past its 10s bound. A plain lookup (secret-tool search) still answered in 3.6s.

Root cause

When the probe timed out, credstore fell back to credentials.json. That file held nothing for the profile, and auth.Store read the miss as ErrNoCredential, which every caller reports as "not logged in". On Linux the probe has a time limit because the Secret Service can stall (#800). A timeout there means the keyring is slow. It does not mean the keyring is absent or empty.

Changes

  • Reads go to the keyring. On Linux, after a probe timeout, a key that credentials.json doesn't hold is read from the keyring with a plain lookup, with no write and a 60s limit. If the keyring answers that nothing is there, that is a real miss. If it doesn't answer, the error is ErrKeyringUnavailable, never ErrNoCredential. Further reads then back off on the #1006 stall schedule (15s doubling to 5m), so later reads don't each wait the full 60s.
  • No plaintext after a timeout. Save and Delete after a probe timeout never fall back to credentials.json. They run a probe with a 60s limit and use the keyring, or fail as unavailable. The longer write probe runs only when a write actually needs the keyring.
  • An unavailable keyring is its own error. ErrKeyringUnavailable now reports as keyring_unavailable: retryable, exit 5 (the try-again status that busy and lock_unavailable use, not 3), with a hint that says the credential is still in the keyring and that logging in again would replace it.
  • The connector waits at startup. It waits out an unavailable keyring on its first credential read with the same rule (tokenRetry) and backoff it already used for its token, so it starts once the keyring answers. If the keyring stays silent for 15 minutes, the same limit the running feed uses, the start ends with keyring_unavailable and a restart hint. The instance-lock check runs before that wait.
  • Plaintext copies are reconciled. Reads after a probe timeout still serve a credentials.json entry first, as #800 expects. That order holds after a write has moved the process to the keyring: a profile that still has a file copy is read from the file, so the switch never brings back an older keyring entry whose refresh token the file replaced. A save or delete that goes to the keyring removes that key's plaintext copy under the whole-store lock, and reports a file it can't check. Reads take the shared store lock while a plaintext file sits beside the keyring. An unreadable credentials.json is reported as unreadable, not as "not logged in" and not as a keyring outage. auth status and doctor report the keyring, where writes go.
  • Answers stay answers, and waits stay bounded. A permission refusal on the direct read or the longer write probe is reported as itself, with no backoff (#866). A direct-read miss for a credential this process holds is asked again after the grace wait. Only one direct read runs at a time, a canceled caller isn't held for the 60s limit, and at most one abandoned go-keyring lookup exists per 5-minute backoff cap.
  • macOS is unchanged. There the probe has a time limit only in a headless session, where a timeout means a locked keychain nobody can unlock, so the #568 file fallback stays (probeTimeoutIsSlowness).

Earlier work this must not undo

  • #578: commands that need no credential still never touch the keyring. The new path runs only on a credential read or write.
  • #568 and the Headless Keyring Probe job: on headless macOS a timed-out probe still falls back to the file, and auth status still completes there. TestATimedOutProbeOnHeadlessMacOSStillFallsBackToTheFile covers this.
  • #800, #805, #581: keyring calls keep their limits, and a timeout is still never reported as a missing login. That now holds for the first probe too. A file credential found after a probe timeout is still served, as TestIssue800DesktopKeyringHandshakeMustBeBounded expects.
  • #664: a keyring is still never swapped for the file. This PR also stops a probe timeout from sending writes to plaintext.
  • basecamp/cli #79 and #1006: a timed-out write still latches. The longer-limit store is only used if its probe reached the keyring, and its later retries use the same 60s limit.
  • #720, #711: refreshes still go through the shared lock. Nothing is saved from memory.
  • #811, #866: failures that don't come from the keyring are handled as before. A permission refusal is still an answer, not an outage.
  • #713, #787, #959: logout, revoke and refused credentials are unchanged.
  • No new environment variables or flags. The hint doesn't suggest BASECAMP_NO_KEYRING.

Tests

  • TestASlowKeyringProbeDoesNotMakeAStoredCredentialMissing reproduces the incident on first construction: the probe times out at 10s, and the credential is in the keyring and not in the file. The read returns the credential with no second probe. A later save goes to the keyring after a 60s probe, and nothing is written in plaintext.
  • TestAKeyringThatDoesNotAnswerIsUnavailableNotMissing: when the lookup doesn't answer either, the error is keyring_unavailable (exit 5, retryable, no login hint). There is no second long read within the backoff, no plaintext write, and the read succeeds after the backoff.
  • TestASlowKeyringThatHoldsNothingIsAMiss and TestATimedOutProbeOnHeadlessMacOSStillFallsBackToTheFile cover the answered miss and the macOS path.
  • TestConnectStartWaitsOutAKeyringThatDoesNotAnswer: the connector's first credential read retries until the keyring answers, and an answered refusal is not waited on.

Red without the fix. With the read change reverted, the two incident tests fail with no stored credential: credentials not found for profile:chef: system keyring unavailable (keyring probe timed out after 10s: context deadline exceeded), the 0.24.2 error. With the change restored, they pass.

Follow-ups

  • basecamp/cli credstore: the first 10s probe is still a write, because credstore always probes that way. A read-only availability check in credstore would remove the remaining write cost from every command.
  • Old plaintext copies: copies that earlier versions wrote to credentials.json on a probe timeout stay there until a write goes through the slow path. Cleaning them from processes that reach the keyring is a separate migration decision. MigrateToKeyring exists but has no caller.

Card: https://app.basecamp.com/2914079/buckets/48521764/card_tables/cards/10485906408