Optional Claustrum vault custody for fallback OAuth accounts

Merged
IC
2mo ago
+11440-525

Stacked on #174, which is one commit and will drop out of this diff once it merges. This PR is the commit on top of it.

Lets a fallback OAuth account keep its credential in Claustrum, a local credential vault reached over an IPC daemon, instead of in our sidecar. Entirely opt-in and per-account: with no vault present, or no account opted in, nothing here runs.

Why bother

The vault owns refresh. Our FallbackAccountManager currently holds refresh tokens on disk and refreshes them itself; a vault-managed account holds a handle instead, and the vault runs the refresh treadmill. Fewer long-lived secrets on disk, and a credential shared by several tools stops being refreshed by each of them independently.

Scope: fallback accounts only, and main is deliberately frozen

Main-account custody is not in this PR and is not a follow-up I intend to open. refreshMainAccessToken reads getAuth() and sends freshAuth.refresh to Anthropic's token endpoint. A vault-managed main account would send a placeholder, get 400 invalid_grant, and our own classifier marks that permanent: true — permanently disabling the main account from routing and sidebar display, with no way back without the retained refresh token.

Fallback accounts are safe because they are ours end to end: our sidecar, our manager, our refresh path.

Read this before trusting the green suite

An earlier revision of this PR had never worked. Production never passed a BindIdentity, so every vault call threw SubcCallError(missing_identity) and the feature was inert — through six bot-review rounds, a cross-family drift pass, and 1385 passing tests.

Nothing caught it because every test either passed identity explicitly or injected a connector stub, and the fake daemon does not enforce what the real client enforces. The fixtures were structurally incapable of reaching the defect.

What found it was one bounded read-only probe against the live daemon, in seconds:

before   SubcCallError             missing_identity
after    ClaustrumCredentialError  action=gone  class=permanent  not_found

not_found is a pass: it proves the call cleared identity and reached the vault, and it's the correct answer for a handle that doesn't exist. Claustrum returns it for revoked and unknown handles alike, deliberately indistinguishable so the API can't be used to enumerate handles — which is also why it maps to permanent.

The cause is worth naming, because the two things are one letter apart in meaning. consumerIdentity and identity are different parameters sitting next to each other. I reasoned carefully about the first and was blind to the second.

Three things worth reviewing closely

Identity is scrubbed explicitly, not by omission. @cortexkit/subc-client reads SUBC_MODULE_ID and SUBC_LAUNCH_NONCE from process.env by default. A plugin shell spawned from a tool call inherits those from whatever supervised module spawned it — so omitting the identity argument silently authenticates as that module, with its route authority. We pass consumerIdentity: null and type-enforce it via Omit.

Separately, every managed call needs a BindIdentity: harness: 'opencode', the plugin's project directory, and a session derived as store- — the fingerprint pattern PrimeManagerRegistry already uses. Stable across restarts, distinct per config store, no new persisted field. I originally argued for a stable id because a vault's audit chain records who asked; reading Claustrum's source showed credential.get ignores session entirely and the limiter scopes on TCP connection id plus handle. The conclusion survived, the reasoning didn't — it stands on route-cache stability, not attribution.

The vault's slow path is kept off the response path structurally. credential.get is bimodal: ~35µs when the credential is resident, but seconds when it lands in the refresh window, with concurrent callers queued behind one upstream exchange. So the request path only ever does a synchronous peek; a bounded warm at startup (100ms, one attempt per account) primes the cache, and any refresh runs detached.

The bound is deliberately short and deliberately not sized for the slow case: abandoning the await doesn't cancel the vault's work, so the refresh still lands and the next call collects it. A bound sized for the slow path would block boot exactly when the slow path stopped being rare.

A 401 is reported only for a credential the vault actually served. Invalidation is version-gated, and the version we hold is usually still current — so a 401 report generated from a sidecar-served token would pass the version gate and invalidate a perfectly healthy vault credential. Version-correct, provenance-wrong. Provenance isn't checkable on the vault's side, because a capability handle authorises a read without identifying who presented it, so the guard is structural on ours: the served credential is bound to the response via a WeakMap, never inferred from account state at report time.

That guard needed a clock seam to test at all. My first two attempts could not fail — they used a credential expiring in 1ms, and the cache only stores when expiresAtMs > now(), so it was never cached, the report path found nothing, and the tests passed with or without the guard. I found that by mutating the guard away twice and getting green both times.

The maintainer round: three findings, all reproduced before fixing

Credential handles stayed out of dumps, sidebar state, and RPC — and then leaked into the log. The report_auth_failure error path logged served.handle verbatim, so a daemon outage during a report persisted the bearer to opencode-anthropic-auth.log. The blindness suite covered every surface except the logger; it now covers the logger (__setLogTestSink + sentinel handle, red on the old code).

The default WebSocket relay never reported a 401. The relay resolves an optimistic synthetic 200 before upstream status arrives; a real upstream 401 surfaces later as an SSE relay_upstream_error whose numeric status never leaves the response_start control frame. Both report call sites gated on response.status === 401, so on the default transport a vault credential could be rejected forever without one report_auth_failure — no invalidation, no version fence advance, no migration. The transform now parses the upstream status out of the relay error event and both call sites report vault-served 401s from that path, provenance fences intact (a relay 401 on a non-vault route still reports nothing).

min_ttl_ms was configured but consulted nowhere. The cache served until literal expiry, then went cold for exactly one turn while the detached warm caught up. A credential entering the min-TTL window is now marked stale — a detached credential.get fires under the existing per-handle backoff — while the still-unexpired token keeps serving. The maintainer's exact red numbers (min_ttl 120000, expiry 121000, now→2000: no second get) go green.

The status modal shows custody state now (a gap live testing found)

First live opt-in, first operator question: "I don't see anything." Correct — the claustrum status existed only in core's text output; the OpenCode modal renders the structured RPC projection, and the field-by-field allowlist (which exists so new fields fail closed) had never been given the fields. The projection now carries claustrumGate: on|off|na and vaultServed per account plus a modal-level detection line, threaded through the closed RPC contract, rendered in the dialog, covered by the blindness suite (sentinel handle asserted absent from payload and props), and pinned by a projection test through the real buildDialogPayload path — mutation-proved, since a test that survives deleting the field from the projection isn't testing the projection.

The idle-expiry incident, and who owns freshness now

The first live opt-in found the design's real gap within five hours. The account gets zero traffic, and the min-TTL refresh triggers on get() — so nothing ever called the vault, the credential expired resident, the peek-based vault-served predicate read that as "not vault-served", and the background timer ran a local OAuth refresh ten seconds after expiry. That rotation superseded the vault's stored refresh token; the vault's first real refresh attempt died invalid_grant and latched. Custody inverted itself on the first idle expiry, and the sidebar showed "needs relogin" for an account whose sidecar was perfectly healthy.

The fix flips background freshness ownership. For a vault-enabled account (a config property — gate plus handle — deliberately not the peek predicate), the background tick issues a periodic vault credential.get with minTtlMs derived as tick + margin (240min + 30min), so every tick either serves fresh or triggers the vault's own upstream refresh inside its window — the vault-seat's rule: with the 120s default, an idle token expires between ticks and the incident reproduces. Local refresh survives strictly as a last resort (vault hard-unreachable with the sidecar expiring, or a latched vault copy with a healthy sidecar), always with a loud custody-override log; latched handles back off at tick cadence so a vault repair stays discoverable without hammering the 64/60s limiter; and the sidebar now distinguishes vaultReauth (vault copy needs re-import) from needsReauth (credential actually dead) — the exact misreport the incident surfaced. The zero-traffic expiry scenario is pinned as a red test whose ownership-revert mutation reproduces the incident's local-refresh signature, and startup no longer double-probes the vault (warmup and the first custody tick raced).

Cross-family review (GLM 5.3, adversarial: independent mutations in a throwaway worktree, including a margin-off-by-one that reddens the min-TTL assertion) returned APPROVE with one finding — the reauth-backoff gap — fixed and re-approved 0/0.

Two more incidents, both second-occurrence failures

Idle-expiry was the design gap; these two were the resident client's state, and both are the same shape as the route-arm bug below — correct line by line, wrong on the second time through.

The client wedged, then retried a superseded token forever. When the resident subc client hit an unrecovered terminal SubcCallError, custody correctly fell back to local — but then retried the superseded sidecar refresh token every 90s, looping 400 invalid_grant. The daemon had never seen a socket disconnect (pid continuous), so the wedge was internal resident-client state, not a transport drop. Fix: reconnect the client on terminal SubcCallError with a 60s backoff, and latch the permanent custody error through refreshBackoffActive so the loop can't re-arm. The GLM review caught that the first latch test didn't actually discriminate the halt — connector tick 2 recovered — so the fixture now fails through get #3.

Booting gate-off, then enabling mid-session, dropped straight to the local ladder. With the custody gate off at boot, claustrumCredentialCache stays null; flipping the gate on mid-session bypassed the vault get and fell into local refresh — the same invalid_grant loop, reintroduced by a different door. Fix: the custody tick connects the client on demand when gate and handle are active, not only at boot.

And a multi-writer clobber underneath both. Concurrent state-file writes could roll back lastRefreshError, un-latching an account that had just latched. The persistence boundary that actually matters is mergeAccountRuntimeState — not the higher-level save — so the fence lives there: an older writer can't overwrite a newer latch, with checkedAt error-vs-error recency deciding ties. This one took two blocked review rounds: the lifted test probes kept making the test writer 1ms newer than persisted state, so the fence looked broken when the test was; a savant trace pinned the real boundary.

When the operator later saw post-restart cache busts across sessions, I byte-diffed the request dumps and traced them to a magic-context memory injection plus a history-compaction fold — the plugin build was exonerated.

Soak: three self-sustaining vault cycles

The custody design is now proven on the exact failure mode that started it, confirmed against the vault's own audit chain (authoritative side, not our logs):

seq 813  refresh_commit  actor=vault  v2 → v3
seq 822  refresh_commit  actor=vault  v3 → v4   17:25:58Z
seq 830  refresh_commit  actor=vault  v4 → v5   20:56:04Z

Three consecutive vault-owned refresh cycles, zero interventions. The third signs its own correctness: v4's ~8h TTL expires ~01:26Z, minus the 270-min minTtlMs window = opens ~20:56Z, and the commit landed four seconds into that window — the tick phase and the min-TTL rule meshing exactly as designed. Zero auth_events, zero FetchRateAnomaly, zero reports, zero vault-wide alarms since gate-on. Idle-account custody — the case that inverted itself and killed the first live opt-in — is now the cleanest-running case: no traffic needed, the periodic tick alone keeps the family alive.

A vault outage must never be worse than having no vault

That constraint decided several things that would otherwise look inconsistent.

The vault owns refresh for an account it serves, so we don't spend that account's sidecar refresh token — it may be a rotated lineage, and spending it returns invalid_grant, which our classifier marks permanent. But a cold cache, a warm in flight, and a failed connection all still permit ordinary local OAuth recovery, and a credential that later becomes resident clears whatever error that recovery persisted.

The clearing side of that had a defect worth describing, because it isn't a race in the usual sense. Clearing was wired to the cold→warm transition, and the warm only runs when a peek misses — so once the cache was warm, nothing could ever reach the clear again. A permanent error landing after the transition was unclearable by construction. A race you can lose repeatedly is self-correcting on the next attempt; this one had no next attempt.

What that costs is latent rather than immediate, and I got it wrong the first time I described it. A vault-served account keeps routing normally with a stale error on it — the refresh is skipped while the credential is resident, so nothing ever reads the error. It arms when the vault stops covering it: on an outage, a revoked handle, or a disabled gate, the account falls back to the local path and the stale permanent error excludes it. The failure surfaces exactly when the sidecar fallback is the only thing left. Clearing now happens whenever a usable vault credential is actually served, gated cheaply so it doesn't touch the common path and dispatched detached so nothing serving a response awaits a locked write.

Same shape as the 401-retry arm that didn't update the route its two siblings updated, and the reconnect/gate-flip pair above. All read correctly line by line. All are only visible if you ask what happens on the second occurrence.

Rate limiting is a correctness constraint here, not a nicety

The vault's limiter is 64 requests per 60s per connection, shared between get and report, and it increments before handle resolution — so a timed-out get still counts. Crossing it doesn't deny service; it writes a durable anomaly record.

Two paths could issue one credential.get per request indefinitely, both found by independent reviewers a layer apart: transport failures and terminal SubcCallErrors bypassed classification, so the per-handle backoff never engaged. Both classify now. Reverting either gives 13 gets for 12 requests, which is the signature.

Terminal needed care rather than a blanket reclassification. route.open failed is terminal and is genuine vault-side unavailability, which must back off — but missing_identity is also terminal, and that's the dead-on-arrival bug above, findable only because it threw loudly. So they back off and log the raw code. Backoff protects the limiter, logging protects debuggability; they were never in conflict.

Live custody evidence (one fallback account, opted in on this machine)

Each leg evidenced by the side that can see it: vault chain rows for import + handle mint; a fresh-process credential.get served version 1 with expiry matching the import; cache residency observable only via successful decode (peek cold→resident); the running plugin's own warm proven by vault served rendering in the modal (that flag is a live in-process cache.peek); and local refresh inert — the vault-served predicate gates all three refresh paths and the log shows zero refresh attempts for the account. The vault side confirmed clean non-refresh serves and, over a multi-hour soak, three consecutive vault-owned refresh cycles (above) with no anomaly records — and reviewed the relay-401 and min-TTL commits against their limiter and version-fence contracts.

Verification

full gate         1448 pass / 0 fail   (#174 head is 1366, so +82)
e2e               29 pass / 0 fail
typecheck         clean
format:check      clean
blocked-URL audit 0 vendor hits
live probe        reaches the vault, returns not_found
live custody      import → mint → get v1 → resident cache → modal renders vault-served
live soak         3 vault-owned refresh cycles (seq 813 → 822 → 830), zero anomalies

The audit line is why this is stacked rather than parallel. #174 makes a live network call from a test a hard failure; these tests inherit that rather than being exempt from it. The inheritance is not hypothetical: the moment #174's fetch-leak invariant met this branch, it flagged this branch's entire vault describe — 25 tests sharing one un-restored fetch mock — which now owns its teardown like its siblings. They run against an injected connector and a fake daemon — no test here touches the real vault, which is live on this machine and serving other consumers.

Mutations, since a guard that has never been broken isn't evidence of anything:

remove the production identity     identity test fails: expected identity, got undefined
remove consumerIdentity: null      consumer_identity expected false, received true
remove the transport classifier    13 gets for 12 requests
revert the quota token precedence  sticky quota test fails
neuter the steady-state clear      late-error clear test fails
restore the naive whole-storage clear   both "does not roll back" race tests fail
infer provenance from account state     2 tests fail, including the stale-resident one
widen the dump input to accept an account   typecheck fails on the structural contract
log the handle on report failure again      logger blindness test fails on the sentinel
drop the relay-401 report wiring            maintainer-shape repro observes only credential.get
serve through the min-TTL window silently   the exact-numbers regression fails (no second get)
remove claustrumGate from the projection    real-path modal test fails by name
reconnect removed on terminal SubcCallError  latch test fails through get #3
revert the mergeAccountRuntimeState fence    both writer-rollback race tests fail

Not included

A warm-skip optimisation using the vault's stale_pending field, which would let startup skip accounts it knows would be slow rather than discovering them by timeout. The field shipped this week and I have the wire shape, but the bounded warm handles that case correctly without it, and I'd rather land the mechanism before the optimisation. If it goes in later it should skip only on an explicit true — absent means not-observed, not false.