Renew an event stream in band before its token expires

Merged
JE
19d ago
+3988-68

bc3 is about to bound an agent's ticket stream by its token's expiry. The welcome advertises the deadline, and at that deadline the server closes the socket with remote unless the client has already sent a fresh ticket over the open socket. Today the connector can only take that close and reconnect, so a held agent stream churns once an hour. This PR teaches the connector to renew in band, and turns renewal into SPEC §23 contract.

What the connector does

  • Only where advertised. A socket is renewed only when its first welcome carries a usable stream_expires_at. That rule applies socket by socket, so a mixed fleet is safe, and a bc3 revert simply stops the advert. An unusable advert (null, a number, an unparseable string) counts as no advert. It is never treated as an invalid frame, because not renewing is always safe, while a fatal advert would send every client back into the same welcome over and over.

  • Schedule. renew-due fires at the deadline minus 3 min, ±30 s, measured against the Clock. That sits inside the CLI's 5-minute lazy token refresh, so the renewal's mint carries a token whose expiry is later than the one the stream was opened with. The renewal after a successful one is never sooner than 30 s, which is the server's one-renewal-per-30-s floor.

  • One attempt. When renew-due fires, the connector arms renew-ack-deadline (10 s), mints one ticket through TicketMinter, writes {"command":"renew","identifier":<the subscription's own, byte-identical>,"ticket":<fresh>}, and waits for renewed or renew_refused. The stream keeps delivering throughout.

  • Retry once, then fall back. Each of these counts as a failure and is retried once, 30 s later:

    • renew_refused
    • renewed with no usable later deadline
    • a mint that fails in any way, or is still outstanding at the deadline
    • no reply by the deadline

    A second failure gives that deadline up. The socket then runs to the lease, and the existing remote → backoff → re-mint → catch-up path recovers. Nothing in renewal is terminal. Nothing touches the shared authorization counter, the position or the checkpoint.

  • The write belongs to the socket. A failed renew write, or one still outstanding at the deadline, takes transition 25.

  • The exchange lives in Streaming. renew-due is stopped across a repair walk and re-armed afterwards for whatever remains of its target. The repair cadence is held while an attempt is in flight, so CatchingUp and Draining keep their timer sets and the connector only ever waits for a reply from Streaming.

  • Which replies count. A reply belongs to the attempt once that attempt's write has begun, since the server can answer before the client sees its own write complete. At any other time a reply is just liveness, and a late renewed is not adopted.

  • Old servers. The command carries the subscription's identifier. A server that won't take it, either because it predates renewal or because the stream isn't renewable, correlates it and answers reject_subscription on that identifier. A confirmed stream reads that as the renewal refused, never as the subscription rejected (bc3 #14143's documented reading). An uncorrelatable command would instead get the protocol-fatal disconnect.

  • Replies are correlated. Replies arrive on the subscription's identifier, and one on any other identifier answers nothing.

  • The ticket stays out of logs. The command carries a bearer credential and is written to the socket only. RenewFailed reports a closed kind (refused | no_reply | mint_failed | invalid_reply), never the server's reason.

New observer callbacks: StreamDeadline(expiresAt), Renewed(expiresAt) and RenewFailed(kind, retrying). There is no new option.

Wire literals: provisional, reconciled with bc3 #14143

The literals were drafted from bc3's renewal design (decision of 2026-10-26), then reconciled against bc3 #14143's open head (553b7e72):

  • The welcome's stream_expires_at is iso8601(3), which matches.
  • {"command":"renew","identifier","ticket"} matches.
  • The replies carry the subscription's identifier. Adopted in c188ddd9.
  • A reject_subscription answering a renew means the renewal was refused. Adopted in c188ddd9.

They stay provisional until #14143 merges (SPEC §23 Provenance; the fixture README's dependency table).

SPEC §23 and the tier-2 family

  • New In-Band Renewal section.
  • Two timer kinds, renew-due and renew-ack-deadline, bringing the total to eight.
  • Streaming's per-state timer set extended.
  • The renewal self-loop listed among the edges outside the 26 rows. The inventory is still 11/26.
  • The Clock's one absolute comparison, the welcome advert in Cable Protocol Details, the observer callbacks, the security invariant, constants, and Appendix A rows.
  • Schema:
    • expectRenew, which pins the ticket and requires the identifier to match byte for byte.
    • serve welcome gains streamExpiresInMs.
    • New serve renewed and serve renew_refused.
    • The two new timer kinds.
    • Deadlines are offsets from the driver clock's reading at the moment the frame is served, which keeps scripts portable across test clocks.

Fixtures:

#Proves
36Two renewals on one socket with no reconnect, with renew-due re-armed for its remainder after a repair walk
37A refusal is retried once, then the lease's remote close is recovered
38A reject_subscription answer is the renewal refused, not the subscription, and the retry succeeds
39No usable advert means no renewal
40Tearing down mid-renewal leaves exactly {backoff}
41Unauthorized renewal mints never count toward the shared authorization counter

Verification

  • Fail-first. I ran the new fixtures against origin/main's connector with the new driver: 36, 37, 38, 40 and 41 fail at their first expectTimers (no renew-due). 39 passes on main by construction, because main never renews. It is a negative-space pin, and rows 17, 18 and 28 below show it has teeth.
  • Mutation kill matrix. I applied each of the 13 new mutants (README rows 17–29) to a scratch copy and ran the named fixture 5 times. All 13 were red on 5 of 5 runs at the final head. Row 29 is ignore-a-renewals-reject, killed by fixture 38. The table below is from the first run:
RowMutantFixtureRed on
17renew-without-advert39step 9: renew-due in the set
18fatal-unusable-advert39step 4: socket torn down
19renew-with-the-socket's-own-ticket36step 21: ticket-1 ≠ ticket-2
20renew-by-redialing36step 19: {backoff}
21retry-renewal-without-limit37step 20
22give-up-without-a-retry37step 15
23terminal-on-a-renewal's-reject38step 15
24leak-renewal-timers-on-teardown40step 16: renew-ack-deadline survives
25count-renewal-mints-toward-authorization41step 21: terminal, not backoff
26rearm-the-welcome's-delay-after-a-repair-walk36step 18: 57m26s ∉ [46m30s, 47m30s]
27keep-renew-due-armed-through-a-repair-walk36step 13
28adopt-an-unsolicited-renewed39step 13

Row 23 first survived against fixture 38: whether the ack deadline or the reject was dispatched first was a coin flip. Fixture 38 now serves a live event behind the reject and expects it delivered before the deadline fires.

  • Tier-3 tests (renew_test.go, cable_test.go):
    • the jitter formula under a fixed rand
    • every failure kind's observer record and the exact retry delay
    • both write-failure edges, plus the canary ticket never reaching the observer
    • a reply that arrives before its own write completes
    • the repair cadence held during an attempt
    • Close from each renewal callback
    • the 30 s floor
    • a late reply not adopted
    • a duplicate welcome advertising nothing
    • the remainder, not a redraw, after a walk
    • a renewal-mint panic propagating
    • lenient advert parsing, the reply frames, and the command's exact bytes
  • Driver self-tests: three new kill probes and two arrival-strict cases for expectRenew.
  • Checks run locally:
    • go test -race ./pkg/basecamp/eventfeed/... and golangci-lint (0 issues)
    • make event-feed-fixtures-check
    • make doc-constants-check
    • make check-orphaned-doc-comments

After review

The bot reviewers and five local review passes found that the select's coin flip could decide parts of an attempt. Four commits close that:

  • The deadline wins ties. When a reply, a minted ticket or a write result is ready at the same moment as a deadline that has already fired, the deadline decides, by the timer's Stop result as in the handshake and confirmation paths. A deadline that hasn't fired is re-armed for whatever remains of it.
  • An early reply waits for its write. A reply that arrives before its own write has reported back is parked until the write's outcome is in. A failed write is the socket's failure, and a write still blocked at the deadline is a socket that isn't reading.
  • Replies are matched by when they were read. The pump numbers frames in the order it reads them, and the writer records the cutoff just before it writes. A late answer to an earlier attempt that was already queued when the command went out can't answer the retry.
  • Seam calls on the deadline path are joined. The deadline cancels the mint, waits for it to return, and re-raises any panic; it does the same for a stalled write. Close and staleness are checked again afterwards.
  • Clock.Now must read wall time. This is now part of the contract, because the renewal schedule compares it with the server's deadline.

What's left is in SPEC §23. A previous attempt's answer read after the new command went out can't be told apart from the new one's, since the server doesn't correlate replies; the cost is one deadline, which the lease and reconnect recover. Teardown cancels a mint or write in flight without waiting for it, the same as a superseded poll call and a subscribe write. Each fix has a tier-3 test that fails on the head before it. The 12 kill-matrix mutants are still red on 5 of 5 runs.

Not covered here

  • The bc3 renewal PR (being built separately), the literal reconciliation above, and the release.
  • The CLI's SDK bump. It is described on the lane, not opened, and waits for this PR to merge.