Fix railway token refresh to warn and continue on transient failures #SUPERLOG

Closed
SU
2mo ago
+92-13

Summary

When Railway's OAuth token endpoint returns a transient error (invalid_response, server_error, etc.), the worker logged ERROR and immediately skipped the installation — even though the stored access token was still valid for up to 5 minutes. This caused noisy incidents and unnecessary pull gaps. A burst of 4 installations hit this at 2026-08-19T00:28:04Z; all recovered fully 4 minutes later with 0 errors.

Root cause

The token-refresh block in puller.ts unconditionally returned early on any !refreshed.ok result:

if (!refreshed.ok) {
  stats.errors += 1;
  deps.log.error(
    { installation_id: installation.id, error: refreshed.error },
    "railway token refresh failed",
  );
  return;
}

invalid_response is set by packages/railway/src/oauth.ts:83 when res.json() fails (Railway's token endpoint returned a non-JSON body). At this point accessToken is still the valid stored token — the refresh fires when expiresAt - now() < 5 minutes, not when the token is expired. The early return throws away the remaining valid window.

This is a recurrence of a previously investigated issue (incident yeasty-skunk / PR #440 which was never merged).

Fix

  • On a failed refresh, log WARN instead of ERROR.
  • Only skip the installation if the token is truly expired (expiresAt <= now()); otherwise continue the pull with the stored accessToken.
  • Move accessToken = refreshed.accessToken and saveTokens(...) into the else branch so they only run on success.
  • Two new tests verify both paths (transient failure with valid token → continues; failure with expired token → skips).

All 8 tests pass (pnpm tsx --test apps/worker/src/railway/puller.test.ts).

Fixes incident wandering-porcupine (f6bf27c2-ab30-4137-8038-c47c5cab5650).

Was this PR helpful? Leave feedback — goes straight to the Superlog team.

Summary by cubic

Gracefully handles transient Railway OAuth token refresh failures by warning and continuing with the stored access token until it actually expires. Previously, any refresh failure logged ERROR and skipped the installation even when the token was still valid, causing noisy incidents and pull gaps.

  • In apps/worker/src/railway/puller.ts, on failed refresh: if the token is expired, log ERROR and skip; if not expired, log WARN and continue using the stored token.
  • Assign accessToken and call saveTokens(...) only on successful refresh.
  • Adds two tests in apps/worker/src/railway/puller.test.ts covering both paths (transient failure with valid token continues; failure with expired token skips).