feat(railway): circuit-break pull cycle on invalid_client credential rejection #SUPERLOG
Summary
Railway's OAuth server is currently returning {"error":"invalid_client"} for every refresh_token grant, indicating Superlog's Railway OAuth app credentials (RAILWAY_CLIENT_ID / RAILWAY_CLIENT_SECRET) have been rejected or revoked. This permanently stalls log/metric ingestion for all 12+ connected Railway installations and produces 12+ railway token refresh failed ERROR logs per minute (one per installation, every pull cycle).
Root cause
invalid_client is an application-level credential rejection — Railway refuses the OAuth app's client credentials, not the per-installation refresh token. Because all installations share the same client_id + client_secret, every installation fails identically on every tick. The puller previously treated invalid_client identically to invalid_grant (user-revoked per-installation access): it logged railway token refresh failed per installation and continued to the next one, hammering Railway's token endpoint 12+ times per minute with requests that will all fail.
Remediation
External action required (unblocks ingestion): Rotate the Railway OAuth application credentials in the Railway developer portal and update RAILWAY_CLIENT_ID / RAILWAY_CLIENT_SECRET in production secrets. This code change alone does not restore ingestion.
This PR (prevents token-endpoint hammering and improves signal clarity): Adds an appLevelOAuthError closure flag in runRailwayPullOnce. When the first invalid_client response is detected:
- The error is logged once with clear credential-rotation guidance instead of per-installation.
- The installation loop breaks immediately — no further token requests are issued for the remaining installations in that pull cycle.
This reduces Railway token-endpoint traffic from 12+ requests/minute to 1, and collapses the error signal from 12+ per cycle to 1 per cycle, making the failure immediately actionable.
Test
Added puller.test.ts case: two installations with expiring tokens, Railway token endpoint returns {"error":"invalid_client"}. Asserts token endpoint is hit exactly once, stats.errors === 1, rotation message in logs, and no intake traffic.
Incident: drifting-narwhal (e3d893f6-dc5d-40ad-9a78-321d178321ab)
Was this PR helpful? Leave feedback — goes straight to the Superlog team.
Summary by cubic
Circuit-breaks the Railway pull cycle when Railway rejects the OAuth application credentials (invalid_client) instead of treating it like a per-installation revocation and failing 12+ times per cycle.
Previously, invalid_client was handled like invalid_grant, so every installation hit the token endpoint with requests that all failed, producing 12+ railway token refresh failed errors per minute. Now the first invalid_client stops the loop: one error is logged with credential-rotation guidance, no further token requests are issued in that cycle, and token-endpoint traffic drops from 12+/minute to 1. Added a test asserting the token endpoint is hit exactly once and no intake traffic occurs.
Rollout action
Rotate the Railway OAuth app credentials in the developer portal and update RAILWAY_CLIENT_ID/RAILWAY_CLIENT_SECRET in production secrets — this code change alone does not restore ingestion.