Support AGENT_SECRETS_KEY rotation with a keyring and rewrap command

Closed
AR
5d ago
+1414-66

Summary

AGENT_SECRETS_KEY could not be rotated: the cipher only knew version 1, so changing the value made every stored integration credential undecryptable. This adds a keyring format in the same variable and a command that re-encrypts stored values.

Key format (same env var, no new config):

  • Single base64 key (current format): version 1.
  • Keyring: "<version>:<base64>,<version>:<base64>". The first entry encrypts new values; the rest only decrypt.

Validation rejects empty values, non-integer or out-of-range versions, duplicate versions, malformed entries, and keys that do not decode to 32 bytes. Error messages name the entry position and version, never the key. The parsed keyring keeps key bytes in a private field so logging or serializing it prints no key material.

Encryption/decryption: encryptIntegrationSecret writes with the current version; decryptIntegrationSecret uses the row's recorded version. Existing version-1 rows keep working under both formats.

Validation in every caller: places that only checked that the variable was set (system capabilities, GCP config, Render routes, the Render and GCP worker jobs) now validate it through the shared parser. Env-object callers use isValidIntegrationSecretKeyring(raw), which takes no process.env fallback. The read-side ?? 1 fallbacks for envelopes without a recorded version now use LEGACY_INTEGRATION_SECRET_KEY_VERSION. Every write site already stores cipher.keyVersion, so the schema .default(1) values are never what a write relies on.

Rewrap command: pnpm --filter @superlog/db credentials:rewrap [--dry-run] [--batch-size=N]

  • integration-secret-columns.ts lists all 28 encrypted columns across 17 tables. That includes nullable envelopes and the Sentry columns that store the nonce as base64 text.
  • For each column it scans rows not on the current version in id order, including ciphertexts with no recorded version, which are treated as version 1. It decrypts each row, re-encrypts it with the current key, and updates with a guard on id, the old key version, the old ciphertext and the old nonce. If another writer changed the row in the meantime, the update does nothing and is counted as a conflict.
  • --dry-run decrypts every pending row without writing, so rows that would fail show up before any old key is removed.
  • Output is one JSON line per column plus a summary line, with no plaintext or key material. It exits non-zero if any row fails or, after a real run, if any row is still on an old version.
  • It can be re-run safely: rows already on the current version are skipped.

Rotation sequence for operators

  1. Optional: deploy 1:<old>,2:<new> first if you need zero decrypt errors during a rolling deploy.
  2. Deploy 2:<new>,1:<old> to every service that reads the key.
  3. Run credentials:rewrap --dry-run, then credentials:rewrap. The run must finish with remaining: 0 and failed: 0.
  4. Deploy 2:<new>.

Tests

  • integration-secrets.test.ts: keyring parsing and validation, no key material in errors or inspect/JSON output, encrypting with the current version and decrypting old versions, legacy single key reading keyring-written v1 rows, a clear error when a version is missing, and a wrong key under a version number failing authentication.
  • integration-secret-rewrap.test.ts (PGlite):
    • a registry-completeness test that fails for any *_key_version column in the schema that is not in the registry;
    • registry shape checks (same table, primary key, bytea/text nonce matches encoding);
    • v1 to v2 rewrap with dry-run, then a real run, then an idempotent re-run, with everything readable afterwards under the new key alone;
    • undecryptable rows reported and left untouched;
    • versions missing from the keyring reported;
    • a concurrent write between read and update winning over the stale re-encryption;
    • argument parsing.
  • Capability, GCP config and GCP worker-job tests for keyring and malformed-key handling.

Validation run locally:

  • pnpm --filter @superlog/db test (273 pass)
  • the api suite against a scratch Postgres (all pass)
  • the affected worker tests
  • pnpm -r typecheck
  • a CLI smoke test against Postgres via postgres-js: dry-run, then run, then an idempotent re-run, a check that values decrypt under the new key alone, and a non-zero exit when a version is missing

Summary by cubic

Supports rotating AGENT_SECRETS_KEY: the variable now accepts a keyring format and a rewrap command re-encrypts stored credentials. Previously the cipher only knew version 1, so changing the key made every stored integration credential undecryptable.

AGENT_SECRETS_KEY now accepts "<version>:<base64>,<version>:<base64>" in addition to the single-key format (still version 1). The first entry encrypts new values; later entries only decrypt. Decryption follows each row's recorded version, so existing version-1 rows keep working, and callers that only checked the variable was set now validate it. Invalid keys are rejected without echoing key material.

Rewrap command

  • pnpm --filter @superlog/db credentials:rewrap [--dry-run] [--batch-size=N] re-encrypts every envelope not on the current version across all 28 encrypted columns.
  • Updates are guarded on id, old version, and old ciphertext so concurrent writes win; the command is idempotent and exits non-zero on failures or leftover rows.

Migration

  • Optionally deploy 1:<old>,2:<new> first to avoid decrypt errors during a rolling deploy.
  • Deploy 2:<new>,1:<old> everywhere, run a dry-run then a real rewrap, and confirm remaining: 0 and failed: 0.
  • Deploy 2:<new> and drop the old entry.