feat(knowledge): let the S3 connector sync with a saved AWS credential

Merged
WA
3d ago
+1483-182

Summary

  • The S3 knowledge connector can now sign with a saved AWS credential owned by the knowledge base's workspace or organization: access keys, an assumed role, or a keyless OIDC role. The STS session token is SigV4-signed as x-amz-security-token.

  • Key connectors can declare an additive auth.savedCredential alternative. S3 declares aws, the only provider that resolves today. Key-only config fields (apiKeyOnly: Access Key ID, custom endpoint) are hidden and not sent when a credential is used.

  • A saved credential signs only for Amazon S3 in a commercial or GovCloud AWS region, where every S3 host is s3.<region>.amazonaws.com. These are refused with a clear message:

    • a custom endpoint (R2/MinIO) or the auto region, so an AWS session never reaches a non-AWS host;
    • China and isolated-partition regions, whose S3 hosts are on other domains.

    R2/MinIO keep using access keys.

  • Resolution forwards the knowledge base's own scope to the same owner check and STS exchange the AWS tools use. Setup checks the acting principal with the existing connector credential check and a provider match. Create and access updates resolve the credential's service through one shared helper, so the provider match cannot be skipped. During a sync, STS sessions are reused until 5 minutes before they expire.

  • Validation errors are redacted against every secret the resolved token carries. For a saved AWS credential that means the secret key and session token, not just the empty bearer.

  • A connector that stores its key cannot be moved onto a saved credential later. To switch, create the connector again. This keeps a retired key from silently taking over if the credential is deleted.

  • Setup error codes:

    • When AWS refuses a credential, setup returns a 400.
    • When STS is throttled or the IdP is unreachable, setup returns a 500 instead of blaming the credential, matching how AWS tools return 503 for transient failures. The failure message and the transient check are shared with the AWS tools.

  • Audit metadata records these connectors as authMode: 'savedCredential'.

  • Setup is behind the global knowledge-connector-saved-credentials flag (env fallback KNOWLEDGE_CONNECTOR_SAVED_CREDENTIALS). Sync is not gated. The v2 catalog and credentialId descriptions say the option works once the deployment enables it.

  • The v2 catalog contract gains optional savedCredential / apiKeyOnly. OpenAPI, CLI API and CLI docs are regenerated, and the connector docs are updated.

  • Data drains are not changed. Org-owned saved AWS credentials, including keyless ones, already resolve. Drains still need three things first:

    • an additive credential reference on data_drains, deploy-gated because older workers parse destination_credentials directly;
    • org credential authorization at drain create and update;
    • new contract and UI surface.

    GCS, Azure Blob and Sentinel also need new resources in the shared Google keyless and Microsoft app resolvers (Cloud Storage, storage.azure.com, monitor.azure.com). These are left for a follow-up.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other

Testing

  • connectors/s3/s3.test.ts:
    • SigV4 Authorization matches @smithy/signature-v4 reference values, with and without a session token.
    • Covers refusal of a custom endpoint, a non-AWS region, and China/ISO regions, and signing for a GovCloud host.
    • Reverting the guards turns these tests red.
  • access-token.test.ts:
    • The knowledge base scope is forwarded and the session is seeded with an empty bearer token.
    • A stored key bypasses AWS; refusals are wrapped as non-transient and throttling as transient.
    • Run tokens are reused and renewed.
  • connector-access.test.ts: refuses moving a stored-key connector onto a saved credential, and allows moving a credential-backed one to another credential.
  • connectors.test.ts (application):
    • Refuses another service's credential on S3 create.
    • AWS refusal maps to 400 and throttling to 500 (the guard revert was watched turning red).
    • A saved AWS session's secret key and session token are redacted from settings-validation results and exceptions. This test was red before the fix.
  • orchestration/connectors.test.ts: the same redaction on connector create, for results and exceptions. This test was also red before the fix.
  • auth.test.ts covers the auth-source decisions; saved-credential-rollout.test.ts covers the setup gate.
  • The neighbouring knowledge, catalog, sync-engine, deletion and AWS-tools suites pass. These checks pass: check:openapi, check:cli-api, check:cli-docs, check:mcp-operations, check:deployment-config, check:file-names, check:test-patterns, check:comment-hygiene.

Deploy notes

  • Leave knowledge-connector-saved-credentials off until this release runs on both the app and Trigger.dev workers. Older workers unschedule a key connector that has no stored key.
  • Once any connector uses a saved credential, rolling back to an earlier release is unsafe even with the flag off. Older workers unschedule those connectors as credential-removed, and they stay idle until someone edits them or syncs them manually.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • Backward compatible: existing S3 rows keep their stored key; stored secret shapes unchanged