chore: proposal - opt-in end-to-end test control surface for local stacks and pull request previews

Draft
TH
16h ago
+2084-14

Proposal for discussion. Draft, not for merge as it stands. This revision takes the opt-in, local-only surface one step further: it can also exist on a pull request preview that opted in. It cannot exist on dev or production, and nothing here turns it on anywhere: a preview needs an infrastructure change and a label that do not exist yet.

Summary

server/internal/e2econtrol is four routes an end-to-end test run uses to put one test organization in a known state and to sign a browser in to it without driving an identity provider's pages.

RouteAuthorised byDoes
GET /rpc/e2e.tenantGram-E2E-Token headerNames the test organization (organization slug, project slug, seeded user id prefix) and the build the server was compiled from. Reads only
POST /rpc/e2e.reseedheaderRestores the test organization's seeded state with the product's seed. Takes no input. 204, or 409 while another seed or reseed is running
POST /rpc/e2e.signInLinkheaderMints a one-time path that signs a browser in as the test user. Single use, 60 seconds
GET /rpc/e2e.signIn?code=…the one-time codeSigns in and redirects to the organization's home

No route takes an organization, a user or a destination from its caller, and no route calls an identity provider. All three are fixed when the server starts, by where the server is:

A local stackA pull request preview
Admitted whenGRAM_ENVIRONMENT is exactly localGRAM_ENVIRONMENT is exactly dev, the database the server is connected to reports its own name as gram_preview_pr_<N>, and the dashboard and server URLs are one host whose first label is pr-<N>
Test organizationThe local development organization (demoseed.LocalSpec)A synthetic organization that exists only in that preview's database, named after N (demoseed.PreviewSpec), with no identity provider organization behind it
Test userThe developer the local seed adopts (git config user.email), a platform admin as on any local stackA synthetic member of that organization alone, with the Admin role, never platform staff, on a reserved address that receives no mail and that no identity provider knows
Reseeddemoseed.Run then RunLocalFixtures, as mise run seed doesdemoseed.Run with no ClickHouse connection, then RunPreviewFixtures: Postgres only
SessionAn ordinary dashboard sessionThe same, confined: it cannot be moved to another organization, and every reseed ends the sessions issued before it

With the token set and neither column satisfied, the API server refuses to start. With no token (the default everywhere) nothing is constructed and the routes do not exist. Only the API server accepts the flag.

Other changes this needs:

  • demoseed.Run skips the whole ClickHouse phase, deletes included, when given no connection. Existing callers pass one and are unchanged.
  • demoseed.PreviewSpec(n) and demoseed.RunPreviewFixtures: the organization's role grants, the grants and settings the local organization also gets, and the test user's membership written last, in one transaction. It refuses a database without the built-in roles instead of creating them, makes nobody staff, creates no API key and writes no registry or catalog rows.
  • sessions.Manager.Issue stores a session then writes its cookie. The invitation callback now calls it instead of its own copy of those two steps; its order and behaviour are otherwise unchanged.
  • sessions.Manager.ConfineUser: UpdateSession, the one place auth.switchScopes and auth.enterDemo change a session's organization, refuses to move a confined user's session. Only the preview's test user is ever confined.
  • server/internal/e2econtrol/openapi.yaml is a small hand-written contract a test tool pins by digest. It now holds the three token routes only: the two public reads the first revision projected are gone, so a test run holds one credential and no API key.

Motivation

The Reflow golden flows depend on two shared flows that only work against a stack booted in the same job: the suite's root checks the seed with direct SQL, and the shared sign-in clicks through the dev-idp. Going through the product instead (reseed through a route, sign in through a one-time link) lets the same flows run against the build a pull request actually deploys. A preview is where that matters most: it runs the production dashboard build and real feature flags.

Why the preview's tenant is synthetic and local to its database, rather than a test organization and user created in the identity provider:

  • It needs nothing created by hand, in the identity provider or in dev's data.
  • Previews share a cache with each other and with dev. With one shared test user, a session minted on a preview would also be a working session for the same user in dev and in every other preview. With per-preview ids, the session resolves to a user and organization that exist in one database only.
  • The token then causes no call outside the preview's own database and cache, and nothing is written to the ClickHouse previews share.

What a preview gives up: telemetry pages show the organization's empty state, and there is no real identity provider sign-in in the loop (login-smoke covers that locally).

Threat model

The token is a secret given to the API server at startup and to the test runner. It is compared in constant time. It carries no individual attribution.

Local stack: the developer or CI job generates the token for its own stack; nothing sets one by default.

A holder of the local token canA holder of the local token cannot
Reset the developer's writable local organization: all of its projects, API keys, environments and memberships are deleted and reseeded, including anything the developer created there. The fixtures restore the developer's membership and the seeded API key, and write the global rows mise run seed writesSelect any other organization as the reset target
Mint, as often as it likes, a single-use 60-second code for an ordinary dashboard session as the local developer, a platform admin on a local stackSign in as anyone else, or choose where the browser is sent

Preview: the token is one secret per environment, handed only to previews that opted in.

A holder of the preview token canA holder of the preview token cannot
Reset the synthetic test organization of any preview that opted in. Only rows of that organization, in that preview's own database, are deleted and rewritten, plus the test user's own rowReset or read any organization cloned from dev, in any preview. The seed's own safety checks (TestDemoSeedSafety) cover tenant isolation, and a test here runs the reset beside another tenant
Sign in to that organization as its test user: organization Admin, not platform staff, a member of nothing elseObtain a staff session, or a session with a second organization: the user is read from the database when the link is redeemed and refused if either is true
Use that session until the next reset, which ends every session issued before it, or until it has been idle for 72 hoursMove the session to another organization or the shared demo organization (ConfineUser), or use it on dev or another preview, where its user and organization do not exist
Reach dev or production through this surface at all: neither is connected to a preview database, so the server there does not start with a token

Stated residuals, none of which this change tries to engineer away:

  • One token serves every opted-in preview of an environment. Preview pods run pull request code and already hold the environment's service credentials, so a token per preview would not narrow who can read it; it would only narrow what a copy leaked from a test runner reaches, which is synthetic tenants.
  • Confinement is held by the API process that has the surface on. If the token were removed and the server restarted against the same preview database while a test session was still alive, that session could enter the preview's copy of the demo organization, and a demo-organization session is honoured wherever the cache is shared. That is the access any signed-up user already has. Giving an opted-in preview its own session cache namespace (the cache.Suffix the session manager already takes) would close it; it is left out because it changes session wiring for every user of that preview.
  • A confined session that tries to leave gets a 500 from auth.switchScopes or auth.enterDemo, not a 403: the refusal is made where the session is written, and those handlers were left untouched.
  • The reset repairs the test organization and the test user's own row, not everything a person could do to the preview's database by hand. A test user promoted to staff, given a second membership, or whose organization was disabled is refused at sign-in rather than repaired, and the cached copy of the user is dropped at every sign-in so a session is judged by the rows the sign-in read.
  • The reset is not atomic and the cross-process seed lock covers the seed but not the fixture transaction after it. One API process runs one reset at a time; two pods overlapping during a rollout could interleave, and the next reset converges.
  • Like every preview, this one signs chat-session tokens with its environment's key, so a token minted for the synthetic organization verifies on hosts that share the key. Previews already hold copies of the environment's real organizations; a per-preview signing key would be the fix for all of them and is not attempted here.
  • The seed's OpenAPI document is written to the shared assets bucket under the organization's slug, so each preview that resets leaves one small object there. The write is create-if-absent, as it is for the demo and local seeds, so a later build that changes the document keeps serving the first bytes.

A sign-in code is never stored: the cache holds a boolean marker under a hash of the code together with the dashboard origin, the organization id and the user's address it was minted for, so a server configured for anything else (another preview sharing the cache) does not know it. It is redeemed with an atomic get-and-delete and carried in a query parameter the request log already redacts.

What the sign-in skips compared with a normal login: the identity provider entirely, so there is no provider session and nothing to record (signins.RecordExchange). Because of that it gives no session where the organization has sign-in enforcement on.

The startup guard

attachE2EControl (server/cmd/gram/deps_e2e.go) makes the one decision, before anything is mounted. The preview check reads current_database() on the pool the reset will write through: the name is the fact that makes a reset contained, it is produced by the preview's own database setup, and dev and production are connected to a database without a pull request number, so they refuse whatever else they are given. Tests drive the decision against a real mux for each environment, database name and host combination, and check that only the API server declares the flag.

A local stack is admitted by its environment name. A deployed image mislabelled local still refuses to start with a token: local mode finds its test user through git config user.email, and the image has neither git nor a git identity. A developer's own binary pointed at a hosted database in local mode is not something this guard can tell from a local stack.

A build tag was considered and rejected: the image that reaches production is the one promoted from dev, so the same binary has to run in both.

What changed since the local-only revision, and why

  • Preview admitted, as above. Dev itself stays refused.
  • One sign-in path, with no identity provider in it. The local-only revision asked the dev-idp emulator for a Magic Auth code and ran the login bootstrap with membership sync skipped. Both modes now read the fixed test user from the database and issue a session. That removed the identity interface, the returned-identity comparison and both emulator-only startup conditions: there is no identity endpoint left for this surface to be pointed at by mistake.
  • No API key. The suite's root no longer reads the public API; a 204 from the reseed means every phase of the seed, its own postflight checks and the member fixtures completed, and the flows that depend on the root assert the seeded state in the dashboard.
  • Session issuance shared with the invitation callback (sessions.Manager.Issue), which the earlier review asked for and was deferred.

Not proven, and what a preview still needs

  • Passing on this commit: the unit tests for every guard, token check, single use, expiry, binding, confinement and the startup decision; a test that mounts the surface as a preview against real Postgres and Redis and goes through reset, sign-in, the confined session and the next reset beside another tenant; the fixture tests under the demoseed_safety tag. On a local stack the root and a golden-shaped flow pass through the routes (reseed, sign-in through a link, a seeded page opened).
  • Nothing here has run on a real preview. A preview only gets the token from infrastructure configuration, which is a separate internal change: a secret, a chart value that reaches the API server container only and refuses to render outside a preview, and a preview-e2e label that opts a pull request in. Until that is reviewed, merged and applied, the surface is off on every preview, including this pull request's.
  • To verify on the first real preview: that the application database role can run the seed (it runs no DDL); the reseed's duration with the preview's two-connection pool; that the dashboard is usable for the test user under real feature flags.
  • Dev is out of scope. It would need its own explicit admission, its own secret, test tenant ids distinct from every preview's, the seed lock extended through the fixtures, and a decision on whether its test user should be backed by the identity provider.