Run the connector's end-to-end scenarios against a local Basecamp too

Closed
RO
14h ago
+981-124

PR 6 of a stack of 8 that tests basecamp connect end to end. Stacked on #826 (the failure scenarios), which is stacked on #823 (the harness) and #822 (the fake Basecamp). It changes no product code: only e2e/connect, the Makefile and CONTRIBUTING.md.

The dev target was not run against a real Basecamp in this PR. No BC3 was reachable where it was written. It compiles, its environment parsing and helpers have unit tests, and each CLI command it runs was checked against the command code. How to run it is below.

Why

The scenarios in e2e/connect ran only against the fake Basecamp. A pass there shows that the connector agrees with the fake. It does not show that the fake agrees with Basecamp. If both run the same scenarios and both pass, the fake is held to Basecamp's behavior. Then a fake-only fault scenario means something.

What

  • Target (e2e/connect/target_test.go) says what a scenario needs in the scenario's words: CLI() (the harness that runs commands against this Basecamp), Agent() (profile, person id, account), OperatorMentionsAgent(t) and UntrustedMentionsAgent(t). Both mention methods return Posted, the comment as Basecamp holds it: recording id, parent (where reply_to points), bucket and project name, type, title, app_url, author, and the content with the agent's mention replaced by a space. The scenarios check the request line against it field by field, so the same assertions run on either target. They find their lines by recording id and comment.created, because the event id is not known before the connector sees it on a real Basecamp.
  • fakeTarget is the earlier behavior: the default world plus a bystander, setup through the CLI, a comment put in the world and published live.
  • devTarget drives a local Basecamp with the CLI under test:
    • It uses the developer's own environment, config and credentials (keyring included), less BASECAMP_TOKEN, BASECAMP_PROFILE and a default account, project or to-do list. Commands run in a new temporary directory, so no repository's .basecamp/config.json is read.
    • It never runs auth agent connect or connect setup, because setup rewrites the developer's connect.json. It reads the setup with basecamp connect show -P <agent> --json and requires operator trust (setup's default) and the project served.
    • Connector state. Each scenario's connector gets a temporary XDG_STATE_HOME. The ledger and the per-agent instance lock are there, so a scenario cannot read or move the developer's ledger and cannot hold their lock. That also means the lock cannot keep a scenario's connector apart from the developer's real one. So it first runs basecamp connect status -P <agent> --json against the developer's own state (status takes no lock). If a connector holds the lock, it stops with an error, because that connector would get the scenario's mentions too.
    • Each scenario posts a new message as the operator (messages create --project <id> --no-subscribe), and checks that the author is the person connect.json trusts. It comments with comments create <message> "... [@agent](person:<agent id>)? Thanks". The CLI's Markdown resolves that to a bc-attachment mention, and agents resolve by id. Every command gets -P <profile> --account <agent's account> --json. A cleanup runs messages trash on the message, with its comments. The expected content comes from the comment Basecamp returned. A regex removes its one mention element, so the check does not use the connector's own stripping code.
    • It refuses a BASECAMP_BASE_URL that is not localhost, a loopback address or *.localhost (hostutil.IsLocalhost), because it posts as the developer's profiles. The SDK accepts ws:// cables on the same hosts.
    • Dev scenarios run one at a time behind a mutex: they share one agent, one credential and one project.
  • Scenarios on both targets: a trusted mention handed off in one request line, an untrusted mention discarded as untrusted_performer with nothing on the recording handed off, stdout only NDJSON with stderr saying it runs, SIGINT exits 130, a second connector for the same agent exits 5 at once, and a request admitted under --hold handed off after kill -9, connect release and a restart. In the second-connector scenario, the check that the refused process asked Basecamp for nothing stays on the fake: only the fake sees every request.
  • Fake-only, marked with fakeOnly(t, why): both lanes published apart, the cable down during a commit, a stale cable, a rate-limited token, failed reads, held reads (backlog), a disconnected agent (two scenarios), an agent connected on another computer, two agents side by side, and connect setup itself. When the dev target is requested they skip with their reason, so the run lists what it did not cover.
  • make test-connect-dev checks the five variables like record-cassettes and fails (it does not skip) with the missing one's name. Then it runs BASECAMP_CONNECT_DEV=1 go test -count=1 -failfast -v ./e2e/connect. It builds without -tags dev, as a release builds. Run directly with BASECAMP_CONNECT_DEV=1, the Go side fails too, and names every missing variable at once. A plain go test ./... runs only the fake. The help list has the target.
  • CONTRIBUTING.md: a section "Running basecamp connect against a local Basecamp" with prerequisites, command, what it does to your state, what it covers, and what stays fake-only.
  • Harness gains Dir (where commands run) beside Home, and an internal run(ctx, ...) so a cleanup can run a command after the test's context has ended.

How to run it against your local BC3

Prerequisites: the event feed enabled and Action Cable served; an operator profile logged in as you; a second profile logged in as another person on the project; an agent connected and set up to serve the project in operator trust; no connector running for that agent.

basecamp profile create rob --base-url http://3.basecamp.localhost:3001
basecamp profile create jason --base-url http://3.basecamp.localhost:3001
basecamp auth agent connect -P agent
basecamp connect setup -P agent --operator-profile rob --serve <project-id>

BASECAMP_BASE_URL=http://3.basecamp.localhost:3001 \
BASECAMP_CONNECT_DEV_AGENT_PROFILE=agent \
BASECAMP_CONNECT_DEV_OPERATOR_PROFILE=rob \
BASECAMP_CONNECT_DEV_UNTRUSTED_PROFILE=jason \
BASECAMP_CONNECT_DEV_PROJECT_ID=<project-id> \
make test-connect-dev

Things a first real run may show (each is a real difference, either in the fake or in an assumption here): the event type BC3 sends for a new comment, the pointer line's lane (expected live, because the poll lane is about 30s behind), the comment title and app_url in the request line, and how BC3 writes a mention element in stored content.

Tests

  • New unit tests: devConfigFrom names every missing variable at once and points at CONTRIBUTING.md, reads a valid environment, and refuses a non-numeric or non-positive project id, a non-local or non-http(s) base URL, and two roles on one profile. devEnv keeps the developer's credentials (HOME, XDG_CONFIG_HOME, D-Bus), drops the token, profile and default ids, and replaces only XDG_STATE_HOME. withoutMention removes one mention element, whatever is rendered inside it (a BC3-style figure, or the fake's empty one), and refuses content with no mention or two.
  • make test-connect-dev with BASECAMP_BASE_URL unset exits non-zero with "Set BASECAMP_BASE_URL ...". BASECAMP_CONNECT_DEV=1 go test ./e2e/connect with nothing set fails the dev subtests with the full list of missing variables and skips the fake-only scenarios with their reasons.
  • Not run: the dev target against a real Basecamp.
  • go test -race -count=10 ./e2e/connect passes. bin/ci is green.

Summary by cubic

Runs the connector end-to-end scenarios against a local Basecamp as well as the fake, so the fake's behavior is held against the real one. Only test code, the Makefile, and CONTRIBUTING.md change.

  • The same scenario assertions run on both targets, using a Target interface that describes each Basecamp in the scenario's own words.
  • devTarget drives a local Basecamp with the real CLI and the developer's own config and credentials, isolating only the connector's state (XDG_STATE_HOME) per test.
  • Scenarios requiring faults only the fake can produce call fakeOnly and are skipped on the dev target, with their reason.
  • make test-connect-dev runs the dev target and fails, not skips, when a required environment variable is missing.
  • Scenarios that need a fault the fake can produce (dropped cable, rate-limited token, disconnected agent, second agent) remain fake-only; the dev run lists them as skipped with their reason.