feat(trace): end-to-end test matrix for Claude Code
Implements milestone 2 of spec issue-3635 (#3635): the framework of the end-to-end suite for Chainloop Trace, and the Claude Code column. This first version runs locally only. The nightly job and the Slack message come later.
The suite is in app/cli/internal/trace/e2e/ and runs with make -C app/cli test-trace-e2e.
- Expected matrix: the suite reads the feature tables of the Trace README (R-002, D-002). A feature row maps to its checks by the feature name. The suite skips cells marked "No" or "Not applicable". It reports features that need a control plane as "not covered" (D-007), and rows with no check as "no check".
- Real headless runs: a harness adapter runs Claude Code with
claude -pand a real model, in a throwaway repository withtrace init --localand a local bare remote (R-003, D-001). The adapter reads the hook messages from the stream-json output, so the welcome message check reads what the user sees (R-004). - Scenarios and checks: three shared scenarios (edit and commit, work from a ticket file, use a skill) serve the checks of 14 features. The checks read the session that the pre-push hook stored in local storage mode (R-005, D-008).
- Repeated runs: a model-driven feature runs 3 times and passes with 2 passes. The result shows the pass rate (R-006, D-004).
- Matrix storage: each run writes its result as JSON. Only a run with no filter becomes the previous result (R-009). The comparison reports a regression (pass to fail) and an improvement (an "Unknown" cell that passes). A cell that fails in two runs one after the other is reported one time only (R-007, D-005). The result also renders as a markdown table (R-010).
- Configuration:
config.yamlpins the harness version and lists the models of each column (D-009, D-013).
Difference from the spec: in each scenario, the agent pushes inside the session. The spec does not say who pushes. A push after the session end stores no spec, because the session end removes the captured specs.
Not in this PR: the CI job, the Slack message, the installation of the pinned harness version, Cursor, OpenCode, Pi, and checks for five rows (Subagents, Spec source pointers, Pasted image pointers, Media skipped by redaction, Shell commands in another checkout).
e2e_test.go:98: result matrix, stored in results/runs/20261019T074829Z.json:
| Feature | Claude Code 2.1.303 / haiku |
|---|---|
| Transcript | pass |
| Token usage and cost | pass |
| Model detection | pass |
| Subagents | no check |
| Line attribution | pass |
| Shell changes attribution | pass |
| Welcome message | pass |
| Session link after push | not covered |
| Credentials warning | not covered |
| Spec capture instruction | pass 3/3 |
| Spec capture reminder | pass |
| Local spec sources read at push | pass 3/3 |
| Skills tracking | pass 3/3 |
| Spec source pointers | no check |
| Pasted image pointers | no check |
| Secret redaction | pass |
| Media skipped by redaction | no check |
| Allow markers in redaction | pass |
| Commit trailer | pass |
| Attestation at push | not covered |
| Local storage at push | pass |
| Shell commands in another checkout | no check |
e2e_test.go:109: suite gap: Subagents / Claude Code / haiku: no check is registered for this feature
e2e_test.go:109: suite gap: Spec source pointers / Claude Code / haiku: no check is registered for this feature
e2e_test.go:109: suite gap: Pasted image pointers / Claude Code / haiku: no check is registered for this feature
e2e_test.go:109: suite gap: Media skipped by redaction / Claude Code / haiku: no check is registered for this feature
e2e_test.go:109: suite gap: Shell commands in another checkout / Claude Code / haiku: no check is registered for this feature
--- PASS: TestTraceMatrix (37.65s)
PASS
ok github.com/chainloop-dev/chainloop/app/cli/internal/trace/e2e 37.942s
AI disclosure: Claude Code helped to write this change.