spec: issue-3635 End-to-end test matrix for Chainloop Trace agents
Summary
Chainloop Trace supports several coding agents ("harnesses"): Claude Code, Cursor, OpenCode, and soon Pi. The Trace README has a feature table that says which features each harness supports. No test checks this table against the real harnesses, and the table can miss features that Trace already has.
This spec first reviews the table for completeness. Then it adds an end-to-end test suite. The suite runs each harness in headless mode with real models, in a throwaway repository with Trace in local storage mode. It checks each feature that the table marks as supported, and it builds a result matrix of harness, model, and feature. A nightly job compares the result with the previous run and posts a Slack message when a cell changes from green to red. The suite does not test features that the table marks "No".
Open questions for reviewers
- Do we set a cost limit for each nightly run? Proposed: measure the cost of the first Claude Code runs, then set a limit for each run.
We would like two reviewers to approve this spec before it merges.
Refs #3635
AI disclosure: Claude Code helped to draft this spec.