CI: make the tree-proof --self-test fail when an assertion fails

Merged
BI
1mo ago
+40-1

User description

node scripts/ci-main-tree-proof.mjs --self-test exited 0 even when one of its assertions threw. main() returned the self-test promise, and the entrypoint's fail-open .catch (meant for the CI proof path, where a proof problem must never fail the changes job) turned the rejection into a ::warning:: plus a "no proof" output. So a broken guard in the self-test could pass CI, and the vitest wrapper only failed by accident because of missing stdout text.

Fix: the --self-test branch now catches its own failure, prints self-test FAILED with the stack to stderr and sets exit code 1. The CI proof path keeps its fail-open behaviour unchanged.

Test: tests/ci-main-tree-proof.test.ts adds a case that forces a self-test assertion failure via the test-only CI_TREE_PROOF_SELF_TEST_FORCE_FAIL=1 hook and expects exit 1, the failure on stderr and no proof outputs. It fails on main and passes here; all 23 tests in the file pass.

Note: #3531 also edits this script (in other hunks); whichever merges second needs a normal merge of main.

Summary by cubic

Fixes the tree-proof --self-test so a failing assertion now fails the process instead of exiting 0. Previously the entrypoint's fail-open .catch handler (meant only for the CI proof path) turned a self-test rejection into a warning, so a broken guard could pass CI.

  • The --self-test branch now catches its own errors, prints self-test FAILED with the stack to stderr, and sets exit code 1.
  • The CI proof path keeps its fail-open behavior unchanged.
  • Adds a CI_TREE_PROOF_SELF_TEST_FORCE_FAIL=1 test-only hook and a test that expects exit 1, a stack frame in the failure output, and empty GITHUB_OUTPUT and step summary files; the test cleanup bounds retries to satisfy the test-runner-safety ratchet.
  • #3531 also edits this script in other hunks; whichever merges second needs a normal merge of main.

CodeAnt-AI Description

Make tree-proof self-test failures exit with an error

What Changed

  • A failed --self-test now reports the error to stderr and exits with status 1 instead of being swallowed by the CI proof’s fail-open handling.
  • Failed self-tests leave GitHub outputs and the step summary empty; normal CI proof failures remain non-blocking.

Impact

✅ Broken tree-proof checks fail CI
✅ Failed self-tests leave no proof outputs
✅ CI proof failures remain non-blocking

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.