fix(agent-shield): fail at the configured min-severity (#1436)

Merged
DO
6d ago
+416-5

Summary

Fixes #1436

The min-severity input of agent-shield-reusable.yml now sets the failure threshold. The scanner's --min-severity only filters its report, and its exit code only reflects critical findings, so a report with only high findings passed the job.

  • The scan step writes the scanner's JSON report to $RUNNER_TEMP. Exit 0 and 2 continue, because 2 only means critical findings exist. Any other exit stops the job with the scanner's code.
  • A new step runs agent_shield_severity_gate, a pure lib in .github/scripts/agent-shield/lib/severity-gate.sh, checked out at job.workflow_sha (same pattern as #839). It fails at or above the threshold.
  • The gate fails closed on an unknown threshold, or a report that is missing, not one JSON document, or off-schema.
  • The job name, inputs and caller stubs do not change. Log output is now severity, title, location and description only.

JSON schema

Read from the ecc-agentshield package source for 1.4.0 (the pinned default) and 1.6.0 (the latest): calculateScore returns findings[] with severity, title, description, file, line, and summary.totalFindings. Both versions print it with --format json and exit 2 only when critical findings exist. I did not run the scanner locally. This PR's own agent-shield / AgentShield check runs the real scanner through the new steps, because this repo calls the reusable by local ref.

Sequencing with #1419

Merge after #1419. That PR changes the same scan step, and this branch does not repeat its changes.

This PR is based on main rather than stacked because the checks only run on PRs targeting main. A trial merge with #1419's branch is clean, and on that merged tree both this suite and #1419's scan-step test pass.

Problem

min-severity did not act as a failure threshold.

Risk

Medium. The check becomes stricter. agent-shield / AgentShield is a required check, so once a channel carries this, a repo with existing findings at or above its threshold fails that check until they are fixed or its threshold is changed. This repo has none, per its latest main run. Other repos pick this up only when their channels move. This PR also adds require_confirmation to agent-shield's ring1->stable hop, so the release waits for a deliberate confirm before reaching repos that are only on stable.

Test plan

  • bats test/workflows/agent-shield/: 18/18 pass, including the reusable's real run blocks against a stubbed scanner.
  • bats test/workflows/dependency-hardening.bats, shellcheck -x and actionlint: clean.
  • New workflow agent-shield-tests.yml runs shellcheck and the suite on changes to these files.

Rollback

Revert this PR, or move the affected channel back to its prior release. No migrations or external state.

Monitoring

Watch agent-shield / AgentShield on this PR and on the first ring to receive it.

šŸ¤– Generated with Claude Code