#1168 feat(testing): artifact validation, upload orchestration, and Linux CI contracts (stage 2)

Summary

Implements staged artifact validation and offline upload orchestration from the flakiness testing dashboard plan (docs/plans/2026-11-05-flakiness-testing-dashboard.md), building on the native reporters merged in #1166.

This PR adds the fail-closed security foundation required before any live Flakiness service uploads: untrusted CI artifacts are consumed with trustworthy GitHub provenance, validated through a public schema and strict security checks, snapshotted into private read-only storage, and orchestrated through an isolated CLI boundary with a durable delivery ledger.

What's in this PR

Commit 1: validate CI report producer provenance (80121b76f)

• scripts/testing/validate-flakiness-artifacts.mjs — validateProducerProvenance(): pure policy enforcement for the trusted GitHub API context. Validates workflow ID, event, branch, job/step identity, PR snapshot binding.
• scripts/testing/validate-flakiness-artifacts.d.mts — TypeScript declarations
• tests/reporting/artifact-provenance.test.ts — 54 provenance contracts

Commit 2: validate and snapshot native Flakiness artifacts (dc767967)

• Adds validateArtifactPayload() and verifyUploadSnapshots() to the same module
• scripts/testing/read-flakiness-artifacts.py — trusted Python POSIX FD bridge: O_NOFOLLOW leaf + descriptor-anchored ancestry, bounded FD-scandir enumeration (Node v22 exit-13 behavior documented), no unsafe fallback
• tests/reporting/artifact-validation.test.ts — 240 artifact contracts (ancestor TOCTOU seams, ZIP preflight/CRC, preview/local targets, batch input, zero-test native failures)
• tests/reporting/fixtures/artifact-*.ts/py — native reporter, ZIP, intake harness, import error
• package.json / package-lock.json — exact dev pins: @flakiness/flakiness-report@0.36.0, yauzl@3.4.0, @types/yauzl@3.4.0
• docs/testing/flakiness-coverage.md — Task 5 trust boundary documentation

Commit 3: upload orchestration, durable ledger, and Linux CI contracts (be9ec9aa4)

• scripts/testing/upload-flakiness.mjs — validateOnly() and executeUpload(): verifies all snapshots before auth, fixed CLI args/env isolation, bounded stdout/timeout, atomic ledger (started → failed/ambiguous/accepted-unverified), no ambiguous retry
• scripts/testing/upload-flakiness.d.mts — typed API declarations
• scripts/testing/flakiness-runtime/ — pinned independent CLI runtime manifest (flakiness 0.327.0, SDK 3.6.0, schema 0.36.0, zod 4.4.3, yauzl 3.4.0); all 44 package integrities verified against official npm tarballs
• tests/reporting/upload-orchestration.test.ts — 23 TDD contracts: validateOnly, auth lazy-read, fixed args, env isolation, testedSha pass-through, all receipt URL states, timeout/overflow/signal, partial batch, ledger blocking, changed-content/corrupt/unsafe-key rejection
• tests/reporting/fixtures/fake-cli-*.mjs — 8 fake CLI scripts for testing
• .github/workflows/reporting-contracts.yml — credential-free ubuntu-latest CI workflow to verify Linux Python FD bridge; no OIDC, no uploads
• tests/reporting/reporting-ci.test.ts — 25 YAML contract tests (triggers, permissions, action SHA pins, env isolation, mutation guards)
• docs/testing/reporting-contracts-ci.md — Linux CI workflow documentation
• package.json / package-lock.json — promotes yaml@2.9.0 as exact dev dependency (already in lock)

Total: 342 reporting contracts (294 original + 23 upload + 25 CI contracts), all passing.

Gates explicitly not crossed

• No live CLI execution — CLI runtime installed and tested against fake scripts only
• No authentication or OIDC — Task 6
• No hosted delivery or service verification — Task 7
• No cutover or legacy dashboard retirement — later gates
• Linux Python FD execution — pending this workflow's first CI run on ubuntu-latest

Linked

• Builds on #1166 (native reporters, stage 1)
• Plan: docs/plans/2026-11-05-flakiness-testing-dashboard.md

Summary by cubic

Implements stage 2 of the flakiness dashboard plan: validates CI artifact provenance and payloads, then orchestrates uploads through a pinned CLI runtime with a durable delivery ledger (342 reporting contracts). This is the fail-closed trust boundary required before any live Flakiness service uploads.

Changes

• Validates producer provenance and native payloads against the public schema with strict security checks; snapshots go to private read-only storage via a Python FD bridge that never loads artifact code.
• Adds upload orchestration with fixed CLI args/env isolation, bounded stdout/timeout, and an atomic ledger that blocks re-upload on ambiguous or failed states.
• Adds a credential-free ubuntu-latest CI workflow verifying the Python FD bridge on Linux, with full git history (fetch-depth: 0) for the git-show wiring tests; Linux support is pending its first successful run.

No live CLI execution, authentication, or hosted delivery in this PR; those belong to later gates.

Confidence Score: 0/5

Do not merge yet: the upload command is incomplete, and the earlier validation, duplicate-delivery, durability, and startup failures remain.

Findings

1.  Upload subcommand is missing ▶
2.  Changed snapshots pass validation ▶
3.  Overlapping calls upload twice ▶
4.  Crash erases delivery record ▶
5.  CLI startup crashes uploader ▶
6.  Isolation tests miss regressions ▶
7.  Snapshot recheck can hang ▶

Summary

Adds artifact checks, private snapshots, upload orchestration, and a credential-free Linux reporting workflow.

• CI reports are checked against trusted run details before they become upload snapshots.
• Verified report snapshots can be passed to an explicitly enabled Flakiness upload.
• Reporting contracts run in a credential-free Linux CI workflow.

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Trusted producer context] --> C[Validate artifacts]
  B[Untrusted directory or ZIP] --> C
  C --> D[Private snapshots]
  D --> E[Verify snapshot files]
  E --> F[Check delivery ledger]
  F --> G[Run isolated CLI]
  G --> H[Record delivery outcome]

_{Reviews (3) · Last reviewed commit: "fix(test): update wiring baseline to 0ab..." · Reviewed by Greptile}