test(v5): audit immutable development manuscript evidence

Draft
TH
1mo ago
+3499-0

Objective: fail-closed manuscript development evidence identity gate

This stacked DRAFT PR adds a read-only, deterministic, model-free audit of the already completed V5 development evidence and the previous development/manuscript crosswalk. It targets draft PR #324 (exact parent 6d2f1dd2e96ac1e1dc62a2743d7c11e8c6bce2a5). Scientific/claim/original frozen files, the working manuscript, original research branch and main remain unchanged under TheHalfMoon/commandMed.

Exact source/evidence safeguards

  • SHA-256 checks eight versioned committed evidence/text sources plus both original private Kaggle R1/R2 original ZIP bytes. Avoids relying on text-only provenance assertions.
  • Confirms R1/R2 independently verified immutable receipts: distinct named kernels version 1, 16,384 ordered rows and 8,192 source tasks in each, frozen exact public RiskCalcs source hash, no weights loaded during offline verification.
  • Checks repeatability95=0 only as measured for the four predeclared primary and three secondary metrics over two full same-configuration runs; preserves near-chance baseline and no cross-seed/hardware/clinical inference.
  • Confirms all four nonclinical benchmark-scale margins equal their pre-R1-output frozen floors, not selected after results.
  • Validates the entire descriptive B1/C1/C2 9-seed/intervention x 2-split x 4-primary-metric SD inventory (72 finite nonnegative source-task SD cells) and rejects false frozen primary family size/confirmatory power claims.
  • Re-hashes all 307 original frozen-entry files and fails closed on altered original manuscript, red-team, claim-ledger, frozen evidence or run inputs.
  • Output explicitly marks independent_peer_review=false, power_adequacy=NOT_QUALIFIED, confirmatory=false, clinical_validity=false, publication_authority=false.
  • Tests reject forged/wrong repeat identity, shifted margin, cherry-picked family, nonfinite SD, missing/mutated input or unwarranted confirmatory/clinical/publication status.

macOS exact-head qualification

  • New focused audit tests: 5 passed.
  • V5 tests: 310 passed + 86 subtests.
  • Full repository: 1,362 passed + 264 subtests.
  • Python compilation, git diff --check, execution of read-only audit against authentic files: PASS, including 307 hashes, 8 bound source files, original 2 ZIPs, 72 inventory cells.
  • Genuine Alibaba OCR zero-cost delegate eligibility preview: 1 implementation Python file eligible; tests and Markdown were excluded by default_path/unsupported_ext. This is NOT an external LLM review verdict.
  • Commit has DCO sign-off, but cryptographic signing requires separate verification. No false GitHub signed-commit assertion or user-account key change.

Hard boundaries

No new model runs, Kaggle reruns, private/PHI data, paid API, source/test sampling manipulation, new scientific hypothesis, calibration settings, domain-floor selection, confirmatory/reserve identity, cross-backbone replication, or publication. A passing model-free audit is not a substitute for independent scientific review. The exact primary non-target Holm family, seed-aware conservative variance methodology and 90% power gate are NOT FROZEN/QUALIFIED, and the claims remain unproven. Do not merge this or ancestor draft PRs without their genuine review, signature and founder gates.

Summary by cubic

Adds fail-closed, model-free audits of the V5 manuscript's locked development evidence, now run automatically on pull requests.

scripts/v5_manuscript_development_evidence_audit.py verifies SHA-256 hashes of nine versioned evidence sources plus both original R1/R2 Kaggle ZIP archives, re-checks all 307 frozen entry files, and validates the 72-cell descriptive SD inventory, the 72-cell development-only calculator ICC inventory, and repeatability metrics. scripts/v5_s1_source_case_pairing_audit.py replays the frozen rule-oracle source generator against the pinned public RiskCalcs JSON to confirm each source case maps one-to-one to a task ID (8,192 cases, 16,384 rows, two ordered prompt variants each) across both full R1 and R2 matrices. scripts/v5_s1_statistical_review_packet.py independently reconstructs the 72 paired-SD and 72 calculator-ICC cells algebraically and bridges them into 24 three-seed reviewer groups, retaining all 12 negative ICC estimates without qualifying power. The 307-entry manifest stays pinned under its own SHA-256, since it is not one of its own entries and could otherwise be rewritten alongside its references.

  • Adds a CI workflow that verifies exact-head checkouts, compiles all verifiers, retrieves the SHA-pinned public RiskCalcs blob, replays the original calculator-ICC inventory from the pinned source matrices, and runs all audits and their regression tests on pull requests touching V5 development evidence. The workflow pins numpy==2.4.6 to match the verified rule-oracle calculator and uses an immutable actions/checkout revision.
  • Adds three docs/ audit reports explaining scope and limitations, plus a generated reviewer-packet artifact.
  • Adds regression tests covering real evidence, scope escalation, forged margins, missing sources, invalid SD values, manifest pinning, ICC provenance and authority flags, negative ICC retention, and source-case pseudoreplication.

Both audits reject forged repeat identity, shifted margins, post-hoc power family claims, nonfinite SDs, missing or mutated inputs, and unwarranted confirmatory, clinical, or publication status. They mark power_adequacy=NOT_QUALIFIED and are not a substitute for peer review.