fix(gemini-ledger): hand the tier report only records traceable to trusted runs

Merged
DO
8d ago
+1264-41

Problem

The Gemini tier snapshot and its history are built from token-usage-* artifacts collected across the organisation. Those artifacts are untrusted input: a pull request run from a fork of a public repository can upload one under any name, with any content. That is harmless while the snapshot is informational, and stops being harmless the moment anything gates dispatch on it. "Gemini trickle under a budget pause" (#2147) makes this its first acceptance criterion: records feeding a dispatch decision must be traceable to trusted runs.

This is that criterion only. The trickle itself is not in this pull request and is not being built yet; see "What this does not do".

Design

What a record can prove. A record carries only what the job wrote, including its own run_id; all of it, and the artifact's name, can be forged. What cannot be forged is what the Actions API says about the run that uploaded the artifact.

The trust rule. An artifact is trusted when both hold:

  1. The artifact listing's workflow_run.head_repository_id equals repository_id (not a fork).
  2. Either its head branch is the repository's default branch, or a look-up of the run confirms the same run id and head repository and that neither the actor nor the triggering actor is Dependabot.

That is the set of runs GitHub gives the repository's secrets to. Whoever controls such a run's code can already call Gemini with the real keys, so a forged ledger gains them nothing; fork and Dependabot runs get no secrets and cannot have made a real call. Anything that cannot be established (a failed or mismatching look-up, no workflow_run, no actor) is untrusted.

"Default branch only" was rejected: on four days of real data it would drop about a fifth of genuine usage (269 of 1,353 artifacts), and missing usage makes Gemini look more available than it is.

Where it sits. scripts/token_report.sh classifies each artifact at listing time, looks up non-default-branch runs, and stamps every record with provenance {run_id, trusted, basis}, replacing anything the artifact wrote under that key. Only trusted records are handed to the tier report. Ignored and unreadable counts go to the log, a warning and the job summary.

"Cannot tell" is not "available". The snapshot gains a collection block (complete or not, counts, reason). A new reader for future dispatch decisions, gq_tier_decision, answers unknown when the snapshot is unreadable, stale (default one hour), from the future, for a quota day that has ended, built from an incomplete collection, not marked trusted-runs-only, or holding records for a model or key with no caps row. Nothing calls it yet. The existing informational gq_tier_can_run is unchanged.

The history cache key gains v2, so the store filled before the filter existed is not restored. The per-day history table loses up to seven earlier days, once.

What this does not do

  • It cannot remove test-fixture records. About 8 of the 1,819 request-counting Gemini records in the sample were real; the rest were written by this repository's engine tests, run by dev-lead agent sessions that inherit the real ledger path, and uploaded by those same trusted runs. No provenance rule can separate them. Filed as #2304. This change detects the visibly fake ones (a model or key with no caps row), warns, and makes the decision reader answer unknown while any are present; 709 of them look exactly like real calls and cannot be counted.
  • It does not build the trickle. The design work for #2147 stopped on three findings, written up on that issue: the Copilot credential in dev-lead is the acting token and cannot be withheld from a job that must push and comment; the ledger above is unsound; and a job's own ledger starts empty, so capacity is not knowable at dispatch time and the caps file bounds one job, not the fleet.
  • An agent session inside a trusted run can still be induced to append to its own ledger. Provenance authenticates the run, not each record.

Risk

  • More API calls in the daily fleet monitor run: one run look-up per artifact that is not on the default branch, about a fifth of them (roughly 270 a day on the sample). The monitor uses the shared maintainer login, whose hourly quota was exhausted twice today by other reads; if that proves tight, the look-ups can be cached per run id.
  • The listing and look-up field names were checked against the live API with a maintainer login, not with the monitor's own token.
  • The JSON Schema change for the collection block was hand-checked; the schema test needs a Python package that is not installed locally, so CI is the first real validation.
  • The collector's cost report is unfiltered on purpose: existing tests require artifacts without a workflow_run to still be collected for it.

Test plan

  • tests/token_report_trust.bats (new, 27 tests), wired into lint.yml: the trust rule (fork, same-repository branch, default branch, Dependabot, an artifact declaring itself trusted, an unreadable or mismatching look-up), the hand-off, the snapshot's collection block, the decision reader's fail-closed cases, and the wiring.
  • Existing suites unchanged and identical to baseline: token_report 44, test_gemini_quota 44, test_gemini_tier_view 48 (plus one that needs the missing Python package locally).
  • Mutation-checked: 24 mutations, each turning a named test red (fork accepted, self-declared trust believed, unreadable look-up accepted, Dependabot accepted, hand-off unfiltered, stale snapshot accepted, incomplete collection accepted, unlisted records ignored, cache not renamed, and others).
  • Re-run by me on this commit after rebasing: the new suite 27 of 27, token_report 44, test_gemini_quota 44, the fixed-path check 18; shellcheck --severity=warning -x and actionlint clean.
  • Not run: the fleet monitor itself.

Rollback

Revert the squash commit. The tier report goes back to reading every collected record, and the old history cache is used again.

Monitoring

  • The next "Actions Fleet Monitor" run's summary should have a "Gemini ledger provenance" section. Expect zero fork and Dependabot artifacts, and a warning about records for models or keys with no caps row until #2304 is fixed.
  • That run's duration and any download failed or timed out warnings, for the added look-ups.

Refs #2147

🤖 Generated with Claude Code