Let reports be written by a local model

Closed
AL
2mo ago
+833-32

Summary

The report and the quiet-pause reviews can now be written by a model on the operator's own GPU instead of Gemini. The live interviewer is untouched and still talks to Gemini.

  • CODETRIAL_GEMINI_REST_BASE, read from the environment like INTERVIEW_ROOM_NAME, points the generateContent calls at another server. Unset, the URL is what it was.
  • scripts/gemini-shim.py (standard library only) answers generateContent from llama-server's OpenAI-compatible endpoint. It converts the Gemini response schema into the JSON Schema that llama.cpp compiles to a grammar, turns thinking off when the request asks for a zero budget, and passes upstream status codes through so the retry rules still see a 503 as a 503.
  • A local model is slower, so any base other than Google's gets longer report deadlines: 45 seconds per call and 240 in all, instead of 20 and 125. Gemini keeps its values. Only the server knows which deadline is in force, so the page's wait before it offers to leave now comes from /runtime-config.js. The page's own default stays the hosted 135 seconds and acts as a floor, so a missing config can never shorten it.
  • When a report names the published problem, the repair prompt now spells out the title and the scenario name to use instead. A smaller model told only names the published problem kept writing "a 'Two Sum' style problem" and lost the report. The title stays out of the failure note the candidate reads.
  • When the improvement plan and the feedback disagree, the repair now names each wrong item by index (a reworded weakness, or a repeat), each improvement with no item, and when there is one of each, the swap. Told only that the plan and the feedback disagreed, the local model sent back the same plan byte for byte on every repair.
  • README documents the variable and the deadlines.

Numbers

On an RTX 5070 Ti with the Two Sum report prompt from tests/golden/prompts.json:

  • gemma-4-12b Q4_K_M, with every change here: 10 of 10 reports passed validation, 28 to 50 seconds each. 7 of the 10 broke the plan rule on their first attempt and were repaired, 6 of them on the first repair.
  • The same model before the plan repair change: 0 of 6 in one run of six and 7 of 10 across an earlier ten, every failure the plan rule surviving both repairs. Before the title repair change, that rule alone cost 2 of 4.
  • Qwen3.5-9B Q6_K, before the repair change: 3 of 8. The failures were improvementPlan rules, not the transport.
  • Interim reviews through the shim: under a second each.

These are one prompt on one machine, so read them as "this works end to end" rather than as a quality comparison with Gemini.

Test plan

  • report_network_budget_covers_every_repair_and_retry_per_generation now checks that both the hosted and the local deadline pay for five attempts.
  • the_browser_escape_hatch_outlasts_the_report_deadline reads the page's default and checks it against the hosted deadline, then checks what the server sends against both.
  • The runtime-config test and the browser test stub carry the new line.
  • binary_web_gives_a_local_report_base_the_longer_wait in tests/cli.rs starts the binary with CODETRIAL_GEMINI_REST_BASE set and checks the page is told 250000. The base is read once per process and nothing else in the suite sets it, so this is the only test that reaches the local branch.
  • Four unit tests for the plan repair: a reworded weakness, a repeated one, several wrong at once (no positional pairing), and that the guidance stays out of other repairs and out of the candidate's failure note. Three of them fail with the guidance removed. The guidance is worked out from the response itself, and a plan that matches its feedback gets none.
  • tests/local_report.rs is an ignored check that makes one real generate_report call through CODETRIAL_GEMINI_REST_BASE. It is not part of the gate.
  • Manually: the built binary serves CODETRIAL_REPORT_ESCAPE_WAIT_MS = 135000 without the variable and 250000 with it.
  • ./scripts/test.sh passes locally, formatting included. cargo mutants --in-diff on this diff: 36 mutants, 33 caught, 3 unviable, none missed. cargo-audit and shellcheck were not installed here.

Not tested: a full interview end to end with the report coming from the local model.