Run reports and the behaviour check on a local model

Draft
AL
2d ago
+1544-84

This now carries #99 as well, as asked there. It is rebased onto current main and folded into three commits: the self-hosted report path (the base URL, the shim and the longer deadlines), repair prompts that say what to change, and the behaviour check through the shim. The old history is still readable on #99.

Summary

The report, the quiet-pause reviews and the interviewer behaviour check can now run on a model on the operator's own GPU instead of Gemini. The live interviewer is untouched and still talks to Gemini.

  • CODETRIAL_GEMINI_REST_BASE, read from the environment like INTERVIEW_ROOM_NAME, points every generateContent call at another server. Unset, the URL is what it was.
  • scripts/gemini-shim.py (standard library only) answers generateContent from llama-server's OpenAI-compatible endpoint. It converts the response schema to the JSON Schema llama.cpp compiles to a grammar, carries function declarations and calls both ways, passes upstream status codes through so a 503 still reads as a 503, and gives up on llama-server when the caller would. A request with no output limit gets one, and --thinking off turns thinking off for every request, which Gemma 4 needs for the behaviour check. A request that asks for JSON without a schema, as the phase judge does, is held to a JSON object: left free, gemma-4-12b fenced its answers and every judgment in an interview was dropped. tests/test_gemini_shim.py runs in the gate.
  • Longer deadlines for a local base only. Any base other than Google's gets 45 seconds a call and 250 for the report, against 20 and 125; 250 is what five calls and the doubling backoffs need. Only the server knows which is in force, so the page's wait before offering to leave and its wait on a regenerated report now come from /runtime-config.js, and the page's own values stay the hosted ones as floors.
  • Repairs that say what to fix. When a report names the published problem, the repair names the title and the scenario name to use; when the plan and the feedback disagree, it names each wrong item by index, each improvement with no item, and the swap when there is one of each. Told only that a rule failed, the local model sent back the same output byte for byte. None of this reaches the candidate's failure note.
  • The behaviour check follows the base, so pointed at the shim it sends what production sends. BEHAVIOR_LOCAL_BASE from #107 stays as the direct route, without the shim.

Numbers

RTX 5070 Ti, gemma-4-12b Q4_K_M on llama.cpp, shim with --thinking off:

  • Reports on the Two Sum golden prompt, this branch: 4 of 4 valid, 14 to 16 seconds each, and 6 of 6 in the two reference runs on #257, 15 to 45 seconds each in one or two calls. On the old branches, the plan repair took this from 0 of 6 to 10 of 10, and the title repair from 2 of 4 to 4 of 4.
  • Behaviour check through the shim: the Two Sum script passed in 9 seconds. On the old branch, 18 of 21 problem runs across seven runs; each miss was a second hint request answered without calling log_hint.

One prompt, one machine: read these as "this works end to end", not as a comparison with Gemini.

Test plan

  • The report budget test holds both deadline pairs to five calls plus backoffs; the escape-hatch and recovery-limit tests hold the page's floors to the hosted deadline and the server's values to both.
  • binary_web_gives_a_local_report_base_the_longer_wait starts the binary with the base set and checks /runtime-config.js says 280000 and 270; without it, the runtime-config test pins 155000 and 145.
  • report-recovery.test.js: the server can raise the retry wait and cannot lower it.
  • Repair guidance: the title test, and seven plan tests (reworded, repeated, several at once, counted once across sections, a missing weakness keeping indexes, a matching plan getting none, guidance staying out of the failure note).
  • a_local_model_writes_a_report is an ignored unit test that makes one real report through the base.
  • Locally on this head: cargo test 1749 passed, 22 ignored; npm test 1230 passed and 1 failed, the lobby layout test, which fails the same way on main on this machine; tests/test_gemini_shim.py passes; clippy and formatting clean; each of the three commits builds on its own. The mutants run is left to CI on this diff.

Tested by hand twice end to end with the report coming from a local model (run 1, run 2). Not tested: anything against Gemini.

Refs #257
Closes #105