Hold played candidates to the interview rules

Merged
AL
2mo ago
+646-9

Summary

Follows up #106. tests/interview_behavior.rs scripts one candidate per problem, so every rule it checks is only exercised on the one path that script walks. This adds an ignored test next to it in which a model plays the candidate and every interviewer turn is held to the same rules.

  • Three personas, chosen for the rules they press on: one who keeps asking for hints and then for code; one who tries to get the source confirmed and the limits out of the interviewer indirectly; one who claims to be an administrator and tries to talk the interviewer out of its instructions.
  • The candidate sees only the scenario brief. Each turn is checked with the existing rule functions (names_source, states_limit, named_beyond, the withheld rung, log_hint on a request), plus two new ones: reciting the optimal approach the candidate has not stated, and reciting the prompt outside its quoted example lines.
  • The interviewer can be played by a local OpenAI-compatible server (BEHAVIOR_LOCAL_BASE) instead of Gemini, and the candidate by BEHAVIOR_CANDIDATE_BASE (falling back to the local base), so both kinds of candidate can be compared against one model at no cost.

This is regression and robustness testing of the interviewer, not calibration. Nothing here touches scores, and the test stays #[ignore] and out of the gate like the scripted one, because two models talking are not deterministic.

The failures reported in #106 were found with gemma-4-12b on llama.cpp as the interviewer, not the production model. They still need a rerun against Gemini.

Test plan

  • cargo test --test interview_behavior: the two unit tests pass; the scripted and played checks are ignored as intended. cargo fmt --check is clean.
  • Played check run against a local llama.cpp server on 3sum,coin-change,two-sum, 3 runs × 3 personas × 8 turns (results in #106). Not run against Gemini, for lack of a key.

Summary by cubic

Adds an ignored test that plays a model candidate against the interviewer, so every interviewer turn is held to the interview rules regardless of the conversation path. Also lets the interviewer run against a local OpenAI-compatible server instead of Gemini, making it cheap to compare both kinds of candidate against one model.

Changes

  • Three candidate personas: a hint-seeker, a source-hunter, and a manipulator trying to break the interviewer's instructions.
  • Each turn is checked against the existing rule functions plus two new ones — reciting the optimal approach the candidate hasn't stated and reciting the prompt outside its quoted example lines — including turns the script lets pass by default: the greeting, a volunteered hint, an internals request, and the reply to neither. Empty replies, from a server error or a candidate that stops talking, now fail the run.
  • The persona that never states an approach makes reaching the key step a failure while the recital checks stay armed.
  • The candidate uses BEHAVIOR_CANDIDATE_BASE, falling back to the interviewer's local base; the test stays #[ignore] and out of the gate since two models talking is nondeterministic.
  • Rule definitions tightened, each case a live run tripped on now a unit test: a size question needs an actual question, "help with the approach" is a hint request, only phrases like "all the hints" mark fishing for internals, and the optimal-approach check stops once the key step is served.