fix(agent-eval): drop harvested queries whose project cannot be read

Merged
BO
6d ago
+68-11

The query harvester no longer keeps a query under a public repository when it cannot tell which project the agent asked about.

Problem. harvest-queries.mjs (added in #531) keeps only queries whose project is on the --public list. Two cases credited a query to the wrong project:

  • A code-mode call that passed projectPath as a variable or expression (projectPath: p, `${root}/x`) was read as "no projectPath", so the query was credited to the session's starting directory. If that directory was a public repository, a query about a different, possibly private, project was kept.
  • A relative projectPath was resolved against the directory the harvester ran in, not the session's directory.

Change. callDirectory() decides the directory: an absolute path as given, a relative one against the session's directory, the session's directory when the call named none, and nothing (the call is dropped and counted) when the value is not a plain string or there is no session directory. literalFields() now reports a present-but-unreadable field as null instead of omitting it.

Two small fixes from the same review: --public accepts a remote URL or a name ending in .git, and the entry check works through symlinks and paths with spaces.

Verification (head of fix/harvest-project-path):

  • New test in __tests__/query-replay.test.ts covers a variable, a call expression, a template string, a relative path with and without a session directory, a non-string value and the unchanged cases. It fails against the old rule (projectPath ?? cwd) and passes now.
  • End to end on a two-call transcript: the literal call is kept, the variable one dropped (kept 1, duplicates 0, dropped 1), with --public https://github.com/bompus/codegraph.git.
  • Full suite: 754 files passed, 40 tests skipped (vitest run --minWorkers=1 --maxWorkers=8).
  • The committed corpus scripts/agent-eval/replay/codegraph.json is unchanged.

README rows checked: none apply (agent-eval tooling, no user-visible feature or number).

Summary by cubic

Fixes the query harvester so it no longer keeps queries under the wrong repository when the agent passes projectPath as a variable, expression, or relative path.

  • A code-mode call with a non-literal projectPath (variable, template string, expression) was read as having no project and credited to the session's starting directory, so a query about another, possibly private, project could be kept under a public repository.
  • callDirectory() now resolves relative paths against the session directory, falls back to the session directory when the call names none, and drops the call when its project cannot be read.
  • --public now accepts a remote URL or a name ending in .git, and the script's entry check works through symlinks and paths with spaces.

Covered by new unit tests for the directory cases and an end-to-end run that keeps a literal call and drops a variable one (kept 1, duplicates 0, dropped 1).

Summary by CodeRabbit

  • Bug Fixes
    • Query records are now attributed using resolved project paths, including paths relative to the session directory.
    • Records with missing or invalid project information are excluded instead of being attributed to an unknown or incorrect repository.
    • Public repository identifiers are normalized when possible, improving consistency in query reports.