feat(insights): rank optional context when sources exceed the budget

Merged
IZ
25d ago
+726-53

When the final profile and recalled-context merge exceeds the native source budget, relevant page excerpts can be omitted before an investigation sees them. This change uses one Jev evaluation to rank competing optional pages for the current subject, then applies the existing native packing rules.

The profile, canonical replies, original homepage, source contents, provenance, and budget stay under native control. Contexts that fit skip evaluation. Oversized input, unavailable billing, provider errors, invalid answers, and the one-second provider deadline return the exact native fallback. Returned usage is recorded before answer validation. The implementation covers fresh investigations and analysis replies; shadow runs require explicit injection, and clarifications/deterministic verification keep their current behavior.

Validation: repository lint and all 33 type-check tasks pass; 699 Insights tests pass (6 existing live tests skipped); 20 shared context/usage tests pass (1 existing service-dependent test skipped); the new ranking tests also pass an explicit TypeScript check. A frozen 48-investigation synthetic baseline/candidate comparison is running; this draft will be updated with the full quality and latency results. No production data, deployment, or merge is included.

Based directly on current staging; no unmerged PR dependency. Known overlaps: #849 shares the Gateway dependency, Jev price mapping, and nearby generation wiring; #842 shares Gateway lock entries; #751 owns broader business-context changes. This slice only changes final-merge page prioritization and reuses native packing.

Summary by cubic

Ranks competing optional pages with a single Jev call so the final profile/recall merge no longer drops relevant excerpts when it exceeds the native source budget.

  • Reuses the existing Jev fetch call from candidate relevance instead of adding a new gateway dependency.
  • One bounded evaluation (with a 1-second deadline) runs only when optional pages compete; contexts that fit skip evaluation entirely.
  • Oversized input, unavailable billing, provider errors, invalid answers, and timeouts return the exact native fallback, with usage recorded before answer validation.
  • Native packing still protects the profile, canonical replies, and original homepage; clarifications and deterministic verification are unchanged.
  • Applies to fresh investigations and analysis replies; shadow runs require explicit injection.
  • Adds typesafe-ai/jev pricing to agent credits.