feat(telemetry): report LLM cycles per chat query

Merged
YU
4d ago
+431-35

Description

Chat query events now report llm_cycles. The count is the chat-loop LLM calls up to and including the call that produced the first answer token. If no answer token arrives, the count is the most calls that one model made. The fleet service uses it to keep deep agent runs out of the headline p95 and to show latency for each cycle class.

  • The chat loop counts each LLM call through the emitter, in the same way as tool usage. Model comparisons report the count of the model that answered first. LLM calls inside tools do not count.
  • Text that a model streams before a tool call is the first answer, so later calls do not count.
  • The emitter records the count in the worker thread when it queues the first answer text. first_answer_ms is recorded when the stream writer reads that text. A slow writer can increase the latency, but it cannot add later calls to the count.
  • Model comparisons without an answer report the deepest model's calls. The models run in parallel, so the counts are not added.
  • Deep research requests report the deep_research mode instead of chat. They keep their tool usage but report no cycle count, because their orchestration is not a chat loop.
  • Query modes are now a QueryMode enum.
  • The new field is one integer. No prompts, tool names, arguments, or outputs are collected.

Deploy order: https://github.com/onyx-dot-app/fleet-management-service/pull/48 must merge and deploy first. Until then, the production fleet service rejects events with the new field or mode. That PR is open and not deployed.

How Has This Been Tested?

  • New unit tests cover these cases:
    • cycles stop at the first answer token, including text before a tool call;
    • unanswered queries count every call;
    • stop-button completion freezes the count;
    • model comparisons use the first model that answered;
    • search and deep research report no count.
  • A two-cycle chat loop counts one cycle before each LLM call. The test fails when the counting call is removed.
  • The stream writer path reports the count for one and two models. The endpoint reports the deep_research mode.
  • A unit test and a threaded _run_models test hold the writer on the answer until the worker starts its next cycle. Both reported a count that was too high before 220b14654c.
  • After the review fixes, 1373 tests passed in backend/tests/unit/onyx/chat, tools, deep_research, server/features/search, and the query telemetry tests.
  • On the first commit, 726 tests passed in backend/tests/unit/onyx/chat and the query telemetry tests. A wider run over chat, utils, tools, deep research, and search had 1887 passed. Its 8 failures are all in test_process_isolation.py. They also fail on an unchanged checkout on this machine.
  • Pre-commit hooks passed on the changed files, including ty, ruff, ruff format, getattr, and lazy imports.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

🤖 Generated with Claude Code

Summary by cubic

Chat query events now report llm_cycles: the chat-loop LLM call count up to and including the call that produced the first answer token, or every call if no answer arrives. The fleet service uses it to keep deep agent runs out of the headline p95.

  • The emitter records each model's count when it queues its first answer text, so a slow stream writer cannot inflate the count; model comparisons report the first model that answered. Without an answer, the count is the deepest model's calls.
  • Text streamed before a tool call counts as the first answer, so the count matches the work first_answer_ms measures. A writer delay can raise first_answer_ms but never the count. LLM calls inside tools don't count.
  • Deep research requests report the deep_research mode instead of chat, keep their tool usage, and report no cycle count. Query modes are now a QueryMode enum.
  • The new field is one integer; no prompts, tool names, arguments, or outputs are collected.

Deployment: fleet-management-service must merge and deploy first. Until then, the production fleet service rejects events with the new field or mode.

Summary by CodeRabbit

  • New Features
    • Query telemetry now distinguishes chat, search, and deep research activity. Chat events include an LLM cycle count, measured through the first answer or through completion when no answer is produced.
    • Deep research events are identified separately and include tool usage, without an LLM cycle count. Search events do not report tool usage.
  • Documentation
    • Telemetry documentation now describes the reported query fields and how LLM cycles are counted.