[cli] feat: page eval download manifests, add a summary download type

Merged
AL
4d ago
+393-32

[!IMPORTANT]
Release after the Osmosis platform supports manifest paging and the summary download type. Platform support is rolling out separately. Until it is live:

  • Paging still works. An older platform returns the whole manifest in one response, and the CLI uses it as before.
  • --type summary fails, because the platform rejects the unknown type.

What

  • Paged manifests. osmosis eval download fetches the download manifest page by page. It sends cursor=0, follows next_cursor until it is empty, and joins the pages into one manifest.
    • Exact repeats across pages (same path and token) are dropped. A rollout can shift onto two pages while an evaluation is still running.
    • A path that appears with a different token is still reported as a conflict.
    • The loop stops with a clear error if the platform ever returns a cursor that doesn't advance.
  • Progress. Rich output shows a running "Found N files so far..." line while the manifest pages arrive. --json and --plain output are unchanged.
  • --type summary. Downloads only summary.jsonl, the per-sample index with status and reward, without downloading trajectories. summary.jsonl still comes with --type trajectories, as before.
  • --type all. Still the four existing types. trajectories already brings summary.jsonl, so all keeps working against platforms without the new type.
  • Unchanged. Benchmark downloads don't page and never send a cursor.
  • Docs. The eval download docs describe --type summary.

Why

Building the whole manifest in one request can take longer than the platform allows for large evaluations, so the download fails before any file transfers. Paging keeps each request small.

--type summary gives the per-sample results most analysis needs in a single fast request.

How to Test

uv run pytest tests/unit/cli/test_eval_download.py tests/unit/platform/api/test_client_benchmark.py

The tests cover:

  • Paging: three pages joined, with the right cursor on each request.
  • Older platforms: a response without next_cursor is treated as a single page.
  • Bad cursors: a cursor that doesn't advance raises.
  • Repeats: exact repeats are dropped, and same-path conflicts are kept as errors.
  • Benchmarks: benchmark manifests send no cursor.
  • Summary type: --type summary alone downloads summary.jsonl, and --type all sends the four existing types.

Checklist

  • PR title follows [module] type: description format
  • ruff check . and ruff format --check . pass
  • ty check --error-on-warning osmosis_ai/ passes; uv run --no-editable pyright --verifytypes osmosis_ai --ignoreexternal has no unexpected errors (only the Harbor harness_agent baseline)
  • pytest passes (new tests added)
  • Public API changes are documented
  • No secrets or credentials included

Devin Review