Analyze: batched prompts on one audio download

Closed
SI
2d ago
+1420-68

Stacked on #1226 (codex/analyze-m4a). Merge that first, then retarget this PR to main.

What

POST /api/songs/analyze/batch answers many short prompts about one audio file in a single model request. Today each prompt is a separate call, so 36 prompts pay 36 base prices. A batch pays the base price once.

Modal (modal/serve_music_flamingo.py)

  • New generate_batch endpoint (POST, requires_proxy_auth, same class and container as generate).
  • Downloads and, if needed, transcodes the audio once, then runs the prompts one after another on the loaded model. Each prompt is its own conversation using the same audio file.
  • The download/transcode step and the inference step are now helper methods (download_audio, run_prompt) that both generate and generate_batch call. generate behaves as before, except its temp file is now removed even when inference throws.
  • Caps: 40 prompts, max_new_tokens 1–2048 per prompt, unique ids. Bad input answers 422 (invalid_batch: ...) before any download.

API

  • FLAMINGO_GENERATE_BATCH_URL is built from FLAMINGO_GENERATE_URL. Modal names each endpoint's host after its method, with underscores turned into hyphens, so the URL ends in -generate-batch.modal.run.
  • Request flow: validate (JSON → auth → body → verifyAudioUrl) → credit gate on one base price → call Modal → charge once → 200.
  • The route sets maxDuration = 800, above Modal's 600 s class timeout, because the prompts run one after another.

Request

{
  "audio_url": "https://example.com/song.mp3",
  "prompts": [
    { "id": "genre", "prompt": "What genre is this?", "max_new_tokens": 64 },
    { "id": "tempo", "prompt": "What is the tempo in BPM?" }
  ],
  "temperature": 0.2,
  "top_p": 1,
  "do_sample": false
}
  • prompts: 1–40 items. id is 1–64 characters and must be unique. prompt is 1–4,000 characters. max_new_tokens is 1–2048 (default 400).
  • temperature: 0–2 (default 0.2). top_p: 0–1 (default 1). do_sample: default false.

Response (200, flat)

{
  "status": "success",
  "answers": [
    { "id": "genre", "response": "...", "elapsed_seconds": 1.4 },
    { "id": "tempo", "response": "...", "elapsed_seconds": 0.6 }
  ],
  "elapsed_seconds": 3.1,
  "prompts": 2
}

Errors:

  • 400: invalid body, with missing_fields.
  • 401: not authenticated.
  • 402: insufficient credits.
  • 422: audio_url is unreachable or not audio.
  • 502: Modal rejected the batch. The body carries { status: "error", error, upstream_status, detail }.
  • 500: anything else.

Pricing

One base price (FLAMINGO_BASE_CREDITS, $0.05) plus usdToCredits(total_seconds × FLAMINGO_USD_PER_SECOND), charged once after the batch succeeds (creditsForFlamingoBatch, chargeForFlamingoBatch). Example: a 90-second batch is 154,940 credits; zero seconds is the base alone. The ledger row has the same fields as a single call: source api, provider modal, the model id, and resource_url when the audio URL is recorded. The usage ledger has no description column, so the prompt count only appears in the charge-failure logs. If Modal rejects the batch, nothing is charged.

Tests

54 new vitest tests in lib/flamingo/__tests__/:

  • Schema (18): limits, defaults, unique ids.
  • Validation (6): 400, 401, 422, and the order of the checks.
  • callFlamingoGenerateBatch (10): request body, URL, proxy headers, non-OK → FlamingoServiceError with the status, bad response shapes.
  • Credits (4): one base plus seconds; zero, negative and non-finite durations price at the base.
  • Charge (4): ledger fields, failures logged instead of thrown.
  • Process (4): charges once on the total seconds, omits a private audio URL, charges nothing on an upstream rejection.
  • Handler (8): 200 shape, gate on one base, 402, gate throwing → 500, 400 before the gate, 502 upstream, 500 other.

Checks:

  • npx vitest run lib/flamingo: 214/214 pass.
  • eslint (--max-warnings=0), prettier --check and tsc -p tsconfig.build.json are clean.
  • The Modal file parses with ast. Its batch validation, single download, answer order and temp-file cleanup (including when a prompt fails) were exercised locally with Modal and FastAPI stubbed out.

Not verified

  • modal deploy has not been run. The batch endpoint has not been called on a real GPU.
  • Timing: a batch at the full caps (40 prompts × 2048 tokens) could exceed Modal's 600 s class timeout. Typical short-prompt batches should fit, but this has not been measured.

🤖 Generated with Claude Code

Summary by cubic

Adds POST /api/songs/analyze/batch, which answers up to 40 short prompts about one audio file in a single model request. Today each prompt is a separate call paying the five-cent base price; a batch pays the base once, plus the batch's total seconds, and is charged after the batch succeeds.

Modal

  • New generate_batch endpoint downloads and transcodes the audio once, then runs each prompt sequentially as its own conversation.
  • Download, transcode, and inference became shared helpers; generate now removes its temp file even when inference throws.
  • Invalid input answers 422 before any download; caps are 40 prompts, 1–2048 max_new_tokens per prompt, and unique prompt ids.
  • Deploy hasn't been run; a full-size batch (40 × 2048 tokens) may exceed the 600s class timeout.

API and pricing

  • The flow validates JSON → auth → body → audio URL, gates the balance on one base price (402 when short), and charges nothing when Modal rejects the batch (502 with upstream_status and detail).
  • maxDuration is 800s, above Modal's 600s timeout, because prompts run one after another.
  • 54 new vitest tests cover schema, validation, credits, charging, and the handler; lint, format, types, and the lib/flamingo suite pass.