feat(retry): add validateResponse to detect empty model responses and trigger fallback
Summary
Fixes #174.
Some model providers return HTTP 200 with an empty body (0 output tokens) on quota exhaustion instead of a proper error code. promptSyncWithModelSuggestionRetry only triggered fallback on thrown exceptions, so these empty responses were treated as success and the fallback chain was never activated — tasks silently failed.
Changes
1. packages/plugin/src/shared/model-suggestion-retry.ts
Add an optional validateResponse callback to PromptRetryOptions:
validateResponse?: (client: Client, sessionId: string) => Promise<void>;
Invoked after each successful prompt attempt (both primary and every fallback). If the validator throws, the error enters the existing catch block and is treated as retryable — the next fallback model is tried.
The validation calls are placed inside the existing try blocks, before return:
try {
await attemptOnce(/* ... */);
if (options.validateResponse) {
await options.validateResponse(client, args.path.id);
}
return; // ← only reached if validation passes
} catch (error) {
lastError = error;
// ... existing fallback logic ...
}
Same pattern for the fallback loop.
2. packages/plugin/src/features/magic-context/dreamer/runner.ts
Wire validateResponse at the two direct promptSyncWithModelSuggestionRetry call sites (task runner + smart-notes evaluator). The validator fetches session messages and rejects responses with zero assistant output:
validateResponse: async (validateClient, validateSessionId) => {
const messagesResponse = await validateClient.session.messages({ /* ... */ });
const messages = shared.normalizeSDKResponse(messagesResponse, [], { /* ... */ });
const output = extractLatestAssistantText(messages);
if (!output) {
throw new Error(`[dreamer:${taskName}] model returned empty response (0 tokens) — possible quota exhaustion`);
}
},
3. packages/plugin/src/shared/model-suggestion-retry.test.ts
4 new tests covering:
- Primary returns empty → fallback[0] succeeds
- All attempts return empty → throws last error
- Primary non-empty → no fallback tried
validateResponseabsent → backward compatible
Design Decisions
Why a callback instead of always checking?
Different callers have different definitions of "valid output." A dreamer task needs non-empty assistant text; a summarization task might need a minimum length; a structured-output task might need valid JSON. An opt-in callback lets each caller define its own validation criteria without coupling the retry layer to any specific output format.
Why not detect empty responses inside attemptOnce?
attemptOnce doesn't have access to the session query parameters (directory, limit) needed to fetch messages, and adding an unconditional message fetch would penalize all callers — even those that don't need validation. The callback approach adds one extra API call only for callers that opt in.
Why place the validation inside the try block?
Placing validateResponse inside the existing try block means its throw is caught by the existing catch block, which already has the fallback loop logic. The error from validateResponse is a plain Error, so isNonRetryable() returns false → fallback continues. No changes to the catch block were needed.
Backward Compatibility
Fully backward-compatible. Callers that don't provide validateResponse are unaffected — the if (options.validateResponse) guard skips the callback entirely.
Verification
bun run typecheck: 0 errorsbun test: 2166 pass, 0 fail (including 4 new tests)bun run lint: 0 issues
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.
Summary by cubic
Adds an optional response validator to the retry helper to detect HTTP 200 responses with empty outputs and trigger model fallback. Prevents silent failures on quota exhaustion and ensures the fallback chain runs.
- New Features
- Added
validateResponsetoPromptRetryOptionsinpackages/plugin/src/shared/model-suggestion-retry.ts; runs after each successful attempt and, if it throws, the next fallback model is tried. - Integrated the validator into
packages/plugin/src/features/magic-context/dreamer/runner.tsfor both the task runner and smart-notes; it fetches session messages and rejects when there’s no assistant output. - Backward-compatible. Tests in
packages/plugin/src/shared/model-suggestion-retry.test.tscover empty→fallback, all-empty→error, non-empty→no fallback, and no-validator behavior.
- Added