PR #899 - acyclic-labs/sdk

Problem

Invalid or refused model tool calls used to abort the turn as a causeless indeterminate outcome. Provider refusals, connection failures and stalled streams could lose their cause, wait indefinitely, or brick the task instead of settling it as failed.

This covers live findings F1, F2, F24.errors, F26.provider and F19. Recovery of an already indeterminate admitted task (F3) remains live-resume's separate scope.

Design

Unknown tools, invalid arguments and pinned policy denials settle as ToolFailed without a start. Admitted execution, output, projection and publication failures settle with a bounded ToolFailureCause. The model receives a paired ToolResultContent::Error and can correct the next call. Authority/evaluation failures and declined approvals remain strict.
Every settlement is durable. One conversation publisher emits calls and results in model order, including refusals without ToolStarted; replay publishes and dispatches nothing again. The summary admits only valid predecessors and funds every new settlement slot.
Supplied and retained history use one result contract. Every result must pair with its issued call. Binding and argument schemas are checked for every paired call: a refused call admits only a shape-valid error; a bound, valid call admits either that error or a projection-valid result. Error messages must be nonempty, control-free and within the render limit. History is not authenticated evidence of execution. This follows the coordinator decision recorded in the 6cf49c018 follow-up; provenance stays deleted.
Dispatch peels wrapped causes. Only uncertainty enters reconciliation; definitive failures retain their cause in task failure frames and status.json, release the lease, and exit headless with code 1.
Provider HTTP statuses and transport failures keep their class. Retryable refusals/connect/timeouts/interruption use one bounded redelivery budget before any event escapes. TLS, definitive statuses and failures after output are final. Both Retry-After forms are honored, including numeric overflow; hints exceeding the wait budget decline redelivery.
ResolvedModel.transport is an Arc<Bounded>, so dispatch, native generation, token count and credential refresh all inherit total and body-idle deadlines. Deadlines are measured on HttpTransport::now() on every poll and expire once the clock passes them (a read exactly at a deadline proceeds); the timer only wakes measurement and is armed one millisecond past the deadline, so platforms of millisecond granularity that drop fractions still pass it. Total time starts at the send. Idle time starts at headers and resets only when nonempty body bytes are consumed. Both expired deadlines precede ready chunks or EOF, including after timer-driver starvation or delayed consumer polling. Timers that fire early re-arm for the remaining time.
Every provider timer (header/body deadline wake-ups and generation/count retry backoffs, portable and native) is created by one constructor, deadline::timer, which admits it on the original ledger before allocation whenever a ledger exists; scripts/qualify-provider-stream.sh rejects any other timer construction in the crate. Re-polling reuses the armed deadline timer. The native unknown-tool refusal uses the existing static native error constructor. Browser timers own and clear their handles and split waits at the signed 32-bit timer limit.

Deletions

Deleted stock_tool_lookup, the ToolFailed-as-command-error branch, the catch-all indeterminate dispatch arm, obsolete retry scenarios, and the transferable FailureProvenance/special retained loader design. All histories use the same prepared-request loader. No separate store or bespoke protocol was added. The wire gains typed tool error content and the Unavailable error code.

Guarantee

Class: G (lost causes and wrong failure scope), B (unfunded native allocations and summary retention), invalid settlement transitions, shortened retry hints, and deadline bypass. The structural boundaries are the failure renderer, summary admission, paired-result validator, settled-exchange publisher, recursive cause classifier, bounded transport, retry policy and the single admitted timer constructor (deadline::timer, enforced by a structural scan). Refusals cannot escape through a command-error ?; every settled call has a record; each call's records are classified once into RecordedCall (settled failure with or without a start, settled success, started-unsettled, unstarted) and settled by one match, so a settlement replays without requiring a start and reconciliation never stops at a settled call; native waits carry their funding; producer readiness cannot bypass an expired deadline.

Proof:

every_result_is_admitted_by_one_contract_on_every_source is a 256-case shrinking property over call standing, pairing, result kind and error shape. It checks validate/prepare/decode and canonical prepare/decode round trips. an_unissued_or_malformed_error_result_is_rejected supplies the negative controls. The existing harness-fuzz/model_request cargo-fuzz target checks round trips; checked-in bound/refused/control/unpaired tool-error seeds run in the lib suite as tool_error_fuzz_corpus_round_trips_only_admitted_exchanges.
inherited_error_results_meet_the_request_contract_whatever_the_volume checks authored prefixes, canonical refusals, empty/control-bearing errors and invalid projected results. refused_calls_survive_an_interruption_after_the_next_context_is_prepared cuts the durable journal between ContextPrepared and ModelStarted: a fresh executor sends the identical refusal history and dispatches nothing. These replace the obsolete provenance-bit approach and close the functional LR-provenance/LR-recovery findings under the coordinator-approved uniform contract.
refused_and_failed_calls_publish_their_exchange_for_the_next_turn uses run_conversation with unknown, schema-invalid, executor-failed and successful calls in one step. The next turn receives the exact ordered exchange; only admitted calls dispatch, and replay does neither. Removing refusal publication fails the canonical order assertion (LR-exchange).
every_call_in_a_response_reconciles_exactly_once (64-case shrinking proptest, 1-5 calls drawn from policy-denied, unbound, succeeding, definitively failing and response-lost) drives one response the way the task runtime does: ordinary run, then a reconciliation-only pass after every indeterminate run. Each pass must settle exactly one more call, the turn must finish within one retry per lost call, every call settles exactly once, executes at most once, lost calls reconcile exactly once, and replay redispatches nothing. denied_call_does_not_block_reconciling_a_later_uncertain_call pins [Deny, Lost] and [Ghost, Lost]. Negative control: with b3bc61a5a's tool_execution.rs both fail ("[Deny, Lost]: reconciliation did not settle the uncertain call"; proptest shrinks to [Deny, Lost, Ghost]).
every_tool_failure_entry_point_carries_a_nonblank_bounded_cause covers every failure kind exhaustively in its constructor match, with shrinking blank/control/UTF-8 causes. Blank and unsanitized cause controls are rejected. Tool execution/replay, durable failures, portable/native policy denial and wrapped uncertain/definitive worker failures have focused regressions.
refused_tools_fund_their_summary_entries asserts exact retained bytes over generated counts/steps. refusal_admission_requires_an_operation_and_an_unstarted_call checks every failure kind against predecessor/identity states. Removing new-slot funding or predecessor exclusivity violates these properties.
a_silent_body_ends_at_the_idle_deadline and a_slow_exchange_ends_at_the_total_deadline check generated histories on a paused clock: stalls before/after partial output, shorter gaps, silent tails, slow trickles and successful completion. Idle gaps are compared against the deadline itself, without windows.
expired_total_precedes_ready_producers covers ready headers/chunks/EOF after total expiry. a_starved_timer_cannot_extend_a_deadline checks idle/total expiry with a timer that never fires, before and after partial output; the timer-only race remains pending as its negative control. idle_expiry_precedes_ready_body_and_empty_chunks extends that proof to ready bytes/EOF and empty chunks; the raw producer is ready while the guarded read fails, and after empty chunks the read lands within one idle window of the last empty chunk but past one from the last bytes, so counting empty chunks as progress fails it. deadlines_expire_once_the_clock_passes_them pins the exact bound for both deadlines with a ready producer: a read at the deadline proceeds, one nanosecond past it fails (an inclusive >= comparison fails the first case); the idle-history property now also generates exact-deadline gaps as tolerated. the_unguarded_body_stays_pending_past_the_deadlines is the original stall control.
native_deadline_timers_are_admitted_exactly_on_the_ledger measures the exact one-allocation timer claim, zero-allocation re-poll and custody return. Its portable control allocates without moving the ledger. native_retry_timers_use_original_ledger_for_every_exchange checks generation/count paths over retry budgets, failures and timer-admission refusal: exact platform timer allocation bytes equal ledger growth, portable callbacks remain unused, denied admission prevents redispatch and all custody returns. Reverting either retry call site invokes the unadmitted portable callback and fails the assertions. The timer scan in scripts/qualify-provider-stream.sh fails on both 6cf49c018 and c1e62955b (it lists the stray .delay/.delay_with_native_work calls) and passes at the head.
native_unavailable_cause_allocates_nothing_before_ledger_admission measures zero diagnostic allocations at the production rejection constructor. The former portable error allocation is its negative control.
Retry-After has a shrinking round-trip/budget property and provider_stream cargo-fuzz coverage with checked-in date/delta/overflow seeds. The delta-only and overflow-discarding parsers permit early retry and fail the assertions. The complete count exchange shares one retry budget across header and body faults (count_body_failures_share_the_exchange_attempt_budget).
Real loopback HTTP regressions through run_headless assert exact sends, failure cause, failed task/exit 1, and no Storage recoding for exhausted refusals/connect failures, body interruption and post-output stalls. A retry followed by success exits 0. Provider e2e scenarios also cover native ledger dispatch, 429/500/400, unreachable hosts, pre-output redelivery and post-output final failure.
Native diagnostic rendering allocation counters, bounded UTF-8 diagnostic properties, and the browser timer ownership/range test include eager-formatting and unowned-timer negative controls. Existing real-shim refresh models remain unchanged; this pass adds no synchronization primitive.

Entry points:

Tool lookup/grants, input schema, pinned portable/native policy, execute/reconcile, ordinary and reconciliation-only replay of every per-call record state (settle_tool_call, settle_tool_refusal, both through RecordedCall::classify), output validation, projection, artifact publication, durable tool runner, failure replay and canonical conversation publication.
ModelRequest validate, PreparedModelRequest prepare/decode, selected_context, checkpoint and inherited-prefix loading, including ContextPrepared crash recovery.
Filesystem command dispatch, task settlement/status publication and headless outcome conversion, including wrapped task-file/native errors.
Portable and native provider generation; verified token counting (headers and entire body); Codex credential refresh through the bounded transport; response SSE decoding/terminal checks; HTTP refusal/status classification; reqwest and browser transport causes; recognized stream-error events.
Header total timers, body idle/total timers and re-arms; portable/native generation and token-count backoff waits; browser timer cancellation and long waits.

Gaps: none. The earlier matrix text requiring an authenticated retained-failure handle was superseded by the coordinator-approved uniform supplied-history contract above. The provenance/recovery/publication and provider rows are covered by the listed production-path proofs. Whole-system DST scenario ownership remains with co-dst under the coordinator's ordering rule; this PR's own proofs are the merge requirement.

Review fixes in this pass

Expired idle deadlines lost to ready bytes/EOF, and empty chunks counted as progress. Both deadlines now precede producers, and only nonempty chunks advance progress; the shrinking starvation/readiness property supplies the negative control.
Native generation/count redelivery still called the portable, unfunded delay method. Both now use the admitted timer contract, propagate admission errors and retain cancellation. The native retry property checks both entry points with exact allocation/ledger counts and denied-admission controls.
The native unknown-tool refusal allocated a portable NotFound error before admission. Rejections now derive their errors through the existing native boundary constructor, with a zero-allocation proof and the prior allocation as its control.
The model-request fuzz corpus lacked the new tool error content. Four checked-in seeds and the shrinking canonical round-trip property cover it. The provider fuzz workspace lockfile now includes its direct httpdate/rustls dependencies, and its fuzz entry point has a qualification compile gate.
This body, TimeoutSpec docs and tool-composition docs now describe the current clock/deadline and uniform admission contracts, including the three 6cf49c018 proofs.

Review fixes in round 2 (Claude)

deadline.rs expired: deadlines failed when the clock reached them (>=), so a read exactly at the idle deadline failed although TimeoutSpec documents it as the longest tolerated interval, and the total timeout reported "exceeded" at equality. They now expire once the clock passes them, and the wake-up timer is armed one whole millisecond past the deadline: a nanosecond margin livelocked the e2e Recorded transport (and would livelock any HttpTransport whose millisecond timer drops fractions) at exactly the deadline. Proof: deadlines_expire_once_the_clock_passes_them, exact-deadline gaps in a_silent_body_ends_at_the_idle_deadline, and a_millisecond_platform_reaches_every_deadline (a truncating millisecond platform reaches both deadlines on the first poll; a sub-millisecond margin stays pending through its bounded poll loop).
deadline_tests.rs idle_expiry_precedes_ready_body_and_empty_chunks: the empty-chunk case had no negative control: its final 100+ ms advance expired the deadline even if empty chunks reset progress. The final advance after empty chunks now lands within one window of the last empty chunk, so that mutation fails.
Timer class not closed structurally: retry and deadline timers were created at four call sites, two calling the portable delay directly, and nothing rejected a new unfunded one. All now go through deadline::timer (counting's two branches collapse to one), and the qualification scan rejects any other construction.

Review fixes in round 3 (Greptile P1)

Denied calls blocked later recovery: a policy denial settles as ToolFailed without ToolStarted, and the reconciliation-only check (reconciliation_only && started.is_none()) returned Indeterminate before the failure-replay branch, so a later response-lost call in the same response was never reconciled and every retry stayed indeterminate. Sequential early returns are replaced by RecordedCall::classify (records.rs) plus one ordered match: settlements replay first whatever their start; a start-less completion, duplicate records or double settlement are Invalid; only an unstarted call stops a reconciliation pass (calls run in order, so nothing after it has started). Refusal replay uses the same classifier and refuses (Conflict) to settle a refusal over a dispatched or differently settled call.

Validation

Round 3 (Greptile P1), head 104b9561ba11da7a4b509254a2660055b8a28f84, via Q:/lanes/bs.ps1 (bb2): negative control with b3bc61a5a's tool_execution.rs fails both new tests; at the head reconciliation_scan 2/0.14 s; executor suite 105/3.11 s (stock_turn_emits_spans_without_prompt_or_tool_fields failed once in the first parallel run via the process-global span capture, then passed alone and in a full rerun of the same binary; no change touches spans); providers lib 249/0.43 s (the known main count-route failure only); e2e providers family 0.27 s; scripts/qualify-provider-stream.sh pass; Windows GNU --lib --tests harness (canonical feature set, 249 s) and providers pass; harness and providers wasm32 browser checks pass; rustfmt clean.

Round 2 (Claude), head b3bc61a5abdce60ffc780fe5715352b8283bbe14, one gate on bb2 via Q:/lanes/bs.ps1: rustfmt; scripts/qualify-provider-stream.sh (timer scan + fuzz compile, 2.0 s); providers lib 249 passed/0.50 s (known main count failure skipped); deadline/retry filter 12/0.012 s; main provider_failure_exit 5/0.67 s; e2e providers 27 scenarios/0.36 s; Windows GNU check (both crates' feature set, providers --lib --tests) 173.3 s; provider wasm/browser check 3.0 s. A first run with a 1 ns wake margin hung the e2e providers family on the Recorded millisecond clock; that run is the observed negative control for the millisecond margin.

Round 1:
Final review commit: c1e62955bdde70cb6b47349f5565ff6d5eabf9dc. Linux tests and all compile proofs ran through Q:/lanes/bs.ps1, routed to bb2. The production source did not change after the gate. Native test compilation occurred once (914.844 s); continuations reused the retained binaries.

Focused suite	Passed	Process wall time
Providers lib (known main count failure excluded)	247	0.812 s
Harness lib (six filters, 52 unique passing tests)	52	0.590 s combined
Main provider-failure/refusal integration	6	0.830 s
Task workflow	23	0.803 s
Provider scenarios	27	0.373 s
Error scenarios	58	0.137 s
Default-tool scenarios	75	1.251 s

Each scenario invocation also passed the runner's two Rust tests. Every runtime suite stayed under 10 s. The new shrinking idle/readiness and native retry properties passed in the providers suite; the native refusal allocation proof and model-request corpus/round-trip checks passed in the focused harness groups. Their negative controls ran too.

Compilation passed: both touched crates, --lib --tests, Windows GNU with the same filesystem-local/native-execution/test-support/grpc/mcp-http feature set (383.468 s); provider wasm/browser --lib --tests (108.726 s); locked standalone provider_stream fuzz target (90.568 s). Rustfmt, git diff --check and the removed-item caller scan passed. The unchanged browser timer runtime proof from the preceding pass is retained.

Earlier evidence remains in the 6cf49c018 follow-up and prior review.

The coordinator identified two unchanged main failures (items_examined 65537/65536): codex_stock_execution_counts_locally_without_a_backend_count_route and native_stock_model_original_lost_ack_uses_native_provider_and_settlement. They are outside this PR; the first is excluded from the focused provider gate. The second was encountered once by the model:: filter with exactly the known counter error; its 27 passing siblings were retained, and the gate continued without rerunning that group. No production limit or test was weakened.

Coordinator: add to merge gate

Run bash scripts/qualify-provider-stream.sh on the build box: it scans the providers crate for any timer created outside deadline::timer (or any platform timer outside the platform transports), then compiles the standalone provider_stream cargo-fuzz entry point with its locked workspace. Its parser properties and corpus already run in the providers lib merge suite; the model_request fuzz target is already compiled by scripts/qualify-ci.sh gate. No GitHub-hosted Rust job was added.

Decisions and overlap

Uniform supplied-history validation follows the coordinator decision recorded at 6cf49c018; a tool error is history, not evidence of a tool effect.
Idle deadlines measure the interval between consumed nonempty chunks, checked before every read; a delayed consumer can expire them even if bytes are now buffered. Timers only wake the next clock check.
MCP's scripted fixture disables automatic compaction because its oversized first exchange has no older summary source; compaction behavior is separate scope.
Approval code remains intact. #895 overlaps the tool-composition documentation and e2e local driver; the refusal assertions use stable diagnostic prefixes.