Carry a task command's cause in its indeterminate outcome

Merged
VA
11h ago
+740-97

Problem and design

Task-command failures were collapsed to a bare indeterminate outcome and trace-only diagnostics. Outcome::Indeterminate now retains an IndeterminateCause: unobserved completion, or a command kind with its failure code/text, unsettled report, or oversized input result. The worker persists the cause; joins, orchestration, observations, local callers and wire status preserve it. Command uncertainty remains ERROR_CODE_INDETERMINATE with the same operation identity and retry/native classifications.

cause is a required field of every indeterminate record, including Unobserved; there are no persisted pre-release records, so no compatibility decoding exists and a record without a cause is rejected. The workspace-wide caller scan also found and repaired the Pi recovery adapter's outdated outcome pattern/construction, which otherwise fails to compile. One borrowed formatter serves both caller errors and wire status, eliminating the temporary clone of every diagnostic string. No stores, protocol messages or approval paths are added.

Decisions and deletions

  • A missing or invalid cause fails decoding; nothing is read as Unobserved by default, so no reader can silently drop a cause.
  • Caller and wire text share a private formatter; no additional public formatting API is introduced.
  • Deleted the wire projection's diagnostic clone and the test's misleading description of a different input as a negative control.

Guarantee

  1. Class: G (erased failure causes), K (durable codec: required cause field), H (redundant owned copies), I (proof coverage). The typed cause is part of the shared outcome; the shared serde boundary requires the cause on every record, and every exhaustive outcome projection carries the field. A borrowed formatter is the sole indeterminate-message formatter for Error and wire observation. The shipping Pi recovery projection also moves the retained cause; its exhaustive match is covered by the qualification compile gate.
  2. Proof: scheduler::tests::indeterminate uses shrinking properties over arbitrary operation identities and Unicode diagnostic strings, enumerating every cause/reason variant. It covers completion event round-trips, the required cause field (every variant written explicitly, byte-identical canonical re-encoding, and rejection of records missing it), scheduler application/copying, native quota copies and owned-byte accounting, retention while parent joins await reconciliation, rejection of premature publications, wire text/identity/classification, and exact allocation savings against an independently initialized pre-fix wire implementation. The allocation oracle rejects the copying implementation. filesystem::task_runtime::tests::worker::indeterminate_command_keeps_its_cause drives all three command outcomes through real admitted dispatch, persisted settlement, host reopen, terminal projection, durable join, caller error and logs. Its negative control erases the diagnostic at the production dispatch boundary and requires that same detector's specific cause-loss failure. runtime::tests::tasks::indeterminate_live_join_keeps_command_causes covers the live error-to-outcome direction. The durable-host projection property ranges over all causes. The shared harness-fuzz/records cargo-fuzz target now round-trips outcomes, completion events and scheduler snapshots, with checked-in completion/failure/unsettled/input-limit/invalid-reason seeds. Negative control: reinstating a serde default on cause makes the missing-cause record decode, failing indeterminate_cause_is_a_required_round_tripping_field.
  3. Entry points: Ordinary/native filesystem task-command dispatch (Failed, Indeterminate, oversized ready result); durable settle_task, task outcome and terminal-status observations; scheduler completion replay, snapshots, quota copying and parent orchestration; live/durable runtime joins; bundle/local turn/direct command/child-drive conversions through into_error; Pi service recovery observation; wire operation_status and encode_error. The single outcome codec covers every persisted embedding; enum matching and the shared conversion preserve the cause at adapters. The shipping Pi recovery caller is compiled on Linux and Windows GNU with all its tests; reverting its cause field must produce the compiler's missing-field errors. Existing author validation covers unchanged bundle/local/client/task-volume adapters; the review's focused proofs cover the new field, persistence and projection boundaries. No synchronization primitive or concurrency algorithm changes.
  4. Gaps: none. The Guarantee matrix has no row assigned to #1020. This PR diagnoses uncertain command completion; it does not claim to fix the separately owned underlying hook, provider, child-lifecycle or filesystem failures.

Validation

Review fixes: 838f0661467519c8d8b24cc414ddaf6961466515, 4788fc0a303497430c0b4ce219d54345228256e1 (required cause field; round-2 gate: focused harness proofs 8/8 passed, 0.10 s, test build 8m56s; acyclic-harness --lib --tests Windows GNU compile 3m22s, passed).

Final build-box gate passed with filesystem-local,native-execution,terminal,tui,grpc,test-support,search-http (the same harness feature closure is used by the records fuzz target):

  • Focused harness proofs: 8/8 passed, 0.17 s aggregate reported test runtime; debug test build 3m12s. This includes the dispatch-erasure and allocation negative controls.
  • Instrumented records fuzz target: 64 runs passed, 0 seconds reported runtime, 14 checked-in seeds including the five added here; cold debug fuzz compilation 9m05s.
  • Shipping acyclic-plugin binary and tests: Linux compile 3m39s, Windows GNU compile 3m59s, both passed.
  • acyclic-harness --lib --tests Windows GNU compile: 5m39s, passed.
  • Qualification command wall time 1198.88 s, final harness Windows command 339.30 s. Runtime proof suites remain below 10 s; build times are recorded separately.

Pi compiler negative control: restoring the pre-fix shipping recovery pattern/construction produced E0027 and E0063 for the missing cause field in both binary and binary-test checks, as required (19.38 s command wall time). The probe restored the exact fixed source bytes in finally; no positive gate was rebuilt merely to recheck.

The author's unchanged-head broad results are retained: scheduler 27/27, durable_host 29/29, stock_turn 3/3, child_agent 2/2, task_volume 40/40, client 59/59, wire 43/43; remaining lib/e2e failures match main gate 129c0bf0f. Its Windows GNU check passed. The unrelated WASM acyclic_test_support resolution failure is also present in main's workspace gate.

The prior test tier asserted only an indeterminate variant and trace text. It missed persisted-schema evolution and did not mutate the production cause path; the new replay property and production-erasure control address those omissions.

Coordinator: add to merge gate

Add scripts/qualify-indeterminate-cause.sh to the build-box qualification gate so the expanded records fuzz target is compiled and exercised with its corpus and the shipping Pi caller compiles on Linux and Windows GNU. The harness unit proofs are also included in the existing all-features harness merge gate. No GitHub-hosted Rust job is added.