fix: preserve cancellation ownership during finalization

Merged
BO
1mo ago
+526-2

A job could still accept cancellation after its backend had finished, while its worker copied artifacts or published terminal metadata. Close cancellation under the existing registry lock before that cleanup, and keep the worker registered until cleanup finishes. Running jobs remain cancellable when their metadata has disappeared.

Reject overlapping starts with the same explicit job ID within one Supervisor instance, including pending starts and finalizing workers. The admission check is atomic and happens before store or worktree writes, preventing a worker from changing a replacement job's cancellation state after metadata loss.

This fixes an existing lifecycle race surfaced by CI for #15. Deterministic tests cover terminal outcomes, migration, metadata loss, concurrent admission and error cleanup. The 147 affected tests, Ruff, wheel/sdist build and release artifact audit pass locally; all five code CI jobs pass on 9d46a0f.

Summary by Sourcery

Close job lifecycle races by making finalization and job-ID admission atomic with respect to cancellation and worker registration.

Bug Fixes:

  • Reject cancellation once a job reaches terminal finalization, preventing late cancellation from changing completed, failed, upstream-error, or cancelled outcomes.
  • Preserve cancellation for genuinely running jobs even when their persisted metadata is missing.
  • Release registered workers and capacity slots after finalization or startup failures.

Enhancements:

  • Reserve job IDs during startup to prevent reuse across concurrent starts, metadata loss, and delayed worker startup while retaining structured validation for invalid IDs.

Tests:

  • Add deterministic coverage for cancellation during terminal cleanup, missing metadata, startup and finalization failures, job-ID reservation, concurrent starts, and retry behavior after failed starts.

Summary by cubic

Closes lifecycle races around job finalization and admission. Cancellation is now rejected once a job starts finalizing under the registry lock — including during backend-output copying and early failures — while running jobs stay cancellable even when their metadata is gone. Job IDs are also reserved at start so a concurrent or retried start with the same ID can't replace a registered worker.

Bug Fixes

  • Reject cancellation once finalization begins, without affecting cancellation of still-running jobs with missing metadata.
  • Reserve job IDs during start to reject same-ID starts even when metadata is lost or the worker thread hasn't launched.
  • Ensure finalization and start failures release the worker handle, capacity slot, and reserved job ID.
  • Add deterministic tests covering cancellation during terminal cleanup, metadata loss, concurrent starts, and terminal-write failures.

Summary by CodeRabbit

  • Bug Fixes
    • Cancellation requests are rejected once a job enters finalization, preventing its terminal status from changing.
    • Jobs can still be cancelled when their saved metadata is missing.
    • Worker capacity is released if saving a terminal status fails, allowing subsequent jobs to start.
    • Concurrent attempts to start a job with the same ID are prevented, while startup failures leave the ID available for retry.
    • Backend failures remain reported as failures even when job output files are copied.