PR #769 - Fooftilly/PRKS

Requested by Nikola · project thread

Summary

Before: prks.jobs existed (PG-2, 030_jobs.sql), but nothing could enqueue, claim or run a job.

After: backend/jobs/ implements the §12.1 runtime on that table. The running SQLite app doesn't import it yet. This is PR 1 of 3 for L-J-1. PR 2 adds the prks worker process and PR 3 adds the embedded worker in web.

queue.py:
- enqueue / enqueue_library_job run inside the command's transaction, so the job commits only with its change (the outbox). A Library job records the epoch its library_command checked. NOTIFY prks_jobs is delivered at COMMIT only, and a coalesced enqueue (dedupe hit) notifies nobody.
- claim is the single-statement claim/reclaim from §12.1. A 23505 on jobs_singleton means the worker lost the race, so it claims again. Any other error is raised.
- heartbeat, complete, fail and release all match id, lease_id and state = 'running', so a stale worker updates nothing.
- fail sets failed when attempts are exhausted (checked first), else cancelled when a queued twin exists, else queued with exponential backoff. A 23505 on jobs_dedupe repeats the statement.
- reap fails expired runs that have no attempts left, which frees their singleton key.
- last_error is a short code (an exception type or PRKS error code), never a message.
handlers.py: the registry that maps a kind to its handler. Each lane registers its own kinds; this PR registers none.
runner.py: run_job runs one claimed job with a heartbeat thread and records the outcome (succeeded / retrying / cancelled / failed / lost).
- Every transaction a handler opens is fenced. A Library job writes through library_command(run.context()), which is RequestContext.for_job plus L-ID-1's lease and epoch checks. An instance job writes through JobRun.command, which first locks the job row with its lease FOR SHARE.
- If the lease is lost, nothing is recorded and the result is discarded.

Related issues

Refs #627
Refs #614

Scope / non-goals

No schema change: 030_jobs.sql and snapshot.json are untouched.
No server.py, prks_app.py or HTTP change.
No job handlers.
No worker loop, LISTEN, graceful stop, prks worker start check, jobs.prune or diagnostics. Those are in PR 2/3.
Shared files: each gets one entry for backend/jobs (the Pyright typed-slice config, the invariant scope list with its test, and the AGENTS.md layout list). #628 (lane B) will likely add lines next to them. Whichever PR merges second rebases and keeps both.

Decision for review

release (used by graceful stop in PR 2) returns the job to queued and gives back the attempt its claim counted, because a stop is not a crash. A job killed without a release still spends its attempt, as §12.1 requires.

Validation

Local (PostgreSQL 18.x throwaway cluster via PRKS_PG_BINDIR=/usr/lib/postgresql/18/bin scripts/test-postgres):

tests.pg.test_jobs_queue: 25 tests. tests.pg.test_jobs_runner: 11 tests. All passed, five consecutive runs.
Each #627 criterion in this slice has a test:
- A worker SIGKILLed after claiming and committing its write leads to lease expiry, then a 4-way race in which exactly one worker reclaims, then an idempotent rerun and completion.
- After a stale worker loses its lease, its heartbeat, complete, fail and release update nothing.
- Repeated crashes end the job failed (lease_expired) and free its singleton key.
- A handled failure with a queued twin cancels A and B runs once. Without a twin, A is requeued with backoff.
- A twin that commits during the failure update forces a real 23505 on jobs_dedupe, which the code retries into cancelled.
- A failure on the last attempt ends failed, never queued, and frees the dedupe key.
- A held singleton claim forces a real 23505 on jobs_singleton. The loser retries and claims another key with no error, and a two-worker stress run never has two jobs of one key running.
- Jobs of a Library in maintenance or of an older epoch are not claimed.
- A restore during a running job refuses the job's later command and completion.
- Error messages and paths never reach last_error or the logs.
I checked that the two forced-race tests are real: with the 23505 handling removed, both fail with the raw 23505.
Full scripts/test-postgres: 491 tests OK. scripts/test-postgres --parity: OK.
ruff check --config ruff.toml prks_app.py backend + new tests: clean. python scripts/check_invariants.py: OK. Pyright typed slice: 0 errors. Pyright pyrightconfig.json: 0 errors.
Default-suite modules touched by the shared files (tests.test_engineering_invariants, test_pg_facade_registry, test_e2e_policy, test_dependency_gate): 320 tests OK.
I didn't run browser E2E locally because nothing the served app loads changed. GitHub Actions runs the Test Gate and the Full E2E Gate. Their results will be reported separately on this PR.

Risks / follow-up

PR 2/3: worker loop with LISTEN/NOTIFY and poll fallback, the idle-tick reaper, graceful stop, prks worker with the §11.1 start check under the shared root.lock (waiting while setup is pending), jobs.prune and diagnostics counters.
PR 3/3: embedded worker in web (the only server.py change in either lane).

Checklist

Relevant tests/checks were run and are listed above.
Documentation/generated files were updated when required.
UI evidence is included when applicable. (No UI.)
No unrelated changes are intentionally included.

Generated by Claude Code

Summary by cubic

Adds backend/jobs/, the PostgreSQL job runtime on prks.jobs (§12.1), so jobs can now be enqueued, claimed, and run. The running SQLite app doesn't import it yet. This is PR 1 of 3 for L-J-1 (#627): PR 2 adds the prks worker process, PR 3 the embedded worker in web.

queue.py enqueues inside the caller's transaction (the job commits only with its change; NOTIFY prks_jobs fires at COMMIT only), claims and reclaims in a single statement with retry on a lost singleton-race 23505, and fences heartbeat/complete/fail/release on lease_id plus state = 'running'. fail, release and the reaper end a Library job whose epoch moved on as cancelled (a restore), never back to queued; a Library in maintenance is not stale. fail also ends the job failed when attempts are exhausted, cancelled when a queued twin exists, else queued with exponential backoff; the reaper fails expired runs with no attempt left, freeing singleton keys. last_error holds a code, never a message.
runner.py runs one claimed job with a heartbeat thread and records the outcome only while the lease is held — a lost lease discards the result. Handler transactions are fenced: instance jobs lock the job row with their lease FOR SHARE and re-read the lease with the lock against clock_timestamp() (a single FOR SHARE computes its output before waiting on the row lock); Library jobs write through JobRun.library_command, fenced by the lease and the epoch recorded at enqueue. heartbeat() inside a fenced transaction is refused at once; progress is reported between transactions. A non-positive lease or heartbeat interval raises ValueError before the handler runs.
handlers.py is the kind-to-handler registry; each lane registers its own kinds, none yet.

Scope

No schema change and no job handlers; the worker loop, LISTEN handling, graceful stop, and jobs.prune are PRs 2/3.
A graceful stop's release returns the job to queued without spending the attempt (a stop isn't a crash); a kill still spends it.
The typed-slice Pyright config, the invariant scope list and AGENTS.md each gain the new files; lane B's PR (#628) adds lines next to them — whichever merges second rebases and keeps both.

Summary by CodeRabbit

New Features
- Added a PostgreSQL-backed job processing runtime with job queuing, handler registration, and execution.
- Jobs support deduplication, prioritized claiming, retries, and worker leases to prevent stale workers from committing changes.
- Library jobs are fenced against outdated library state.
Documentation
- Documented the job runtime and its responsibilities, including that it is not yet used by the running application.