PR #777 - Fooftilly/PRKS

Requested by Nikola · project thread

Summary

Before: after #769, backend/jobs/ could enqueue, claim and run one job, but no process ran jobs.

After: python -m backend.jobs.worker --root ROOT [--testing] is the prks worker process (the prks entry point itself comes with L-D-1, as for prks admin). This is PR 2 of 3 for L-J-1. PR 3 runs the same worker embedded in the web process.

worker.py: the Worker loop, one job at a time.
- Start (§4.4, §11.1). prks worker joins a root that web binds: while the root is missing or has no prks-root.json, it waits on the poll (worker_waiting reason=root_missing or root_unbound) and creates nothing there. Once the root is bound, it takes the root's shared root.lock for its whole run. Before claiming anything it runs the schema check (verify_schema), then root_pairing.start_process(…, bump=True) under the lease. A schema that is not_migrated or has a pending migration, or an unreachable database, is waited for on the poll like an unfinished setup, with nothing bumped. A newer_schema or a checksum mismatch exits 1. While setup is unconfigured or bootstrapping, it claims nothing and checks again on each poll. Once setup is ready, it reconciles if needed, bumps once and starts claiming. A non-retryable refusal (foreign root, stale copy, testing mismatch) exits with code 1 before any claim. A retryable one (a busy start key, a failed marker write) is retried on the poll, so a worker started after a failed web reconcile gets past the start key and reconciles.
- Wakeups. LISTEN prks_jobs runs on a dedicated connection, since a pooled one can be recycled. Every enqueue NOTIFYs at COMMIT. A poll (default 5 s) is the fallback, and the listener reconnects on failure.
- Idle tick. Each tick runs the reaper and, at most once per prune_every (1 h), enqueues jobs.prune. The prune job is deduplicated and singleton, so any number of workers run one prune at a time.
- Graceful stop. On SIGTERM or SIGINT the worker claims nothing more (a job whose claim returns after the stop is released without starting), and the running job may finish within stop_grace (30 s). Past that, the job is marked lost, then released: it is queued again at once without spending an attempt. The abandoned handler's later fenced writes are refused, and its end is never recorded as succeeded or failed, even when the release itself failed.
- Errors. An unreachable database for a claim, the reaper or the prune enqueue is counted and logged as a code, then retried on the next poll. It never stops the worker.
prune.py: the jobs.prune handler. It deletes succeeded and cancelled jobs that finished more than 7 days ago, in batches, each its own fenced transaction, with heartbeat(progress=…) after every batch, so even a run shorter than one batch reports what it deleted. failed, queued and running jobs are never touched.
diagnostics.py (§12.4): WorkerStats, the process-local counters (claimed, succeeded, retried, cancelled, failed, lost, released, reaped, errors), and queue_depth(db), the queued/running/failed counts and the oldest age by kind. Neither holds a payload, an error text or Library metadata.

Related issues

Refs #627
Refs #614

Scope / non-goals

No schema change: 030_jobs.sql and snapshot.json are untouched.
No server.py, prks_app.py or HTTP change. The embedded worker in web, PRKS_EMBEDDED_WORKER and a diagnostics endpoint are PR 3/3.
Until the cutover, the SQLite web process holds root.lock exclusively, so prks worker can't run beside it: it exits with root_in_use (tested).
Shared files: one entry each in the Pyright typed-slice config, the invariant scope list with its test, and the AGENTS.md layout line for backend/jobs/.

Decisions for review

jobs.prune also deletes old cancelled rows. §12.1 names only succeeded rows. Cancelled rows are history too (a twin did the work, or a restore superseded the job), and keeping them forever would let the table grow without bound.
The run's thread. A job runs on its own thread so a stop can give it a deadline. A handler past the deadline cannot be killed. Its lease is released and every later fenced transaction of its own is refused, and the process then exits.

Validation

Local (PostgreSQL 18 throwaway cluster via PRKS_PG_BINDIR=/usr/lib/postgresql/18/bin scripts/test-postgres):

tests.pg.test_jobs_worker: 30 tests, passing on repeated runs. The stop-race and stuck-listener tests each fail on the head before their fix. The previous signal handler hangs when a SIGTERM arrives while the main thread holds an Event lock, and the new handler-locks test covers that path. One of them starts a worker before its root exists: it waits through root_missing, root_unbound (the directory stays empty) and setup_pending, then runs a job once web has bound the root and bootstrap is done.
- A NOTIFY wakes an idle worker whose poll is an hour long. The poll claims a job inserted without NOTIFY.
- Jobs run one after another and are counted. An unreachable database is counted and retried.
- The idle tick reaps exhausted runs. Prune is enqueued once over many idle ticks, and not at all without its handler.
- Prune deletes only old succeeded and cancelled rows, across several batches, and records progress.
- Graceful stop: the running job finishes and nothing more is claimed. A job past the deadline is released (queued, attempt given back), reclaimed exactly once, and its abandoned handler's write is refused. A failed release still abandons the run. An abandoned handler that returns early or raises on run.lost leaves the row running with no attempt spent.
- A real SIGTERM to a worker process lets its running job finish, and the process exits 0.
- Queue depth holds no payload or Library id. Unknown counters and non-positive intervals are refused.
- prks worker started before bootstrap claims nothing while setup is unconfigured, then bootstrapping. Once ready it bumps exactly once (generation 2) and runs the job.
- A worker for a foreign root exits 1, claims nothing, prints no connection string and releases its lease.
- A worker after a failed web reconcile reconciles and bumps (generation 3).
- A retryable refusal is retried until it clears. A root held exclusively exits 1 with root_in_use.
- A worker whose schema has a pending migration waits without bumping, then starts and bumps once migrate has run. A database with a migration newer than this PRKS exits 1 and claims nothing.
Full scripts/test-postgres: 563 tests OK (after merging master at 79a7ad0). scripts/test-postgres --parity: OK.
Ruff on backend/jobs, tests/pg and scripts: clean. python scripts/check_invariants.py: OK. tests.test_engineering_invariants: OK. Pyright typed slice: 0 errors. Pyright pyrightconfig.json: 0 errors (3 existing pymupdf import warnings).
No browser E2E locally: nothing the served app loads changed. GitHub Actions runs the Test Gate and the Full E2E Gate.

Review fixes

Codex P2 (a stop raced a run that hadn't started its heartbeat): fixed in e4c8906.
Schema check before the start check (Nikola, P2): 82d2ff0.
Prune progress for runs shorter than one batch: 82d2ff0.
An abandoned run recorded as succeeded or failed (Nikola, P3): 9c788cc.
A job claimed after a stop started its handler (Nikola, P2): 4f5c933. It is now released at once without starting.
Codex P2s on 4f5c933: a mistyped root is never written to, and a stop interrupts the startup wait for the listener: 2556f2d. Then 5f41320 (Nikola, P3): an unbound or missing root is waited for rather than refused, so a worker that starts before web on a first run still waits (§13.3).
Two more stop edge cases (Nikola, P2): 9d2a806. A job's begin is atomic with stop(), and a listener stuck connecting can't hold a stop (connect timeout, bounded close). 6924447 (Nikola, P3): signal handlers only record the request. prks worker runs the worker on its own thread and the main thread calls stop() as ordinary code, so a SIGTERM can't deadlock on the stop events' internal locks.

Risks / follow-up

AGENTS.md conflicts with #775 on two adjacent layout lines. Whichever PR merges second keeps both lines.
PR 3/3: the embedded worker thread in web (after the same schema and start checks, no second bump), PRKS_EMBEDDED_WORKER, the diagnostics endpoint and the operator docs.

Checklist

Relevant tests/checks were run and are listed above.
Documentation/generated files were updated when required.
UI evidence is included when applicable.
No unrelated changes are intentionally included.

Generated by Claude Code

Summary by cubic

Adds the standalone job worker so queued PostgreSQL jobs run without manual claims. It verifies the schema before claiming and waits for pending migrations or unfinished setup; a missing or unbound root is waited for too, creating nothing, with the shared lease taken only once web has bound the root. Incompatible schemas and roots are refused.

Uses LISTEN with a polling fallback (its connect has a 10-second timeout), reaps expired jobs, and schedules jobs.prune on idle ticks; a stop interrupts a listener that cannot connect instead of waiting out the poll.
On shutdown, lets the running job finish within its grace period; otherwise marks it lost before release so its heartbeat stops and later writes are refused. A job claimed or started after the stop is released without starting, under a reentrant lock that makes the decision atomic with the stop and safe for a second signal.
Prunes succeeded and cancelled jobs older than seven days in batches, reporting progress after each batch; adds privacy-safe worker counters and queue depth diagnostics.

No schema changes. Until the PostgreSQL cutover, the SQLite web process holds root.lock exclusively, so a standalone worker cannot run beside it.