L-J-1 PR 3/3: embedded job worker in the web process and job diagnostics

FO
16h ago
+1326-31

Summary

Before: prks worker existed (#777), but nothing ran jobs in the process people actually start. Until the PostgreSQL cutover the SQLite web process holds the root exclusively, so a separate worker can't run beside it, and there was no way to see the queue.

After: with PRKS_AUTH set, python prks_app.py also runs one job worker thread, and the Library owner can read job diagnostics at GET /api/diagnostics/jobs. Without PRKS_AUTH (the SQLite runtime) there is no worker thread, /api/diagnostics/jobs answers 404 and psycopg is never imported. Only the static /api/openapi/job-diagnostics.json document is served there too, like the other slice documents. This is PR 3 of 3 for L-J-1.

How:

  • backend/jobs/embedded.py (new):
    • web_jobs(db, environ) reads PRKS_EMBEDDED_WORKER: unset or 1 prepares one EmbeddedWorker on its own worker pool (application_name prks-worker, 4 connections), 0 prepares none, and anything else is refused before a pool opens.
    • WebJobs.serve(httpd) starts the worker, runs serve_forever(), and stops the worker gracefully once serving ends.
    • When serving ends, the listening socket closes at once (new connections are refused), before the worker's grace period.
    • On the main thread, SIGTERM and SIGINT handlers only record the request (the #777 SignalStop), and a watcher thread calls httpd.shutdown().
    • Graceful stop is the #777 behavior: nothing more is claimed, the running job gets stop_grace (30 s), then it is released without spending an attempt.
    • A worker that is refused (for example a database newer than the code) or fails is logged as a code and shown in diagnostics. The web process keeps serving.
  • Start check. The web process already ran the §11.1 pairing check and bump before it opened the root, and it holds that root for its lifetime. So the embedded worker's start check (worker.schema_start_check) only verifies the schema: it waits while a file is pending and is refused when the database is newer. It never bumps a second time.
  • prks_app.start_jobs(auth) runs right after start_identity. It imports the jobs code only when auth is set, and exits 2 on a refused setting or an unusable worker pool. run_server(..., jobs=...) serves through WebJobs.serve when jobs is set; the SQLite path is unchanged.
  • Diagnostics (§12.4):
    • GET /api/diagnostics/jobs needs ADMIN_LIBRARY, so owners only: editors and readers get 403, and nobody signed in gets 401.
    • It returns the embedded worker's state (starting, waiting, running, stopping, stopped or refused), a reason code, and the process-local counters, plus queued/running/failed counts and the oldest age by kind.
    • A database that cannot be read answers 503 unavailable.
    • Contract: backend/api_contract/jobs.py (Pydantic, extra="forbid"), with the OpenAPI fragment at docs/api/openapi-job-diagnostics.json, also served at /api/openapi/job-diagnostics.json.
    • Both routes are on the log-safe route list.
  • Docs: docs/job-worker.md (operator notes for PRKS_EMBEDDED_WORKER, prks worker, shutdown and diagnostics), the AGENTS.md backend/jobs/ line, and a pointer in architecture §12.4.

Related issues

Refs #627
Refs #614

Scope / non-goals

  • No schema change.
  • No frontend client for the diagnostics. The OpenAPI fragment is not added to the TypeScript family manifest until a client needs it.
  • prks worker beside an SQLite web still refuses with root_in_use until the cutover; the embedded worker is the one-process path.
  • Shared files: one entry each in the Pyright typed slice, the invariant scope list with its test, and the AGENTS.md layout line. tests/pg/http_fixtures.py gains a jobs= argument.

Decisions for review

  • backend/db/migrate.py no longer skips browser E2E. The embedded worker's schema check imports it from the served app (behind PRKS_AUTH), and test_no_e2e_skipped_python_file_is_reachable_from_the_served_app correctly flagged it. It is removed from the postgres-migration-infrastructure rule; the .sql files and the snapshot tooling stay exempt.
  • Signal handling only when jobs run. SIGTERM/SIGINT handlers are installed only on the PRKS_AUTH path, so the SQLite runtime keeps today's Ctrl-C behavior.
  • A refused worker does not stop web. The web start check already decided the process may serve, so a worker problem shows in diagnostics and the log instead.
  • Diagnostics are owner-only. Queue depth spans the instance, though in 0.8 the instance has exactly one Library.

Validation

Local (PostgreSQL 18 throwaway cluster via PRKS_PG_BINDIR=/usr/lib/postgresql/18/bin scripts/test-postgres):

tests.pg.test_embedded_worker (22 tests):

  • Lifecycle. The worker starts with the server and runs a job while it serves. After shutdown() it reports stopping while the job finishes, then stopped, with the success counted. With stop_grace=0 the job is released (queued, counted released). With no worker, the server just serves.
  • No second bump. root_pairing.start_process is never called.
  • Schema. A pending schema file makes the worker waiting (pending) while the server serves, and it runs the job once migrate applies the file. A newer database makes it refused while the server keeps serving.
  • SIGTERM. A real SIGTERM to a child process serving through WebJobs.serve lets the running job finish, and the process exits 0.
  • PRKS_EMBEDDED_WORKER. Unset, empty or 1 turns it on. 0 opens no pool and reports embedded_worker: null. Other values are refused before a pool opens. prks_app.start_jobs exits 2 on a refused value or a missing worker URL, prints no connection string, and prepares the worker without starting a thread.
  • Endpoint, behind the real admission. The owner gets 200 with the queue by kind; the response is validated against the model and carries no payload text or Library id. Editors and readers get 403 and nobody signed in gets 401. An unreadable queue answers 503, and the SQLite runtime answers 404. The counters match diagnostics.COUNTERS and the contract.

Other local checks:

  • tests.test_api_contract_jobs: the checked-in OpenAPI artifact matches the generator, the bodies validate with openapi_core, unknown keys and states are refused, and the routes log as themselves.
  • tests.test_embedded_worker_off: a fresh interpreter importing prks_app and backend.server loads no psycopg and no backend.jobs module, and starts no thread.
  • tests.test_e2e_policy, tests.test_logging_privacy and tests.test_engineering_invariants pass.
  • Related PG suites pass: test_jobs_worker, test_jobs_runner, test_jobs_queue, test_http_session, test_http_admission, test_web_start and test_identity_hardening.
  • Full scripts/test-postgres: 602 tests OK.
  • Ruff is clean on backend, prks_app.py, tests/pg and scripts. Pyright typed slice: 0 errors. pyright: 0 errors (3 existing pymupdf warnings). python scripts/check_invariants.py: OK.
  • The default unit suite (python run_tests.py) could not run in this container because PyMuPDF is not installed. No browser E2E ran locally. GitHub Actions runs the Test Gate and the Full E2E Gate (this diff touches server.py, so the full gate runs).

Migration / compatibility impact

  • New env var PRKS_EMBEDDED_WORKER (default on), read only when PRKS_AUTH is set.
  • New /api/diagnostics/jobs and /api/openapi/job-diagnostics.json on the PRKS_AUTH runtime.
  • Without PRKS_AUTH, the only change is the static /api/openapi/job-diagnostics.json document.

Risks / follow-up

  • PRKS_EMBEDDED_WORKER=0 is only useful after the cutover, when prks worker can join a root beside web.
  • #627 criteria "worker graceful stop" and the §11.1 worker start check were delivered by #777; I'll tick them on the issue.

Checklist

  • Relevant tests/checks were run and are listed above.
  • Documentation/generated files were updated when required.
  • UI evidence is included when applicable. (No UI change.)
  • No unrelated changes are intentionally included.

Generated by Claude Code

Summary by cubic

Runs one job worker thread inside the web process on the PostgreSQL runtime (PRKS_AUTH set), so background jobs run in the process operators actually start. Library owners can now read worker state and queue depth at GET /api/diagnostics/jobs; the SQLite runtime is unchanged (no thread, 404 on the route, never loads psycopg).

New Features

  • PRKS_EMBEDDED_WORKER controls the worker: unset or 1 runs one thread on its own worker pool, 0 runs none for a separate prks worker, anything else exits 2.
  • The worker starts with the server and stops gracefully on shutdown or SIGTERM/SIGINT: no new claims, the running job gets stop_grace (30 s), then its lease is released without spending an attempt.
  • Once serving ends, the listening socket closes before the grace period, so new connections are refused at once instead of sitting in the backlog.
  • Its start check only verifies the schema — the web already ran the §11.1 pairing check and bump — so it never bumps a second time.
  • A refused or failed worker is logged as a code and shown in diagnostics; the web process keeps serving.
  • GET /api/diagnostics/jobs is owner-only (403 for editors/readers, 401 unsigned), returning the worker's state and counters plus queue depth by kind; an unreadable database answers 503.
  • The OpenAPI fragment is served at /api/openapi/job-diagnostics.json; operator notes live in docs/job-worker.md.

Migration

  • PRKS_EMBEDDED_WORKER=0 is only useful after the PostgreSQL cutover, when a separate prks worker can join the root beside web.
  • backend/db/migrate.py is served code now, so it is no longer exempt from browser E2E runs.

Generated by Claude Code