feat(mothership): run scheduled jobs as durable unattended chat turns

Merged
WA
7d ago
+34815-119

Summary

Restores the runtime for scheduled agent jobs on the TypeScript worker. The schedule tick claims each due job occurrence and runs it as a durable, leased chat turn in the job's own chat. There is no creation surface yet (no API, UI or agent command), so this PR ships the runtime only, and the lane has nothing to run until that surface lands.

  • Unattended turns are durable leased chat turns (request/lifecycle/unattended.ts, chat/application/admit-unattended-turn.ts). They are admitted like a typed send: the user message, the run and its recovery intent commit together under the chat lock. The new recovery kind is unattended_stream, and abandoned-run takeover drives one on after its Sim process dies, keeping it non-interactive and mounting only the secrets the job recorded. The headless wake lifecycle cannot do this: it holds no controller lease, so nothing can continue its run once the pod is gone.
  • Occurrence = run. In one transaction the claim records the occurrence (job_execution_logs, queued) and advances the schedule. The occurrence id is a UUIDv5 of the schedule and the instant, and it is the turn's message, stream and run id. Two ticks, or a reconciler retry, therefore admit the turn at most once. A reconciler re-dispatches lost dispatches, finalizes ended turns exactly once under a row lock, and stops a run at its six-hour limit through the same Stop delivery a person's Stop uses.
  • Budgets:
    • a per-run spend cap: settled cost is tracked per billing request, which survives takeovers, and is answered with a job_run_budget card;
    • three failed runs in a row turn the job off and notify the job's people;
    • max_runs;
    • a 15-minute minimum cadence;
    • an owner who loses access turns the job off.
  • Outcome is a field on the run record (met, not_met, blocked), not a tool call scanned out of a response.
  • Schema: migration 0437 adds the occurrence columns plus three indexes, built CONCURRENTLY. Script migration 0036 archives (never deletes) every job row the retired Go runner wrote. It commits per batch and is idempotent.

Why this runs without a flag

  • Nothing can create a job. No route, Copilot tool, CLI command or UI writes a source_type = 'job' row: schedule deploys insert workflow schedules only, and the job create paths were removed earlier.
  • Legacy rows are archived before the new tick runs. Script migration 0036 runs in the migrations step, after the SQL migrations and before the app starts. The claim only reads rows that are unarchived and not disabled or completed.
  • An idle tick costs one indexed probe. With no live jobs, the lane runs one query, jobLaneWorkQuery. Its two EXISTS legs probe the partial indexes workflow_schedule_due_job_idx and job_execution_logs_open_idx. It claims nothing and reconciles nothing. An integration test seeds 2,000 archived jobs and 5,000 finished logs, then asserts no claim, no reconcile, no sequential scan and fewer than 20 buffer reads.
  • Missing prerequisites mean no work at all. Without COPILOT_API_KEY or REDIS_URL the lane returns before any query (unit test).
  • Why approval holds can come later. Production-effect approval holds for unattended runs are not part of this PR. Until they exist, an unattended run could do anything its owner can. That is safe only because no job row can exist. The holds must land before any creation surface.

Companion: simstudioai/mothership#818

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: ___________

Testing

  • lib/mothership/request/lifecycle/unattended.integration.ts (real Postgres and Redis): one unattended turn survives three controller deaths mid-run and finishes once. It ends with one user message, one assistant message and one run row, and every recovered leg stays non-interactive with the job's secrets.
  • lib/mothership/jobs/lane.integration.ts covers:
    • concurrent claims recording one occurrence;
    • a lost dispatch retried once;
    • the death of the Sim process mid-run, finalized exactly once;
    • one-shot jobs and max_runs;
    • busy-chat skips and lost owner access;
    • until_complete;
    • the three-failure streak;
    • the spend cap;
    • the time-limit Stop, whose test fails on the previous stop path because the run never settles;
    • the idle-tick probe.
  • lib/mothership/jobs/lane.test.ts checks that the lane makes no DB calls when its prerequisites are missing.
  • packages/db/script-migrations/0036_archive_legacy_scheduled_jobs.integration.ts covers batching, idempotency, and keeping completed jobs completed.
  • Reviewers should focus on the shared-path edits in start.ts, recovery-config.ts, recover-stream.ts and admit-turn.ts: the recovery kind is now explicit, and the chat-move step is shared with typed sends.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)