feat(mothership): run scheduled jobs as durable unattended chat turns
Merged
WA
7d ago+34815+34815+01234567890123456789012345678901234567890123456789-119-119-012345678901234567890123456789
Summary
Restores the runtime for scheduled agent jobs on the TypeScript worker. The schedule tick claims each due job occurrence and runs it as a durable, leased chat turn in the job's own chat. There is no creation surface yet (no API, UI or agent command), so this PR ships the runtime only, and the lane has nothing to run until that surface lands.
- Unattended turns are durable leased chat turns (
request/lifecycle/unattended.ts,chat/application/admit-unattended-turn.ts). They are admitted like a typed send: the user message, the run and its recovery intent commit together under the chat lock. The new recovery kind isunattended_stream, and abandoned-run takeover drives one on after its Sim process dies, keeping it non-interactive and mounting only the secrets the job recorded. The headless wake lifecycle cannot do this: it holds no controller lease, so nothing can continue its run once the pod is gone. - Occurrence = run. In one transaction the claim records the occurrence (
job_execution_logs,queued) and advances the schedule. The occurrence id is a UUIDv5 of the schedule and the instant, and it is the turn's message, stream and run id. Two ticks, or a reconciler retry, therefore admit the turn at most once. A reconciler re-dispatches lost dispatches, finalizes ended turns exactly once under a row lock, and stops a run at its six-hour limit through the same Stop delivery a person's Stop uses. - Budgets:
- a per-run spend cap: settled cost is tracked per billing request, which survives takeovers, and is answered with a
job_run_budgetcard; - three failed runs in a row turn the job off and notify the job's people;
max_runs;- a 15-minute minimum cadence;
- an owner who loses access turns the job off.
- a per-run spend cap: settled cost is tracked per billing request, which survives takeovers, and is answered with a
- Outcome is a field on the run record (
met,not_met,blocked), not a tool call scanned out of a response. - Schema: migration
0437adds the occurrence columns plus three indexes, builtCONCURRENTLY. Script migration0036archives (never deletes) every job row the retired Go runner wrote. It commits per batch and is idempotent.
Why this runs without a flag
- Nothing can create a job. No route, Copilot tool, CLI command or UI writes a
source_type = 'job'row: schedule deploys insert workflow schedules only, and the job create paths were removed earlier. - Legacy rows are archived before the new tick runs. Script migration
0036runs in the migrations step, after the SQL migrations and before the app starts. The claim only reads rows that are unarchived and not disabled or completed. - An idle tick costs one indexed probe. With no live jobs, the lane runs one query,
jobLaneWorkQuery. Its twoEXISTSlegs probe the partial indexesworkflow_schedule_due_job_idxandjob_execution_logs_open_idx. It claims nothing and reconciles nothing. An integration test seeds 2,000 archived jobs and 5,000 finished logs, then asserts no claim, no reconcile, no sequential scan and fewer than 20 buffer reads. - Missing prerequisites mean no work at all. Without
COPILOT_API_KEYorREDIS_URLthe lane returns before any query (unit test). - Why approval holds can come later. Production-effect approval holds for unattended runs are not part of this PR. Until they exist, an unattended run could do anything its owner can. That is safe only because no job row can exist. The holds must land before any creation surface.
Companion: simstudioai/mothership#818
Type of Change
- Bug fix
- New feature
- Breaking change
- Documentation
- Other: ___________
Testing
lib/mothership/request/lifecycle/unattended.integration.ts(real Postgres and Redis): one unattended turn survives three controller deaths mid-run and finishes once. It ends with one user message, one assistant message and one run row, and every recovered leg stays non-interactive with the job's secrets.lib/mothership/jobs/lane.integration.tscovers:- concurrent claims recording one occurrence;
- a lost dispatch retried once;
- the death of the Sim process mid-run, finalized exactly once;
- one-shot jobs and
max_runs; - busy-chat skips and lost owner access;
until_complete;- the three-failure streak;
- the spend cap;
- the time-limit Stop, whose test fails on the previous stop path because the run never settles;
- the idle-tick probe.
lib/mothership/jobs/lane.test.tschecks that the lane makes no DB calls when its prerequisites are missing.packages/db/script-migrations/0036_archive_legacy_scheduled_jobs.integration.tscovers batching, idempotency, and keeping completed jobs completed.- Reviewers should focus on the shared-path edits in
start.ts,recovery-config.ts,recover-stream.tsandadmit-turn.ts: the recovery kind is now explicit, and the chat-move step is shared with typed sends.
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing
- No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)