PR #4934 - kdlbs/kandev

[!TIP]
PR walkthrough: Open the visual walkthrough

Remote executor plugins cannot own an environment safely today: the host gives them no workspace path, no repository intent, no proof the agent stopped before destroy and no durable record of a half-finished cleanup, so a crash or an archive mid-launch can orphan the remote resource. This adds those host capabilities behind negotiated, versioned capabilities, so a plugin that opts in can provision, connect, drain, destroy and recover without leaking, and plugins that opt into nothing behave as before.

Important Changes

Capability negotiation: a versioned host capability registry, echoed on profile validation; allocation is refused when host and provider disagree.
Workspace and repository: the workspace descriptor (path and identity) and repository intent are passed to provision.
Tunnel: a host-opened tunnel for agentctl control and API traffic, bound to installation, environment, generation and port.
Drain and cleanup: destroy requires a drain receipt. The task-environment cleanup job is the single authority that drains and destroys on archive. Claim-scoped cleanup_progress and recovery_progress checkpoints fence stale writers.
Drain contract table: the cleanup reason and drain outcome pairs a provider may rely on are listed in the system design. not_attempted is sent only with launch_failed and only when no agentctl handshake ever started; each cleanup claim acquisition has a unique id that the provider echoes on checkpoints; a bounded periodic sweep retries handoffs and retained cleanup_pending rows.
Durable bootstrap: durable bootstrap and guarded creating-to-ready publication across host restarts.
No orphaned resources: a resource a launch provisioned but could not record is stored in unrecorded_plugin_executors and destroyed by the cleanup under its own claim, so archiving during provisioning does not orphan it. That destroy carries the launch cleanup reason launch_failed and drain_outcome: not_attempted, the no-runtime evidence a launch rollback sends.
Handed-off resources: a cleanup_progress checkpoint may omit the workspace descriptor while none is recorded, so a provider can checkpoint before destroying a resource a launch handed off before binding a node. The periodic sweep leaves a target whose task has an active cleanup job to that job.
Profile snapshot: the recorded profile snapshot is sent on destroy, attach, inspect, resolve and the tunnel header, so a provider needs no process-local state.
Archive with a hung agentctl (plugin_remote only): the graceful agentctl stop and its lifecycle-lock wait are bounded (15 s or a third of the remaining archive budget), so a frozen agentctl no longer uses up the archive budget. A busy lock reports the stop as unconfirmed and the existing cleanup runs: drain uncertain, then the provider retains and may quarantine the node. Other runtimes keep the old path; no SDK or contract change.
Persistence: SQLite and PostgreSQL, with the same lock order (task row, then inventory lock, then execution row).
e2e-only: under the e2e mock profile a loopback or file:// repository remote is accepted, cloned without credentials and takes no Git credential lease, and agentctl's materialize endpoint accepts a host-less file:// locator, and the host forwards KANDEV_E2E_MOCK to the provider's agentctl; production is unchanged.
Destroy logging: every plugin executor destroy logs one structured line with the cleanup reason, drain outcome, claim, provider response and which unconfirmed condition fired. Every refused provider checkpoint logs its cause and the expected and received claim ids.
Design docs:
- docs/specs/executors/requirements/remote-executor-plugin-native-workers.md
- docs/specs/executors/system-design/remote-executor-plugin-native-workers.md
- docs/plans/executor-plugin-native-workers/plan.md (work orders 00-08, 17)

Validation

go test -count=1 (lifecycle with -race) over ./internal/plugins/... ./cmd/plugin-fixture/... ./internal/task/... ./internal/agent/runtime/lifecycle/ ./internal/orchestrator/executor/ ./internal/gitcredentials/ ./internal/repoclone/.
- At 8962b8bdeb, ./internal/agentctl/server/api/ ./internal/orchestrator/executor/ ./internal/task/service/ ./internal/repoclone/ ./internal/plugins/...: 7191 passed, 194 failed. ./internal/agent/runtime/lifecycle/ with -race: 5083 passed, 28 failed (counts include subtests).
- Every one of those failures also fails on the merge base 173d1ef2d8 (the same 194 and 28 tests, none new): host-dependent worktree, Kubernetes-prepare, alias and archive-reclaim, and repository-selection tests.
cd pluginsdk && go test ./...: 164 passed, 0 failed.
PostgreSQL lock-order tests were run on PG 15 only. CI covers 16 and 18. No PostgreSQL server was available for the latest fixups: their new SQL tests (handoff delete and transfer guard, list-all handoffs) ran on SQLite, and their 2 PostgreSQL variants were skipped, not run.
Laptop rig (real Nomad agent and the private Nomad provider plugin, e2e mock agent), on 8962b8bdeb: smoke, archive during provisioning (5 runs), restart during provisioning, and loopback http and file:// repository runs all pass, in dev mode and with TLS, ACLs and a scoped token.
Live rig results above were recorded at f257db1d49. The node launch wrapper commit (2e9d23fad6) is not covered by those live runs; its checks are the unit tests listed in the node-wrapper section.
Head f257db1d49 (this PR), same rig in secure mode: that 9-scenario suite passes; a frozen-agentctl archive returns in about 40 s (it returned 500 after 180 s on 8962b8bd), the provider quarantines the node about 70 s after the freeze, a second launch is refused, operator readmission works and the next task runs; one real Codex agent turn (codex-acp) passes end to end. The dev-mode suite passed one commit earlier, at c100d78791.
New lifecycle tests for the bounded stop fail on 8962b8bdeb and pass on the head. The lifecycle package has the same 25 failing tests (host-dependent, e.g. macOS temp-dir and symlink paths) on 8962b8bdeb and on the head; none fail only on the head.

Node launch wrapper (added on this branch)

An operator-owned launch wrapper, set per node and not per task, can be required for every agentctl agent and utility launch. It is generic: core knows only a wrapper argv prefix and a required flag, and no sandbox product name appears in core.

Specs: docs/specs/agents/requirements/node-launch-wrapper.md and docs/specs/agents/system-design/node-launch-wrapper.md. The "no executor-level wrapper" exclusion in generic-launch-wrappers.md is narrowed to point at them.
Behaviour: the node wrapper is the outermost argv. A profile wrapper cannot clear or bypass it. A required node wrapper that is missing or invalid refuses the launch, with no unwrapped retry.
Checks: owner and mode checks on the file (read from the opened descriptor), on each ancestor directory and on argv[0], which must be an absolute path. The owner is set by flag, default is the agentctl process uid.
Coverage: managed agent process, one-shot args, interactive passthrough, ACP utility prompt and probe, OpenCode model listing, Codex app-server utility. Not covered on purpose: user shells, scripts and piped processes, MCP and helper processes.

Restart fix (follow-up message after a backend restart).
After a survivable backend shutdown, a follow-up message to a running plugin_remote task got no reply. Two causes:

Recovery asked the surviving agentctl for its delivery stream by session ID, but the peer's stream is named by incarnation and generation, so it answered 409 OWNER_MISMATCH and recovery failed every 60 s. Fixed by reading the peer's own stream for a legacy execution with no pinned stream, and checking session, incarnation and generation match (114a85bc8b).
The plugin_remote agentctl had no durable journal, so every recovered execution was legacy: published Running, never Ready, and a later launch hit cannot configure while agent is running. It now gets a journal under the provider-declared workspace, like Kubernetes (f7860bcaf9). The plugin's workspace removal deletes it with the environment. No SDK, proto or other-runtime change.

Live (local attach, mock agent, branch build): message right after restart replied at about +61 s (was none in 90 s); message 70 s after restart replied in 1.0 s (was session FAILED); default suite 9/9.

Limits: environments created before this change stay on the legacy path until recreated; the first reply waits for the 60 s provider retry; the durable-bootstrap re-create has no journal; only the mock agent was tried.

Worker-side API URL for the readiness probe (7530eb2735).
After provisioning, core probes the host API through the worker. On a remote worker (ssh node) the host URL 127.0.0.1:<kandev port> is not the host, so the probe failed with api_unreachable although the provider's forward was up. A provider can now return an optional worker_api_url (proto field 11, additive) and core probes that URL when set, else host_api_url as before.

Negotiated capability worker_api_url v1, requires api_url_forward. A provider that did not negotiate it and sends the field is refused; one that does not opt in behaves exactly as today.
Validation: http, loopback host (127.0.0.1, ::1, localhost), a port, no userinfo, path only /, no query or fragment. Only for provider_forwarded launches. Agentctl still forces the ping path.
Spec: REQ-EXECUTORS-NATIVE-WORKER-009 (new file, the native-workers file is at its size limit), plus capability-versioning and manifest docs. Open: whether sdk-release.json needs a change_kind bump.
Live (laptop, ssh rig, Nomad agent blocked from reaching Kandev): before, smoke failed api_unreachable; after, smoke replied in 8.1 s; unblocked control passed; default suite 9/9; ssh-restart pass. A patched plugin on the old core is refused at install (unknown capability), an unpatched plugin on the new core behaves as before.

Failed-launch drain fix (f70b18657d). Found on a real ssh node: agentctl died at startup and the node was quarantined. A provider checkpoints artifact_staging and bootstrapping from inside its provision call, so the launch's stored phase was ahead of the launch's local copy; the drain outcome was written with the stale phase and the store refused the move back to allocating, so the fallback cleanup_pending write ran and the provider never got the launch_failed destroy. The drain outcome is now recorded as cleanup_pending with the retained resource from the start. Tests: a failed provision after the provider checkpointed rolls back as absent; no handshake gives not_attempted, a handshake without a credential gives unreachable, a handshake with a credential gives drained. Local rig (agentctl exits 1 at startup): node no longer quarantined with this fix and the plugin fix; default suite 9/9, restart-message pass. Open: core's own handshake failing because agentctl died still reports unreachable.

Allocation directory token for the node wrapper (dd5007f704, 9f3c1ceb2a, 14853d2601). A node wrapper policy may use {{kandev.alloc_dir}}, a per-launch directory owned by the node's launcher and outside the agent's writable workspace. agentctl resolves it only from --node-launch-wrapper-alloc-dir (never from the environment or a request), requires an absolute, resolved, existing directory owned by the process owner, not group or world writable and not inside the workdir, and refuses a policy that uses the token without a value. REQ-AGENTS-NODEWRAP-006 AC .1 to .6; the node-launch-wrapper.v1 capability is unchanged (an older agentctl rejects the unknown flag and the token, so it never starts unwrapped). Tests cover token placement, each refusal, the value never coming from a request or the environment, and the unused-value case. Live (local rig): a recording wrapper received the directory; default suite 9/9, restart-message pass. Not proven: the real sandbox runner on a real node.

drain_receipt@2: a failed launch no longer quarantines a clean node (c7e889b40c, b815d79db3, f425b5a158, 9653ec926f, plus pinning tests 06755defc3). not_attempted now means the host never held an agentctl credential for this runtime. Gated by a new negotiated drain_receipt@2 (registry Min 1, Max 2; @1 hosts and records behave exactly as before; the version comes from the record's stored capabilities). Under @2: no handshake ever started plus no token gives not_attempted for any cleanup reason (was launch_failed only); a handshake that started but never produced a recorded credential (EOF, deadline, connection refused) also gives not_attempted, via two new sticky envelope fields credential_recorded (set in the same checkpoint that names the token secret, in both the launch and the durable-recovery paths) and handshake_rejected. A rejected handshake stays strict (unreachable), as do a recorded credential whose secret is unreadable, a non-durable provider after a started handshake, and any record without the new fields. The plugin still verifies the job, terminal allocations, node contract and worker state; only the drain is skipped. Preconditions are stated in the spec: agentctl starts no agent, shell or managed process before an authenticated request when a bootstrap nonce is configured (with none, authentication is off on loopback), and its only startup process is a code-server orphan sweep (pgrep, may SIGKILL an orphaned code-server host-wide; recorded as a known hazard, not changed). New spec files remote-executor-plugin-native-workers-drain-receipt-v2.md (requirements and system-design); the D1 table, AC 005.4 and capability-versioning are updated; the proto change is comment-only. Pinning tests prove the premises: only documented routes accept requests without a token, spawning routes start nothing unauthenticated, the wrapper is validated and never executed at startup, and the token reference is checkpointed before CreateInstance in both paths. Lifecycle failures equal base (the same 24 environment tests).

Decided on this branch:

Ancestor directories must be owned by the configured owner or root and not group or world writable. There is no sticky-bit exception.
argv[0] must be an absolute path, so PATH cannot redirect the wrapper.
The kandev backend's own passthrough runner refuses an agent launch when the agentctl it adopted advertises node-launch-wrapper.v1. Limit: a wrapper the backend was never told about is not detected, and that runner behaves as before (AC-AGENTS-NODEWRAP-005.4).

Not covered, follow-ups:

ACP client terminals (commands an agent runs as tools). The agent process is wrapped, but these tool commands are not. Covering them needs the node wrapper plumbed into the ACP client and terminal manager.
A host-side check of node-launch-wrapper.v1 before dispatch. The worker already fails closed, so this is a clearer error and defense in depth, not a bypass.

Tests: new tests in launchwrapper, process (node_wrapper_test.go), utility (node_wrapper_test.go), api and cmd/agentctl, including the composition-order test TestNodeWrapperIsFirstArgvOfSessionCommand. Run locally: -race passes for the new tests. TestGenericWrapperPrepare fails on this branch and on f257db1d49 alike (macOS /var vs /private/var), so it is pre-existing and not changed here.

Commits: the spec is a separate commit. The code and tests are one commit (26 files), because the per-commit lint needs the implementation to compile with the tests. The PR is one logical change, but it is large; splitting is possible if reviewers ask.

Possible Improvements

Medium risk: persistence and lock-order changes. Recorded secrets are revealed to the provider on every attach, inspect, resolve and tunnel-open call, because the provider needs its cluster credential each time. A lease-scoped credential would narrow that.

Checklist

If I do not have repository write access and this is a large architectural change, I discussed the direction in a linked issue before opening this PR.
This PR contains one logical change; unrelated work is split into separate PRs.
I have performed a self-review of my code.
I have manually tested my changes and they work as expected.
My changes have tests that cover the new functionality and edge cases.
If my change touches UI files (apps/web/), I have added or updated Playwright e2e tests in apps/web/e2e/ and verified them with make test-e2e.
I checked whether this affects public docs in docs/public/** and updated them or noted why no docs change is needed.

Preview Environment


URL	https://kandev-pr-4934-bwo7.sprites.app
Commit	`f274c85`
Agent	Mock agent

Updates automatically on each push. Destroyed when the PR is closed.