PR #689 - preloopdev/preloop

Stacks on #688 (fix/bound-tls-leftovers) → #685 (fix/bound-shared); retarget to main after they merge.

Closes the CLI half of the "one hung external dependency freezes the engine" audit (rows #15, #16, G11, plus the CLI side of row #12). Every outbound call the CLI makes on a path an operator or a script waits on now has a deadline, and every timeout error names the dependency and the deadline so a hang is distinguishable from a failure.

What was unbounded

Row	Location at 44d51b523	What now bounds it
#15	`preloop-cli/src/main.rs:517-530` `build_client()` — no `.timeout`, no `.connect_timeout`; used by `runner_capacity` (:4004), `fetch_run_job_logs` (:5498), `upload_git_bundle` (:4450), `latest_run_id` (:5685)	`ENGINE_CONNECT_TIMEOUT` 10 s + `ENGINE_READ_TIMEOUT` 600 s on the client, plus a per-call total through one `engine_request` funnel: `ENGINE_CONTROL_TIMEOUT` 30 s, `ENGINE_LOG_FETCH_TIMEOUT` 300 s, `ENGINE_BUNDLE_UPLOAD_TIMEOUT` 900 s
#16	`update.rs:138-141` (release client), `:731-736` / `:980-983` (releases + compare JSON), `:391-437` (`smolvm --version` / `machine create --help`), `:1285-1298` (`systemctl`)	`RELEASE_CONNECT_TIMEOUT` 10 s, `RELEASE_IDLE_TIMEOUT` 120 s, `RELEASE_METADATA_TIMEOUT` 30 s, `SMOLVM_PROBE_TIMEOUT` 10 s, `SYSTEMCTL_TIMEOUT` 30 s
#16	`app_manifest.rs:440-446` (manifest conversion)	closed by main's refactor before this branch landed: the conversion moved into `preloop_runner_server::github_app_setup::convert_manifest_code`, which posts through `shared_http::CLIENT` (30 s total, 10 s connect, `shared_http.rs:201-208`), so the CLI no longer builds a client of its own. The per-call timeouts I had added for the old in-CLI client were dropped in the rebase onto the stack head as redundant — the row's deadline exists, one layer down.
#16	`main.rs:1586-1589` (`cosign … .status()`, no deadline, inherits the terminal)	`COSIGN_VERIFY_TIMEOUT` 600 s
#16	`main.rs:686-700` (AgentENV guest probe loop: `aenv resume` + 20 probes)	`AENV_CONTROL_TIMEOUT` 30 s per resume/probe, `AENV_READY_DEADLINE` 120 s for the whole readiness wait
#16	`init.rs:2511-2584` (`docker`/`podman` build, save, inspect; the build/save path is sync inside an async fn)	`CONTAINER_BUILD_TIMEOUT` 3600 s, `CONTAINER_SAVE_TIMEOUT` 1800 s, `CONTAINER_INSPECT_TIMEOUT` 60 s (plus the `sysctl` hypervisor probe, `SYSCTL_PROBE_TIMEOUT` 10 s)
#16	`push.rs:108-146` (`git push`, `merge-base`, the `git_output` funnel; sync waits on an async path)	`GIT_CONTROL_TIMEOUT` 60 s, `GIT_TRANSFER_TIMEOUT` 1800 s, `GIT_STAGE_TIMEOUT` 300 s, plus every interactive credential path disabled
#16	`store_admin.rs:491-496` (`tokio_postgres::connect`) and `:576-580` (`SELECT pg_advisory_lock`, waits forever)	`CONNECT_TIMEOUT` 10 s on the driver config and around the handshake; `MIGRATION_LOCK_WAIT` 600 s via a `pg_try_advisory_lock` retry (2 s interval) that names the advisory-lock id and how to find the holder
#16	`server_install.rs:1427-1437` (`launchctl bootstrap` / `load -w`), plus the `systemctl` sites in the install/uninstall path	`LAUNCHCTL_TIMEOUT` 60 s, `SYSTEMCTL_TIMEOUT` 60 s
#16	`aenv_egress.rs:216-241` (`sudo -n`, `systemctl is-active`)	`SUDO_TIMEOUT` 60 s, `SYSTEMCTL_TIMEOUT` 30 s
G11	`app_manifest.rs`, `update.rs`	`update.rs` as above; `app_manifest.rs` moot (see the row above)
G11	`github_app.rs:1496-1505` `verify_repo_access_at`	no change needed — the audit row is stale. It builds its client from `shared_http::github_client_builder()`, which already sets `timeout(30 s)` + `connect_timeout(10 s)` (`crates/preloop-runner-server/src/shared_http.rs:201-208`), so all four probes it issues are bounded. I did not add a redundant per-request timeout; the row needs a correction instead of code.
#12 (CLI half)	`main.rs` `preloop serve` observability config: the OTLP exporter's blocking client still used reqwest's platform verifier (macOS `SecTrustEvaluateWithError` → trustd/securityd, no deadline)	one line, per the TLS slice's request: `.with_otel_client_builder(shared_http::with_tls_roots_blocking)`, so export TLS verifies against the shared in-process root set. Carried in this branch because `main.rs` is this slice's file; the API is the TLS slice's (#688).

Why these numbers

Client-level vs per-request. build_client() carries only the connect deadline and an idle-read bound, deliberately: preloop logs --follow and the run event feed are silent for minutes and last for hours, and a client-wide total would cut them. The idle bound is 600 s, above the engine's own 5-minute live-log liveness window (live_logs.rs IDLE_TIMEOUT = 300 s), so only a genuinely dead connection trips it. One-shot calls get a total through engine_request, which is also what the tests exercise.
Per-use sizing. Control ops (status/list/submit/cancel/rerun/push bookkeeping) 30 s; a durable log fetch whose body is fully buffered 300 s; a 512 MiB bundle upload 900 s. Long work gets generous bounds rather than none: image build 1 h, image save 30 min, git push 30 min, cosign verify 10 min, aenv resume+probe 30 s each with a 2-minute readiness budget. Nothing that legitimately takes minutes is cut short.
Non-interactive by default. Every bounded child gets stdin = /dev/null and kill_on_drop, so a prompt cannot become an invisible wait; git additionally gets GIT_TERMINAL_PROMPT=0, GIT_ASKPASS=false, SSH_ASKPASS=false, GCM_INTERACTIVE=never.
credential.helper is deliberately not cleared in push.rs, unlike the engine's snapshot fetches: preloop push pushes to the developer's own origin, where the OS keychain / ssh-agent helper is the authentication path, so clearing it would break every properly configured checkout instead of fixing a hang. The deadline is what kills a helper that blocks anyway. (Agreed with the git slice.)
Shared plumbing. The kill_on_drop + timeout + kill/reap policy now lives in preloop_observability::deadline (this branch stacks on it); this PR keeps only the per-caller *_TIMEOUT consts. store_admin.rs uses a local CONNECT_TIMEOUT = 10 s matching the Postgres slice's store_pg::CONNECT_TIMEOUT, kept local so the two land independently.

Tests (fail-before evidence)

main.rs::a_silent_engine_cannot_hold_a_cli_request_open — a listener accepts and never writes a byte, and a client built through the same constructor build_client uses must fail with a timeout. Before: no deadline existed at all, so this is the incident's exact client-side symptom.
main.rs::a_control_request_deadline_ends_a_stalled_response_body — headers arrive, the promised body never does; the client's idle bound is set far above the request total in the test, so only engine_request's total can end it. Proves the per-call deadline is load-bearing, not the client default.
init.rs::container_engine_calls_end_at_the_deadline — a fake container engine that sleeps 30 s is killed at a 300 ms deadline, and the error names the command and the deadline. Before: .status() waited the child out (30 s here, forever for a real hang).
push.rs::run_command_kills_a_child_that_never_exits — the never-exiting child is killed at 300 ms with git push + the deadline in the message; push.rs::git_children_are_non_interactive asserts the child environment is exactly 0|false|false|never.
update.rs::smolvm_probes_end_at_the_deadline_instead_of_hanging — a fake smolvm sleeping 30 s is reported unusable at 200 ms for both probes.
store_admin.rs::a_stalled_postgres_connect_ends_at_the_deadline — a silent TCP listener; the error names Postgres, the handshake and the deadline.
store_admin.rs::a_held_migration_lock_fails_instead_of_hanging — a second session holds pg_advisory_lock, and migrate_postgres_within must fail at a 500 ms lock wait naming the lock (SELECT * FROM pg_locks … hint). Ran green here against a local Postgres cluster; skipped when PRELOOP_TEST_POSTGRES_URL is unset, the repo's convention for optional database coverage.
The shared runners themselves are covered by preloop_observability::deadline's own tests (base branch), so this PR does not duplicate them.

Gates

cargo fmt --all --check → clean.
cargo clippy --locked -p preloop-cli -p preloop-runner-server --all-targets -- -D warnings → clean (0 warnings), run after the migration onto preloop_observability::deadline and again after the OTLP line.
cargo test --locked -p preloop-cli → 330 passed, 0 failed (317 unittests src/main.rs + 5 tests/no_client_autostart.rs + 8 tests/pre_push_hook.rs), including the six CLI tests above. Re-run after the rebase onto the current stack head.
PRELOOP_TEST_POSTGRES_URL=… cargo test --locked -p preloop-runner-server --lib -- store_admin → 4 passed, 0 failed, both new tests executing against a real local Postgres cluster (no skips).

Deliberately left, with reasons

preloop debug's REPL and preloop shell guest commands (guest_exec_command callers in debug_session.rs) stay unbounded: they are interactive sessions whose duration is the operator's. The row's item — the probe loop in main.rs — is bounded.
Non-daemon local helpers in server_install.rs (getent, useradd, groupadd, usermod, cp, chmod, chown, ln, plutil, setpriv) are untouched: no daemon can wedge them, and the row named launchctl.
store_admin's migration statements keep the server's own statement/lock timeouts: schema DDL and backfills must not be killed by the serving-path guards (agreed with the Postgres slice).
Stdio behaviour changed slightly where the shared runner captures output: run_systemctl/run_ok echo their captured streams right after exit instead of streaming live, and systemctl is-active preloop.service's state word is no longer echoed (its status logic is unchanged). launchctl's stderr is now surfaced in the failure message.
CLI GitHub calls are outside the engine, so send_observed/the breaker does not apply to them; they are deadline-bounded instead.
Stacked-branch note: this branch is based on #688 (which is based on #685), so the diff below is only this slice's commit. The preloop serve OTLP builder line at the top of this body is the TLS slice's API applied in my file, as agreed with that slice.
Rebased onto the updated stack head (33b35585a, which carries the TLS slice's rebase and main's newer app_manifest.rs/init.rs): the only conflict resolutions were keeping both sides (main's SshTarget/SSH_FORWARD_PORT and new tests alongside mine) and dropping the now-redundant app_manifest.rs timeout hunk, because main moved the conversion into the server crate on the already-bounded shared client.