feat(runner): port broker long-poll websocket probe (v2.338.0)

BN
1d ago
+1012-3

Spec

broker-websocket-probe (v2.338.0 protocol sync). GitHub is instrumenting
broker websocket reachability ahead of moving delivery off HTTP long-poll;
run-service opts in per job via the variables map.

Upstream evidence (v2.338.0 checkout)

  • src/Sdk/DTWebApi/WebApi/WellKnownDistributedTaskVariables.cs:
    RunnerBrokerWebSocketProbeUrl = "system.runner.brokerwebsocketprobeurl"
  • src/Runner.Worker/JobExtension.cs (~L646, ~L1006, ~L1298):
    CheckBrokerLongPollWebSocketAsync spawns on the variable with the job's
    SystemVssConnection credential; _brokerWebSocketProbeToken.Cancel() +
    await at job end; result appended as ConnectivityCheck telemetry with
    message broker_websocket_telemetry:<json>.
  • src/Runner.Common/BrokerServer.cs: RunLongPollWebSocketProbeAsync /
    ConnectWebSocketAsync — connect, hold (count non-close messages as
    PingsReceived, echo text frames), reconnect with random
    10s–300s backoff (MinDelayForWebSocketReconnect/MaxDelayForWebSocketReconnect),
    never throws.
  • src/Sdk/DTWebApi/WebApi/BrokerWebSocketProbeResult.cs: field names and
    EmitDefaultValue semantics mirrored (Connected/Errors always emitted,
    rest skipped at default).

Implementation

  • client/broker.rs: BrokerWebSocketProbeResult uses the upstream camelCase
    JSON contract and omit-on-default serialization. The probe reuses the
    workspace's websocket stack with the runner's TLS roots, scheme-specific
    proxy CONNECT routing, and the existing dependency set.
  • worker/job_extension.rs: BrokerWsProbe — spawn gates on the exact
    variable key (absent or empty → inert), finish cancels + awaits and pushes
    the ConnectivityCheck/broker_websocket_telemetry: entry.
  • worker/job_runner.rs: spawn before remote action preparation so setup
    failures also collect probe telemetry; collect at job completion into
    rpt.connectivity_telemetry, which completejob already drains.
  • Malformed/non-websocket probe URLs return a setup error immediately; a
    cancellation observed before an attempt does not create connect-failure
    telemetry. Websocket payload text is never written to logs.
  • Inert by default: preloop never advertises the variable, so spawn
    returns None and the default path is untouched.

Notes / deviations

  • wss:// uses the configured HTTPS proxy and ws:// uses the HTTP proxy;
    loopback and matching no_proxy entries bypass the proxy. HTTPS proxy
    tunnels use the reporting client's TLS trust configuration.
  • The bearer is sent only to the server-provided system.* probe URL, as in
    the official runner. The variable is not workflow-controlled.
  • JobCompleted close reason: probe reports job_completed when cancelled
    while holding a socket, matching upstream.

Verification

  • On macstudio:
    cargo fmt --all -- --check — pass
  • On macstudio:
    cargo clippy -p preloop-runner --all-targets -j 4 -- -D warnings — pass
  • On macstudio:
    cargo test -p preloop-runner --lib -j 4 -- ws_probe — 5/5 pass
    (inert paths; connect+hold+echo+cancel; connect-failure telemetry; and
    missing SystemVssConnection token).

Summary by cubic

Ports the upstream broker long-poll websocket probe (v2.338.0) so preloop-runner opts in when the job message advertises system.runner.brokerwebsocketprobeurl. When present, a background task connects to the URL with the job's SystemVssConnection bearer, holds and reconnects with 10–300 s random backoff until the job finishes, and reports the result as ConnectivityCheck telemetry prefixed broker_websocket_telemetry:. The probe never affects the job outcome and is inert when the variable is absent.

Notes

  • Cancellation that races a connect attempt is counted as a connect failure (connect_failed), matching upstream.
  • A missing SystemVssConnection token or an unsupported probe URL scheme is reported as an error result, and teardown is built so job finalize cannot stall: echo writes race cancellation, the close handshake is bounded to 1 s, and a cancelled hold drops the socket immediately.
  • WS/WSS connects route through HTTP_PROXY/HTTPS_PROXY via an HTTP CONNECT tunnel; loopback addresses and no_proxy entries bypass the proxy. The probe shares the reporting client's TLS trust configuration, so a custom CA bundle applies to websocket connections as well.