feat(cli): add init workflow readiness checks

Merged
BN
7d ago
+2169-29

What changed

  • Added the stable-URL question to the App credentials step. It accepts --public-url or a valid PRELOOP_PUBLIC_URL without prompting, otherwise explains Tailscale Funnel, Cloudflare named Tunnels, and reverse proxies before offering a blank-to-skip prompt. URLs must be HTTPS and non-loopback, and trailing slashes are normalized. Skipping states plainly that push/PR webhooks and PR checks will not work.
  • Added the workflow readiness step after credentials/golden. It scans local .github/workflows/*.yml|yaml and .github/actions/**/action.y*ml, fetches non-local --repo targets through GitHub when a configured App/PAT can read them, reports secrets.NAME and bracket references with file locations and scope, excludes engine-minted secrets, and offers the existing hidden preloop secret set NAME --repo OWNER/REPO path in an interactive terminal. Missing secrets never fail init.
  • Added the runs-on report beside the secret report. It parses string/list/object forms and dynamic expressions, uses the engine's real runner matching and golden-selection rules, reports base/declared/hosted-Linux/external-OS/undeclared/dynamic/group cases, and emits an exact PRELOOP_RUNNER_LABELS=... line when custom labels are missing.
  • Renumbered the wizard to five steps, added --no-secrets-scan, and added JSON objects for credentials URL, secrets, and runner labels.
  • Updated docs/cli_reference.md and CHANGELOG.md.

Example fixture output

Against a temporary clone of preloopdev/python-starter-template:

{"step":"secrets","decision":"missing","result":{"stored_in":"config file","targets":[{"repo":"preloopdev/python-starter-template","source":"checkout","workflow_files":2,"action_files":1}],"secrets":[{"name":"OP_SERVICE_ACCOUNT_TOKEN","reads":[".github/workflows/build_and_publish.yml:24"],"configured":null,"command":"preloop secret set OP_SERVICE_ACCOUNT_TOKEN --repo preloopdev/python-starter-template"}],"missing":[{"name":"OP_SERVICE_ACCOUNT_TOKEN","repo":"preloopdev/python-starter-template"}]}}
{"step":"runner_labels","decision":"ok","result":{"labels":[{"label":"ubuntu-latest","verdict":"hosted_linux","severity":"ok"}]}}

Against preloopdev/AgentENV, the real run reported OCR_LLM_AUTH_TOKEN, OCR_LLM_MODEL, and OCR_LLM_URL as missing, and reported ${{ matrix.os }} as dynamic plus ubuntu-24.04/ubuntu-latest as hosted Linux.

Verification

  • cargo test -p preloop-cli — 250 passed.
  • cargo clippy -p preloop-cli --all-targets -- -D warnings — passed.
  • cargo fmt --all -- --check — passed.
  • Real binary, isolated config/home, AgentENV fixture — JSON workflow scan and preflight completed successfully.
  • Real binary, isolated config/home, python-starter-template fixture — found OP_SERVICE_ACCOUNT_TOKEN, emitted the repo-scoped set command, and completed preflight.
  • Real binary --no-secrets-scan --json — emitted a skipped secrets object and still emitted the runner-label object.
  • Real binary invalid --public-url http://127.0.0.1:9090 — emitted JSON error and exited 2.

Not verified: an actual browser/App-creation flow, live GitHub App/PAT remote workflow fetch, or an engine-backed interactive secret write. Those paths reuse the existing setup/secret command implementations; unit tests cover URL validation, local scanning, scope matching, label classification, JSON shaping, and dynamic/object runs-on parsing.

Devin Review

Summary by cubic

Adds workflow readiness checks to preloop init: App auth now asks for a stable HTTPS webhook URL, and a new step scans workflow files for missing secrets and reports every runs-on label against the engine's actual matching rules.

  • The public URL question is scoped to App auth only; with a PAT the flag is ignored. On a re-run with kept credentials, --public-url points to preloop setup github --via app --public-url <URL> instead of being applied.
  • The wizard is renumbered to five steps; --no-secrets-scan skips the secret scan (the runner-label report still runs).
  • Missing secrets are listed with exact preloop secret set commands; the scan never fails init because a repo can't be read or a secret is missing.
  • JSON output now includes credentials URL, secrets, and runner label decisions.

Summary by CodeRabbit

  • New Features
    • Added a preflight step to preloop init that scans selected repositories for missing workflow secrets and checks workflow runner labels against engine pool rules.
    • Missing secrets can be configured interactively; reports include suggested commands and runner-label guidance. Use --no-secrets-scan to skip the secrets scan while retaining runner-label reporting.
    • GitHub App setup can now collect a stable public HTTPS webhook URL. JSON results include URL details and workflow scan decisions.
  • Documentation
    • Updated the CLI reference and changelog with the new wizard step, scan behavior, URL requirements, and related options.