fix(github): subscribe new Apps to workflow_job (#625)

Merged
BN
6d ago
+260-34

Evidence

  • crates/preloop-runner-server/src/github.rs:4953-4986 adds workflow_job to the creation-time default event list.
  • crates/preloop-cli/src/app_manifest.rs:135-153,163-196 adds actions: read only when the generated event list contains workflow_job, so a manifest that subscribes nothing (no --public-url) or an operator who drops the event from PRELOOP_GITHUB_APP_DEFAULT_EVENTS does not widen the App; :752-770,802-826 asserts both directions.
  • crates/preloop-runner-server/src/github_app.rs:1593-1608,1641-1697 gates workflow_job on Actions permission and warns with the grant → installation acceptance → checkbox sequence.
  • docs/setup.md, docs/self-hosting.md, docs/github-app-webhook.md, docs/github-app-api-compat.md, and CHANGELOG.md document creation-time-only subscriptions and manual remediation for existing Apps — including Apps created without --public-url, which were shipped with no default_events at all — and point at App settings → Permissions & events → Subscribe to events, not the webhook edit screen.

Design

GitHub only accepts App event subscriptions at creation time; its App webhook configuration API cannot add events later. New manifests therefore request workflow_job and, only when that event is actually subscribed, actions: read (GitHub renders no checkbox for the event without the permission). Existing Apps are checked via GET /app; missing workflow_job is reported as permission-blocked until Actions access is granted and the installation accepts, after which the operator can tick the newly visible event in App settings. Consuming the event (turning queued into pool demand) is the pool work tracked separately in #625 — no adapter projects it into a run yet, which the webhook guide now states.

Reproduction (REVIEW.md §1a)

The PR's assertions in crates/preloop-cli/src/app_manifest.rs::tests::manifest_points_webhooks_at_a_public_url were applied to a checkout of main (66b68926d) and run against main's production code:

$ cargo test -p preloop-cli app_manifest::tests::manifest_points_webhooks_at_a_public_url
running 1 test
test app_manifest::tests::manifest_points_webhooks_at_a_public_url ... FAILED

failures:

---- app_manifest::tests::manifest_points_webhooks_at_a_public_url stdout ----

thread 'app_manifest::tests::manifest_points_webhooks_at_a_public_url' panicked at crates/preloop-cli/src/app_manifest.rs:763:9:
workflow_job can only be subscribed at App creation time: [String("push"), String("pull_request"), String("check_run")]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    app_manifest::tests::manifest_points_webhooks_at_a_public_url

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 253 filtered out; finished in 0.00s
error: test failed, to rerun pass `-p preloop-cli --bin preloop`

Main's manifest ships ["push","pull_request","check_run"] and no actions permission, so a new App never receives a workflow_job delivery — and GitHub exposes no API to add the subscription afterwards.

At this head the same command passes (test result: ok. 1 passed), as part of cargo test -p preloop-cli app_manifest:: → 8 passed; 0 failed, including
manifest_permissions_grant_actions_only_for_workflow_job (Actions read with workflow_job, absent without it) and
manifest_disables_webhooks_without_a_public_url (no webhook ⇒ no Actions grant).

Verification (REVIEW.md §7)

  • cargo fmt --all --check: clean.
  • cargo clippy --locked -p preloop-cli -p preloop-runner-server --all-targets -- -D warnings: clean (Finished dev profile ... target(s) in 8m 37s, no diagnostics).
  • cargo test -p preloop-cli: 255 passed; 0 failed (tests/pre_push_hook.rs: 8 passed), including app_manifest::'s 8.
  • Server-side subscription tests, cargo test -p preloop-runner-server --lib -- github_app::tests webhook_health: 36 passed; 0 failed, including required_trigger_events_are_real_github_webhook_events, missing_trigger_events_reports_only_absent_required_events, missing_events_record_whether_their_permission_is_granted, and the new missing_workflow_job_warning_leads_with_the_actions_grant (warning names the event, the actions grant, the installation-accept step, and the post-permission tick). The narrower -- manifest_default_events missing_workflow_job_warning_leads_with_the_actions_grant run is 3 passed.
  • uvx zizmor .github/workflows/: No findings to report. Good job! (13 ignored, 144 suppressed).
  • ast-grep scan --error (just sg-scan-strict) reports 20 error-level findings in config.rs, blob_store.rs, and preloop-orchestrator/src/lib.rs — files this PR does not touch. The output is byte-identical on main, so that gate is red repo-wide and not attributable to this change.
  • Workspace tests, control migrations, and the conform replay are the required CI checks on this head (rust shard 1-4 of 4, property-tests-fast, rust-lint, Server light conformance, Runner light conformance); they gate the merge.

Acceptance

New App manifests include workflow_job, and actions: read exactly when it does; startup health warnings identify the permission gate and exact manual App-settings remediation for older Apps.

Devin Review

Summary by cubic

Subscribes new GitHub Apps to workflow_job in the manifest flow and grants actions: read only when the event list carries that event, since GitHub renders the event's checkbox only with the permission. GitHub exposes no API to change event subscriptions after App creation, so older Apps need the event ticked by hand; the startup warning now spells out the grant → installation-accept → checkbox sequence. No adapter consumes the event yet, so deliveries are acknowledged without acting on them.

Refactors

  • The manifest and its permission decision now live in the shared github_app_setup module, so the CLI loopback flow and the engine-hosted --public-url flow make the same event/permission call, and the form page names the permissions the manifest actually requests.

Migration

  • Apps created before this change need workflow_job ticked under Settings → Permissions & events → Subscribe to events, after granting actions: read so the checkbox renders; delivery starts only once the installation accepts the permission change.

Summary by CodeRabbit

  • New Features
    • GitHub Apps created through the setup flow now subscribe to workflow_job by default when a public URL is configured, and request the Actions: read permission.
    • The setup page displays the permissions requested during App creation.
  • Improvements
    • Startup warnings now explain how to add missing event subscriptions and permissions, including when changes must be made manually in GitHub settings.
  • Documentation
    • Updated GitHub App setup guidance and configuration references to cover the new event, permission requirements, and manual setup for existing Apps.