ci: test what a change reaches, not everything on every push

Merged
AL
2d ago
+136-32

Closes #224

CI now tests what a change reaches instead of everything on every push.

What a run tests

EventPackagesFirefox and WebKitSlow browser files
Pull requestthe ones it changes and their dependentswhen the player, the runtime or the demo changedleft out
Push to mainthe same, from where main wasthe sameincluded
Weekly (Mon 05:00 UTC), by handallyesincluded
Only design/, docs/, root .md, LICENSEno runno
  • A new scope job reads the change: pnpm's ...[<commit>] filter for packages and apps, plus the author package and apps/skills for skills/. Any other file outside the packages that is not docs (the lockfile, a workflow, the root's package.json) tests everything.
  • pnpm typecheck still covers every package: each package's check writes the types the next one reads, and it takes seconds.
  • The slow files are stalling-host, worker-update and large-media, listed in the player's vite.config.ts and left out under H5P_TEST_SKIP_SLOW. They take 115 s of test time in Chromium and 120 s in Firefox. Main's run after the merge has them, so design/testing.md now says to wait for it before publishing the player.
  • The VS Code job runs when the extension, the player, the runtime or the libraries changed.
  • to-scorm is not among the slow files: it runs on every pull request that tests apps/skills, being the one check of the h5p-to-scorm skill's text against the embed README.

Testing

  • The scope step, simulated on a fresh shallow clone with a fake commit per kind of change, run as a pull request's run:

    ChangeTestedEnginesVS Code
    packages/authorauthor, demo, skillsnono
    skills/h5p-author/SKILL.mdauthor, skillsnono
    design/author.md plus packages/authorauthor, demo, skillsnono
    packages/playerplayer, verify, embed, author, extension, demo, skillsyesyes
    packages/runtimethe same, and the runtimeyesyes
    apps/demo pagedemothe app's tests onlyno
    packages/embedembed, demo, skillsnono
    packages/verifyverify, author, demo, skillsnono
    packages/cmi5cmi5, demonono
    packages/vscodethe extensionnoyes
    pnpm-lock.yaml, another workfloweverythingyesyes

    It found one bug, fixed before this commit: pnpm counts a file outside the packages as the root project's, and the root's test script tests everything. The step now leaves the root out.

  • vitest list --project browser: 36 files, and 33 with H5P_TEST_SKIP_SLOW=1; the three named are the ones missing.

  • The player's typecheck passes. The workflow parses, with each job's needs and if as intended.

Not verified

  • This pull request's own run tests everything, because it changes a workflow, and leaves out the three slow files, as any pull request's run does. Selecting fewer packages shows up first on the next pull request after the merge.
  • A push to main fetches the commit main was at, to compare with. GitHub serves a fetch by commit; a forced push falls back to testing everything.

Release

None: no package's published files change.


Summary by cubic

Closes #224: CI now tests what a change reaches instead of everything on every push. A new scope job picks the packages a change touches and their dependents via pnpm's ...[<commit>] filter. Pull requests test only those, skip the player's three slow browser files, and run Firefox and WebKit only when the player, runtime, or demo changed. Pushes to main run the same selection including the slow files; weekly and manual runs test everything. Docs-only changes (design/, docs/, root markdown, LICENSE) start no run.

  • pnpm typecheck still covers every package, in seconds.
  • The VS Code job runs when the extension or its dependencies changed.
  • Non-doc files outside the packages (lockfile, workflows, root package.json) test everything.

Testing

  • Scope selection verified on a fresh clone with a fake commit per change kind; one bug found and fixed: pnpm counts such files as the root project, whose test script tests everything, so the step now excludes the root.
  • This PR's run tests everything (it changes a workflow) and skips the slow files as any PR does.

Release

  • No published files change.