ci: record what a hung test run is blocked on

Merged
SA
1mo ago
+22-1

About 5% of Test jobs hang. #8270 caps the cost at 10 minutes and #8286 streams the logs, which caught two hangs:

  • One froze in useDashboardSidebarShortcuts.test.ts, whose only test calls Bun.spawnSync. bun printed killed 1 dangling process at the 5 s test timeout and then never moved. This is the shape of oven-sh/bun#43697 (1.4.2, Linux, KVM guest: spawnSync spins and the test timeout cannot end it).
  • One froze a bun task and a node task within 120 ms of each other, in a file that spawns nothing.

The logs do not show what the stuck process waits on. This change records it:

  • At 345 s a background shell writes every thread's state, CPU, wait channel, plus each bun, node and turbo process's current syscall and open files, to a file.
  • At 360 s timeout kills the whole process group. The step fails with exit code 124 instead of running to the job limit.
  • The next step prints the file when the Test step failed.

A normal Test step takes about 70 s (p90 about 90 s), so 360 s does not touch a healthy run.

Verified on this PR with the timers set to 15 s and 40 s: the step ended at 40 s with exit code 124, no orphan processes, and the dump listed each process (bun test --isolate in do_epoll_wait, turbo in futex_wait_queue, bun x in do_wait).

This is diagnostic. Remove it when the cause is fixed.

Summary by CodeRabbit

  • Chores
    • CI test runs now have a time limit and provide additional diagnostic information when a run fails or is canceled. This helps make test-run issues easier to investigate; there are no user-facing feature changes in this update.