fix(workspace-fs): keep idle watchers warm across workspace switches
HEAD: patiently:pr/fs-watcher-keep-alive → BASE: superset-sh:main
What & why
Refs #7877 (cause 1).
FsWatcherManager.subscribe() disposed a root's watcher as soon as its last listener left, so every workspace switch threw away the old workspace's watcher, and switching back crawled the whole tree again. On a monorepo that's ~23k inotify watches rebuilt per switch at 150–400% host-service CPU, and under steady switching the crawls stack into bursts of up to 377k.
A watcher whose last listener leaves now stays attached for 60 s, and a re-subscribe in that window reuses it. At most 3 idle watchers are kept, and beyond that the longest-idle one is disposed, so the idle cost stays bounded as the worktree count grows. Expiry, eviction and close() dispose exactly as before, and idleKeepAliveMs: 0 restores immediate disposal.
It builds on #7865's attach model rather than around it. Shared pending attaches, cancellation and the catch-up event are unchanged, and releasing a watch whose attach is still in flight still cancels it; only a fully attached watcher goes idle. A watcher is never handed back on a dead tree: a suspended one (root deleted) is released instead of parked, and before reuse the root's dev/ino/birthtime is compared with the one recorded at attach, because a root deleted and recreated while idle isn't always reported.
How I tested it
- New
watch-idle-keep-alive.test.ts(fake backend) covers:- reuse within the grace period, and expiry after it
- LRU eviction beyond the cap, and a re-subscribed watcher not counting against it
close()disposing idle watchersidleKeepAliveMs: 0- a subscribe racing an in-flight expiry
- a repeated unsubscribe
- a deleted root, and a root replaced while idle with no delete event
- The workspace-fs suite, lint, and the workspace-fs, host-service and desktop typechecks passed in CI on our fork: https://github.com/patiently/superset/actions/runs/36232163346. That run is
main2370f60d4 plus this PR, the hover PR and two small ignored-dir listing changes. - A/B on
main2370f60d4, same machine, scripted 30-minute workload (method in #7877). That fixed arm also included the hover PR and the two listing changes, which had no observable effect in that run:
| control | fixed | |
|---|---|---|
| host service incl. git (core-seconds per hour) | 3466 | 1878 (−46%) |
| host service above one core | 43% of samples | 20% |
| inotify watches p50 / max | 23k / 377k | 68k / 97k |
The median is higher by design (up to 3 warm workspaces); the bursts are gone. Nothing is user-visible, so there are no screenshots.
Checklist
- PR title follows conventional commits (
type(scope): subject) -
bun run lintandbun run typecheckpass (fork CI run above) - "Allow edits from maintainers" is checked on fork PRs. Not available: this PR comes from an organization-owned fork, and GitHub doesn't offer that option for those. Happy to push any changes you'd like.
Summary by cubic
Keeps watchers warm across workspace switches so switching back no longer re-crawls the whole tree. Previously dropping a root's last listener disposed its watcher immediately, so every switch rebuilt ~23k inotify watches at heavy host-service CPU; a watcher now stays attached for 60 s and is reused if re-subscribed in that window.
At most 3 idle watchers are kept, with the longest-idle evicted first, so idle cost stays bounded as worktree count grows. Expiry, eviction, and close() dispose as before, and idleKeepAliveMs: 0 or maxIdleWatchers: 0 restores immediate disposal. An idle watcher is never reused on a dead tree: suspended watchers are released instead of parked, and the root's dev/ino/birthtime is verified before reuse because a recreated root isn't always reported while idle.
Summary by CodeRabbit
- New Features
- File watchers can now remain active briefly after the last listener unsubscribes, allowing them to be reused if another listener subscribes during the grace period. The default grace period is 60 seconds and can be configured.
- Idle watchers are capped at three per manager by default; older idle watchers are closed when the cap is exceeded. Setting either limit to zero closes watchers immediately.
- Watchers are reused only while their subscriptions and watched roots remain valid.