#1184 fix(core): prune dependency trees during startup scans

Issue for this PR

Closes #1183

Type of change

• Bug fix
• New feature
• Refactor / code improvement
• Documentation

What does this PR do?

Recursive startup scans were walking the whole project tree—including node_modules, .git, and generated output—and only filtering matching files afterward. On a dependency-installed monorepo, the two startup MCP scans alone consumed about 12.93 CPU-seconds per launch.

The root cause was that Glob.Options declared no ignore field and the wrapper did not pass one to glob. This PR restores that contract so callers can prune subtrees during traversal.

The final design uses two policies:

• Glob.DEPENDENCY_IGNORE: dependency, vendored, VCS, virtual-environment, and tool-cache trees. Content discovery uses this so authored SQL, DDL, and favicons under directories named build, dist, out, or target remain visible.
• Glob.DEFAULT_IGNORE: the dependency set plus generated/output directories. MCP discovery uses this stricter policy because discovered configuration can select local commands or remote endpoints.

Applied at four scan sites:

• Datamate MCP transport discovery
• External MCP config discovery
• Project favicon discovery
• Default altimate-code check SQL/DDL discovery

Both MCP consumers retain a defence-in-depth result filter. Datamate now derives that filter from the shared policy and normalizes relative paths to forward slashes before matching, removing its duplicated POSIX-only directory list.

Review fixes included

This revision addresses every validated review comment, including the late vendor-policy warning:

• Output-directory favicons remain eligible while dependency favicons are pruned.
• Authored SQL/DDL under output-like directory names remains eligible while vendored SQL is pruned.
• Datamate traversal and post-filtering share one broad policy.
• vendor/** is now part of the narrow dependency policy and remains inherited by the broad policy; shared glob, default SQL/DDL, and favicon regressions cover it.

A direct Datamate regression makes malicious node_modules, build, and dist configs lexically earlier than a safe authored config and proves that only the authored transport wins.

Performance evidence

Measured on a 16-core M4 with this repository and dependencies installed:

Scan	Before	After
syncDatamateUrlFromVscodeMcp	10.32 CPU-s	0.07 CPU-s
discoverExternalMcp	2.61 CPU-s	0.07 CPU-s
Combined	12.93 CPU-s	0.14 CPU-s
Raw **/mcp.json glob	6.24 CPU-s	0.06 CPU-s

Whole-process development serve startup dropped from 21.3 CPU-s to 4.1 CPU-s before idle.

Verification on final head

Final head: 22b3fdacadfb2b2b1237dc245b550bf5948f8bef

• bun test test/util in packages/core: 35 passed, 0 failed.
• Project/check suites before the late policy correction: 180 passed, 0 failed. On final head, the directly affected suites pass 38 project and 89 check-command tests.
• MCP/Datamate suites: 56 passed, 3 skipped, 1 todo, 1 known pre-existing home-config isolation failure. Every directly relevant MCP/Datamate regression passed.
• bun run typecheck: 13/13 tasks passed, including the push hook.
• Targeted lint across all eight touched files: 0 errors.
• Council consensus: ship after a direct Datamate consumer regression; that gate is satisfied.
• Codex Security full-PR diff scan 7fb0f303-9139-4dff-88fb-bc42e00ea9d5: complete coverage through a3d21a957c, 0 findings. Supplemental final-head scan 3cb61974-5908-4d90-962f-0a52478bc944 covers a3d21a957c..22b3fdacad, complete coverage, 0 findings. Marker Guard, require-markers, and typecheck pass on final head.
• All review threads are resolved.

Known baseline limitations

• No release binary was rebuilt for the after measurements.
• No concurrent Linux reproduction was rerun for this PR.
• Performance measurements are macOS/arm64; the pruning mechanism is platform-independent.
• Full-repository lint currently reports 5,870 warnings and one unrelated baseline error, so touched files were linted separately.
• One release-validation test leaks the developer's real home MCP configuration; the same documented baseline failure remains unrelated to this diff.

Screenshots / recordings

Not a UI change.

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR
• I have addressed and resolved all actionable review comments
• I have completed consensus and security review

Summary by CodeRabbit

• New Features

  • Added standard exclusion patterns for dependency, version-control, build, and virtual-environment directories.
  • Added support for custom exclusions during recursive file scans.

• Improvements

  • SQL, DDL, MCP configuration, and favicon discovery now skips dependency folders while preserving authored files in output directories.
  • Ignored subtrees are pruned earlier for more accurate and efficient scans.

• Tests

  • Added coverage for custom exclusions and consistent synchronous and asynchronous scanning behavior.
  • Added validation for discovery across dependency, generated, and authored configuration files.

---

[!NOTE]
Medium Risk
MCP and Datamate discovery still gate which local commands run; behavior is tightened with shared ignore policies and retained post-filters, but any glob policy mistake could miss or expose configs.

Overview
Restores ignore on Glob.Options and forwards it to glob, so recursive **/… scans prune node_modules, .git, and build trees during traversal instead of walking the full tree and filtering matches afterward. That behavior had been dropped from the wrapper, which made startup MCP scans very expensive on dependency-installed monorepos.

Introduces shared Glob.DEPENDENCY_IGNORE (deps/VCS/tool caches only) and Glob.DEFAULT_IGNORE (deps plus dist, build, target, etc.), with patterns shaped as /**/dir/** so subtrees are pruned, not post-filtered.

MCP discovery (Datamate transport + external discoverExternalMcp) now passes DEFAULT_IGNORE into the scan and keeps a Glob.match post-filter so symlink edge cases still cannot surface vendored commands. Datamate drops its duplicated POSIX path-segment filter in favor of the shared policy.

Content discovery uses the narrower set: default check SQL/DDL globbing and project favicon scans skip dependency/vendor trees but still see files under output-style directory names (build/, out/, …).

Adds core glob regression tests plus consumer tests for check discovery, MCP exclusion, favicon selection, and Datamate transport when malicious configs appear earlier in sort order.

^{Reviewed by Cursor Bugbot for commit 22b3fdacadfb2b2b1237dc245b550bf5948f8bef. Bugbot is set up for automated code reviews on this repo. Configure here.}