Inspect: skip oversized files before reading; never parse plain JSON for TODOs (#1337)

Merged
AF
1d ago
+670-50

The automatic TODO scan could read and fully parse very large generated files, several at once, just to look for comments. On one reporter's machine, 2.1 GB of generated JSON drove the daemon from 0.8 GB to 56 GB of heap (#1337).

What changes

  • Plain .json is no longer scanned for TODOs, because it can't contain comments. .jsonc is still scanned.
  • The TODO, metrics, complexity and duplicate scanners check a file's size before opening it. Files over 1 MiB are skipped unread and listed in skipped_files with the reason over the 1 MiB size limit, and the result reports complete: false.
  • Reads that grow between the size check and the read are cut at the limit.
  • These scanners share a 32 MiB budget for source being read, parsed or walked at once. Small files still run across the whole thread pool, and a single file is always admitted.
  • The inspect contribution cache version moves to 43, so results from the old unbounded scans aren't reused.

Measured on Linux, with 393 MB of synthetic JSON plus a small Rust source: peak RSS for the TODO scan fell from 625 MB to 73 MB, with the same TODO found. On AFT's own tree, the combined TODO and metrics scan is 10.7% faster than main (median of three interleaved rounds).

Gate: 520 inspect library tests, 377 inspect integration tests, 187 list-envelope tests, the inspect cache tests (after pinning version 43), formatting and the Windows check.

Not in this change: the JavaScript/TypeScript dead-code and unused-export readers (Oxc engine) still read whole files. That's a follow-up.

The skipped-file reason text was approved by the maintainer.


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Bounds memory use in the TODO, metrics, complexity and duplicate scanners so oversized generated files are skipped unread instead of fully parsed, which previously drove daemon heap from 0.8 GB to 56 GB on a 2.1 GB JSON fixture.

  • Files over 1 MiB are rejected before opening, reported in skipped_files with reason over the 1 MiB size limit, and mark the result complete: false.
  • Plain .json is no longer scanned for TODOs since it can't contain comments; .jsonc still is.
  • The scanners share a 32 MiB budget for source being read or parsed; small files still run across the whole thread pool and a single file is always admitted.
  • Reads that grow between the size check and the read are truncated at the limit.
  • The inspect contribution cache version moves to 43 so results from old unbounded scans aren't reused.

Peak RSS for a TODO scan on Linux fell from 625 MB to 73 MB, and the combined TODO and metrics scan is about 10.7% faster than main on AFT's own tree. The JavaScript/TypeScript dead-code and unused-export readers still read whole files; bounding them is a follow-up.