[JSC] Less memory per function that never runs, and code that can be dropped and decoded again

Merged
JA
1mo ago
+5947-465

15 commits on cf1b36ec8703. Each commit is one feature, builds on its own, and carries its
tests; they are ordered by dependency and can be reviewed (and reverted) one at a time.

Why

A large bundled CLI application (ESM, ~1,800 modules in ~900 chunks, standalone executable with an embedded bytecode cache)
sits at an idle prompt with ~100 MB of anonymous memory of which JSC code structures are the largest part, and at ~170 MB
after a session. Almost all of it belongs to code that never runs or ran once:

  • 30,000 top-level function declarations are instantiated at module link time; 83% are never read.
  • 12,700 LLInt-only CodeBlocks carry 15.3 MB of metadata for 4.2 MB of bytecode: 37% call link records of call sites that
    ran at most once, 23% value-profile predictions nobody reads.
  • Every UnlinkedCodeBlock carries value/array profile arrays that only the DFG ever benefits from.
  • Unlinked code decoded from the cache is never dropped, although decoding it again costs ~2.5k instructions per function
    against ~111k for a re-parse; module bodies keep their linked and unlinked code although they run once.

Nothing here changes behaviour observable from JavaScript. Every feature has an option (all default on unless stated) so it can
be switched off individually.

Commits

1. A ModuleProgramExecutable keeps one module environment symbol table for its whole life

A fix for the base, found while testing this series against module loaders (#522), and the invariant the rest relies on.

  • Bug: records of several loaders share an executable, its declarations' FunctionExecutables and their optimized code; that is
    sound only because every environment is made from the executable's ONE symbol table, so that the second environment
    invalidates SymbolTable::singleton() (the DFG folds the scope of resolve_scope closure variables through the table in the
    metadata, and of imports through topLevelExecutable->moduleEnvironmentSymbolTable()). ScriptExecutable::clearCode()
    cleared the table and getUnlinkedCodeBlock() made a new one (and a new declarations vector) when it generated the code
    again. A record that had made its environment but not run yet when all code was deleted (waiting on a dependency suspended
    at a top-level await) regenerates the code; the next loader adopts the executable with the new table; functions compiled
    after that bake the newest loader's exporting environment into code the earlier loaders run too: an imported binding read
    in loader a returns loader d's value (JSTests/stress/module-loaders-share-one-environment-symbol-table.js fails on
    cf1b36ec8703). With lazy declarations (commit 5) the same split would also hand a record executables specialised on
    another record's environment, hence a RELEASE_ASSERT(environment->symbolTable() == executable->moduleEnvironmentSymbolTable())
    on the lazy read path there.
  • Fix: the table and the declarations vector are made from the executable's first unlinked code and stay (an environment keeps
    its table alive anyway). Code fetched again is asked for in the code generation mode of the first (the environment's layout
    depends on it); getOrMakeExecutable only adopts an executable whose mode is the one the record would ask for. An executable
    whose code was deleted is still not adopted, and no longer sits in the clearable-code set for the sake of its table.
  • Audit of everything specialised on "the" module environment in shared code (closure-var and ModuleVar resolve_scope
    folding, tryGetConstantClosureVar, GetLazyClosureVar's fast case, FunctionExecutable::singleton()): all reduce to the
    one-table rule; details in the commit message.

2. Allocate an UnlinkedCodeBlock's value and array profiles when it first reaches the Baseline JIT

Options::useLazyUnlinkedValueAndArrayProfiles

  • What: m_valueProfiles/m_arrayProfiles (two FixedVectors) become one ButterflyArray (ValueAndArrayProfiles) allocated when a
    BaselineJITPlan is first created for a CodeBlock of the unlinked code. The number of value profiles is derived
    (numParameters + metadata.numValueProfiles), the number of array profiles is kept as a 32-bit count in existing padding.
    Builtins never get them. The bytecode cache no longer stores the value profile count (format revision 5 -> 6; this is the only
    bump in the series, commits 3 and 5-7 change the format too and stay on 6).
  • Concurrency: the arrays are allocated on the main thread; the pointer is published with a release store and the collector
    and compiler threads that fold profiles (CodeBlock::updateAll*Predictions) load it once per fold with acquire and skip
    folding when it is null. Freed only in ~UnlinkedCodeBlock.
  • Numbers: synthetic app (2,000 modules x 25 functions): 4.5 MB of the 6.0 MB the first three commits save.

3. Move UnlinkedCodeBlock's out-of-line jump targets into its RareData

An empty hash map per code block for a case few blocks have. With commit 2: sizeof(UnlinkedCodeBlock) 208 -> 192, one size
class down for all unlinked code blocks (static_assert in release, 64-bit, non-Windows builds). No new RareData on the
synthetic app (902/3009 before and after).

4. Index the bytecode cache's two-character atom table by identifier character class

512 KB per VM (zeroed, so all touched) -> 32 KB: 64 x 64 classes ($ 0-9 A-Z _ a-z); other pairs use the verified
direct-mapped cache the three-character names use.
Commits 2-4 together, synthetic: RssAnon 184.4 -> 178.4 MB; startup instructions -0.5..-0.8%, exercise -1.0..-1.6%.

5. Instantiate module function declarations on first read

Options::useLazyModuleFunctionDeclarations (predictFunctionForUnprofiledLazyClosureVarForTesting for tests)

  • What: InitializeEnvironment leaves the module environment slot of a heap-allocated function declaration empty. The function
    object and its FunctionExecutable are created when the binding is first read: get_from_scope with the new LazyClosureVar
    resolve type (LLInt, Baseline, LOL: load, test for empty, slow path; DFG GetLazyClosureVar, FTL lowering with a slow path
    call), or JSModuleEnvironment::readVariable() for by-name lookups, module namespace objects, WebAssembly imports and the
    debugger. ResolvedLazyClosureVar is what BytecodeGenerator emits for a module's own declarations; the linker turns
    ModuleVar and closure variables that are declaration slots into LazyClosureVar. An empty slot that is not a declaration
    is still TDZ. The type/control-flow profilers keep the eager path.
  • Module loaders (#522): records that share a ModuleProgramExecutable share the declarations' FunctionExecutables
    (linkedFunctionDeclaration / linkFunctionDeclaration next to functionDeclaration); the record that reads a declaration
    first links it, each record fills its own environment with its own function objects. Whether a slot is a declaration slot
    depends only on the module's code, so every CodeBlock of an UnlinkedCodeBlock links a given get_from_scope the same way.
    Optimized code that one loader warmed up meets empty slots when the next loader runs it:
    JSTests/modules/module-loaders-lazy-function-declarations.js drives that through DFG/FTL code without the testing option.
  • Concurrency: slots are only filled by the mutator. A compiler thread that reads an empty slot through
    Graph::tryGetConstantClosureVar gets no constant. GetLazyClosureVar clobberizes as a read of the slot that can fire
    watchpoints (symbolTablePutTouchWatchpointSet on first instantiation) and allocate. The record's list of uninstantiated
    declarations is visited and released under cellLock().
  • Numbers (CLI application, 12 s after the prompt): FunctionExecutable 55.1k -> 30.0k, function objects 156k -> 131k, GC heap
    -6.3 MB, RssAnon -7.2 MB. Synthetic (2,000 modules x 46 declarations): FunctionExecutable 154k -> 66k, heap 72.6 -> 59.1 MB,
    startup instructions -6.5%.

6. LLInt get_from_scope: test for the closure variable resolve types first

Site mix at the prompt of the CLI application: ClosureVar 54.8%, LazyClosureVar 34.4%, GlobalProperty 9.8%, GlobalVar 1.0%.
Three compares fewer per closure variable read, two more per global one; pays for the empty check of commit 5.

7. Leave a module's uninstantiated function declarations in the bytecode cache payload

With commit 5, a module decoded from a payload that stays around does not create the UnlinkedFunctionExecutables of
declarations nobody read: m_functionDecls[i] stays null, ModuleFunctionDeclarationSlots remembers the Decoder and the record
array, UnlinkedCodeBlock::functionDecl() decodes on first request (mutator only; other threads use
functionDeclIfDecoded()). A never-read module function: 256 B of cells -> an 8-byte empty slot + a 4-byte table entry.
--help of the CLI application: 505 M -> 470 M user instructions.

8. jsc shell: ASAN builds keep locals on the real stack

detect_stack_use_after_return=0 as the shell's default ASAN option: the fake stack is not scanned conservatively, so every
multi-module test died under --collectContinuously / --slowPathAllocsBetweenGCs in ASAN builds.

9. Call sites in LLInt / Baseline metadata get their CallLinkInfo and ArrayProfile on their second execution

Options::useLazyLLIntCallLinkInfos

  • What: the metadata of the non-varargs call opcodes holds one pointer to a CallSiteData { DataOnlyCallLinkInfo, ArrayProfile }
    (96 B) allocated on the site's second execution; until then it points at one of two per-VM shared records ("never" /
    "once") that look like a polymorphic call to the new llint_unlinked_call thunk, so the LLInt fast path is the old one plus
    one load and no branch. sizeof(Metadata): op_call / call_ignore_result / tail_call 96 -> 8, construct 80 -> 8, super_construct
    88 -> 16, iterator_open 120 -> 48, iterator_next 136 -> 64.
  • Concurrency: a shared record is flagged and every mutator of CallLinkInfo RELEASE_ASSERTs it is not one. A site's own record
    is fully initialized before a store-store fence publishes it and lives as long as the MetadataTable. Baseline code never
    meets a shared record (setupWithUnlinkedBaselineCode allocates all of them first). Compiler threads and the collector go
    through forEachLLIntOrBaselineCallLinkInfo() / getArrayProfile(), which skip sites without their own record ("no
    information").

10. Keep the predictions of a MetadataTable's value profiles out of line, and only for code that has warmed up

Options::useLazyValueProfilePredictions, thresholdForValueProfilePredictions (40)

  • What: 8-byte buckets in front of the table; the predictions are an array off LinkingData, allocated by the first
    mergePrediction() with something to record. A still-interpreted CodeBlock below the threshold keeps samples in its buckets: GC
    end only clears buckets whose cell died.
  • Concurrency: the array is published with a compare-and-swap (mutator, marker and compiler threads may all get there first);
    readers without an array see SpecNone.

11. op_get_from_scope does not need the global variable's watchpoint set in its metadata

24 -> 16 bytes; the DFG parser finds the set in the symbol table entry under the table's lock. Field names kept.
Commits 9-11, synthetic (31.8k interpreter-only blocks): metadata 28.2 -> 15.8 MB (+0.2 out of line), 887 -> 496 B per block,
RssAnon at startup 127.2 -> 113.8 MB. Per call site: never run -73, first run +48, second run +445 instructions.

12. VMInspector::codeBlockCensus and $vm.codeBlockCensus()

Counts used by the tests of commits 13-14 (and by embedders that want to see where code memory goes).

13. Code decoded from a persistent bytecode cache can be dropped and decoded again

Options::useCodeRecoveryFromBytecodeCache

  • What: a code block decoded from a persistent payload remembers its payload (16-bit index into the per-VM
    PersistentBytecodePayloads) and record offset (32 bits), both in existing padding (sizeof unchanged).
    UnlinkedFunctionExecutable::returnCodeToCache() puts an executable back into its m_isCached state.
    VM::shrinkFootprintNow(flags) / shrinkFootprintWhenIdle: KeepCodeThatNeedsParsing, KeepCodeInUse,
    LeaveCollectionToCaller; Heap::deleteAllUnlinkedCodeBlocks takes which kinds to drop. VM::entryCountFromOutside() and
    Heap::lastActiveCollectionTime() let the embedder decide when the program is at rest.
  • The drop path never reads the payload (the embedder has usually paged it out by then): live child executables are remembered
    weakly under the parent's record offset and adopted by the block decoded from that record later, so closures made before and
    after a drop share code.
  • Identity and lifetime: a slot of PersistentBytecodePayloads is one tree of unlinked code = the payload's bytes as decoded for
    one SourceProvider (embedders wrap the same bytes in a new CachedBytecode + SourceProvider per load). Nothing remembered under one
    slot is handed to code decoded under another; code private to one executable (a module whose loader has its own module
    scope) is not registered, since its Baseline code assumes one resolution of that scope. Every Decoder and every code block of
    a slot hold it (retain/release, ~UnlinkedCodeBlock, ~Decoder); with the last one the payload/provider references go and the
    index is reused: reloading one file 50 times leaves 3 slots, not 52.
  • Contract: shrinkFootprintNow returns false (nothing done) under JS or from inside the collector; the Keep modes wait for a
    running collection, the flagless mode returns false instead; lastException is cleared (its stack kept dropped code alive twice);
    VM::deleteAllCode (flagless) now also deletes what a cache can hand back (heap after shrink + GC 5.4 MB instead of 31.7 MB on a
    synthetic application). A CodeCache entry decoded from a payload is committed and dropped in every mode: a provider that has
    the payload decodes again, one with equal source text but no payload parses.
  • Concurrency: all of it runs on the mutator. Heap::deleteAllUnlinkedCodeBlocks completes the compiler threads' ready plans
    (that allocates) BEFORE its HeapIterationScope and under DeferGC, then re-asserts that no collection runs, so nothing marks
    while returnCodeToCache turns an executable's two code block slots into {Decoder, offsets}; both directions publish with
    m_isCached in the middle (fences), and visitChildren reads flag, slots, flag. OnlyWithoutLinkedCode (restricts the
    recoverable kind) skips blocks anything links against; otherwise all plans are completed first, so no compiler thread holds a
    block that is dropped. Heap::lastActiveCollectionTime() is atomic (zero until the first collection that saw the mutator busy).
    Lean decoder only.
  • Numbers: re-decode 2.5k instructions per block vs 111k re-parse; synthetic: unlinked code blocks 19,252 -> 2,383, GC heap
    33 -> 23 MB.

14. Module and program code that has run is released right away

Options::useRunOnceCodeRelease; useSharedModuleFunctionExpressionExecutables (default off)

  • What: once JSModuleRecord::evaluate sees the body finished, the executable drops its linked code, and its unlinked code and
    CodeCache entry if they can be decoded again; the environment symbol table and the declarations' executables stay.
    Interpreter::executeProgram clears the ProgramExecutable it created.
  • Module loaders (#522): the executable counts the records that adopted it and have not finished; the code goes with the last
    one (covers a loader suspended at a top-level await while another finishes). A later loader adopts an executable that only
    released recoverable code and has it decoded again (the symbol table and m_functionDeclarations are the executable's for
    life, commit 1; ClearCode::All withdraws a released executable from adoption). With commit 5/7 the module record no longer references
    releasable unlinked code for the sake of unread declarations (it decodes them from the payload).
  • Concurrency: linked code that left the interpreter or has a Baseline compile queued is left to age out
    (GlobalExecutable::canReleaseLinkedCodeNow).
  • Numbers: synthetic: ModuleProgramCodeBlock 2,001 -> 1, UnlinkedModuleProgramCodeBlock 2,564 -> 2, GC heap 39.4 -> 24.3 MB,
    extra memory 11.8 -> 1.5 MB, startup instructions -4.4%.

15. Heap: live size after the last collection and total bytes allocated, for embedders

Heap::sizeAfterLastCollection() and Heap::totalBytesAllocated() (running total kept in updateAllocationLimits() right before
the per-cycle counters are reset; mutator thread only). For an embedder's idle detection: a busy program whose heap does not grow
still allocates.

Not in this PR: dropping cold LLInt CodeBlocks in any collection and block sealing were written and measured; neither moved
resident memory on the application, both were off by default, so they were left out.

Numbers on the CLI application (RssAnon MB, RssFile 46-51 MB throughout; two to three runs each; rows 1-4 were measured with the series on the previous base dfd696443b, row 5 on this branch; runs of one round are comparable, rounds differ by up to ~8 MB)

fresh prompt (10 s)idle prompt (100 s)105 s after a 20-turn session240 s after
before117-12596-104168-174154-169
commits 2-8108-11786-96156-160150-155
commits 2-11102-10988-95159-162152-154
all (embedder calls shrinkFootprintWhenIdle at deep idle)9886156135-138 (131 dropping everything)
all, this branch on cf1b36ec8703 (same-run baseline 124-127 / 105-106 / 173 / 163-169)106-10893-94158-163135-140

CPU (instruction counts; wall clock on the measuring box is noise):

beforeafter
--help, user instructions (like-for-like builds on the previous base)505.7 M471.2 M (-6.8%)
--help, this branch, same binary with every option of the series off / on551 M513 M (-6.9%)
20-turn session, whole process19.01 G18.83 G (-0.9%, noise ~1%)
interpreter-only micro loops (calls / property access / closure scope)-0.41% / -0.06% / -5.0%
module call loop: LLInt / Baseline only / DFG+FTL (useConcurrentJIT=0)-1.0% / +1.6% / identical to the instruction
global-variable-heavy loop, LLInt+1.9% (dispatch order of commit 6)
wake-up after a deep-idle shrink~+20 M instructions once

Tests

New: JSTests/modules/lazy-function-declarations*.js (6, run under 32 option sets: tiers, eager thresholds, no-cjit validation,
collectContinuously, bytecode cache fill/hit incl. persistent payload), module-loaders-lazy-function-declarations.js;
JSTests/stress/bytecode-cache-unlinked-value-and-array-profiles.js, llint-lazy-call-link-info{,-async,-relink}.js,
value-profile-predictions-{on-demand,after-cold-collection}.js, get-from-scope-global-variable-watchpoint-lookup.js,
shrink-footprint-recovers-code-from-bytecode-cache.js, module-code-released-after-evaluation.js,
module-function-expression-executables-shared.js, module-lazy-function-declarations-outlive-released-code.js,
module-loaders-share-released-code.js, module-loaders-share-one-environment-symbol-table.js,
shrink-footprint-while-compilations-are-ready.js, shrink-footprint-decodes-code-again-repeatedly.js (also with --destroy-vm),
bytecode-cache-persistent-payloads-are-released.js.

Results on this branch, against a base list taken with jsc built from cf1b36ec8703 itself (release, and Debug+ASAN
where noted); every difference from base was re-run on both binaries:

suitethis branchbase (cf1b36ec8703)
every commit of the series builds (release jsc) and passes the tests it adds or changes, at that commit15/15
lazy declaration tests (6) + module loader tests (5, incl. the new one) x 32 option sets, release352/352
same, Debug+ASAN352/352
JSTests/modules (122 files), default and no-cjit-validate114 pass / 8 fail, both113 / 9 (the 8 + the new loader test)
full JSTests/stress x {default, no-cjit-validate, no-JIT, collectContinuously, low thresholds, eager jettison, no concurrent JIT}35 / 34 / 80(+1 timeout) / 36 / 36 / 34 / 33 failures of ~5,65038 / 38 / 84(+1) / 39 / 40 / 40 / 37: the same tests plus the new tests that need this branch (and module-loaders-share-one-environment-symbol-table.js, which fails on the base because of the bug commit 1 fixes); the only others are three tests that flip between runs on both binaries (recursive-try-catch.js, int8-repeat-in-then-out-of-bounds.js, codeblock-aging-ftl-idle.js)
bytecode cache fill + forced hit: bytecode-cache-*.js and the new cache tests, all their option variants47/47 (ASAN 44/44 over the same files without one ten-minute test)19/15 (the new tests)
bytecode cache, stress list (1,357 runs)1326 / 311325 / 32 (same 31 + one flaky)
bytecode cache, JSTests/modules, plain and with a persistent payload113 / 9 both (ASAN 113 / 9)112 / 10; the ninth is module-loaders-changed-dependency.js, which writes new source files on every run and so cannot pass a forced cache hit on either binary
test262 module-code, import, export, dynamic-import, import.meta, eval-code, function dirs (3,718)16 failuresthe same 16
Debug+ASAN, 1,504 stress tests selected by name (call, profile, llint, osr, tier, inline, construct, spread, iterator, closure, scope, module, function, tdz, bytecode-cache ...) x 6 option sets17 / 17 / 18 / 17 / 18 / 16 failures (default, eager jettison, low thresholds, no concurrent JIT, no JIT, collectContinuously) and 3 collectContinuously timeouts out of 1,521 files: exactly the (option set, test) pairs that failed in the same sweep before the rebase, each of which fails on the base release build too, is the 160 KB / 1.5 MB stack cap of a test meeting Debug+ASAN frames, or is a slow test under collectContinuously + ASAN on a loaded machine
aarch64: all of JavaScriptCore/WTF/bmalloc cross-compiled for darwin-aarch64 (offlineasm ARM64 LLInt, DFG, FTL)compiles; not run

Seen while testing, not caused by this series (reproduces on the base with no shrink): ASSERTION FAILED: addResult.isNewEntry in
CachedBytecode::copyLeafExecutables on the cache-filling run with --useJIT=0 --forceCodeBlockToJettisonDueToOldAge=1 when generated
unlinked code is regenerated after a collection (stale raw-pointer keys in m_leafExecutables).

LowLevelInterpreter32_64.asm is not updated for commits 5 and 9-11 (no 32-bit target is built from this repository).