perf(tables): stream stable token assignment for native coordinates
TL;DR
Table token assignment builds a dictionary for every candidate and sorts all candidates to select one winner. This change selects the first maximum in one pass for built-in coordinates, uses smaller score records for other scalar types, and preserves the complete assignment triple and geometry arithmetic.
All five selected Linux SND fixtures completed two baseline/candidate pairs with exact full-output parity. Captured typed geometry also preserves the complete assignment triples. The source has full exact-head Oracle Pro approval and green CI. No merge, release, or deployment was performed.
Selection and compatibility
The change keeps the earliest container on ties. It also keeps scores for threshold-rejected packages, zero-area package behavior, and forced assignment. Keep keyed sorting for foreign scalar types: mixed NumPy/Python arithmetic can produce NaN-separated scores for which a maximum scan changes the winner. Both structure_to_cells calls, containment NMS, all-pairs geometry, crop-specific OCR, and model execution are unchanged.
Native selection uses constant scratch space and linear-time selection. The compatibility path still collects and sorts scores. Required output lists remain in both paths.
Linux SND evidence
Same pinned image and dependencies, 2 CPUs and 8 GiB RAM. Each fixture ran in fresh processes, in baseline/candidate then candidate/baseline order. Image, filesystem, and model caches were shared and uncontrolled. Finance evidence was retained from the original run; Civil p006, Design-thinking, FCM, and ijerph completed in the continuation.
| Primary / pair | Partition baseline / candidate | Assignment function baseline / candidate |
|---|---|---|
| Finance, first | 27.028s / 25.356s | 2.361s / 2.271s |
| Finance, reverse order | 25.023s / 25.054s | 2.290s / 2.125s |
| Civil p006, first | 70.112s / 65.925s | 0.259s / 0.249s |
| Civil p006, reverse order | 65.082s / 65.640s | 0.262s / 0.259s |
All ten pairs have exactly equal full extraction JSON, including HTML cells, numeric text, spans, and order. Finance and Civil use NumPy scalar coordinates and take the compatibility path. FCM and Design-thinking have no assignment calls and serve as unaffected controls. Ijerph also exercises table inference. The later primary pairs show no end-to-end gain. Timing differences in the first pairs include cache warm-up, startup cost, and post-wake load variation. Parent timers include trace-copying and serialization overhead; nested durations overlap.
| Native synthetic kernel | Median baseline / candidate | Peak traced bytes baseline / candidate |
|---|---|---|
| 400 containers × 1,500 packages | 1.004s / 0.873s | 404,564 / 217,728 |
| 4,096 containers × 1 package | 6.637ms / 10.001ms | 1,321,556 / 263,864 |
| 4,096 containers × 2 packages | 13.331ms / 14.922ms | 2,313,992 / 264,124 |
All 11 synthetic cases preserve the exact triple. Allocation peaks include returned lists. The eligibility scan costs more time in the large-container, very-few-package cases, despite lower allocation peaks. These measurements do not establish a production latency or RSS improvement.
Captured typed replay passed for 116 calls plus 11 synthetic parity checks. All original assignment triples match. A 669-container × 855-package Finance call measured 1.111s baseline / 1.037s candidate, with traced peaks 529,524 / 287,968 bytes. A small Civil call (1 × 52) measured 0.259ms / 0.293ms, with peaks 8,816 / 9,296 bytes. Measurement used warmed inputs, medians of three untraced calls, alternating arm order, and separate allocation tracing. New document process wall time totaled 873.686s; including prior work and conservative interruption/replay charges, at most 1,531.762s of the original 2,700s budget was used.
The original Civil candidate run, interrupted by auto-hibernation, is retained as interrupted evidence and excluded from completed comparisons. Fresh pairs replace it for comparison. All new document attempts completed without errors or caps. The cumulative experiment budget includes that interrupted attempt charged at its full 330-second cap. Optional Civil p001 remains excluded because its known metadata-detection tail precedes token assignment. Existing baseline extraction quality remains imperfect. No corpus PDFs or raw document content are committed.
Validation and adoption
- Focused Linux SND tests: 54 passed. The no-sort regression fails on baseline and passes on candidate. Test dependencies were isolated in scratch storage; the baseline module was restored and hash-verified before document runs.
- All required and additional CI checks passed on Python 3.11–3.13 at
604e068b98d119daa1fa1dafbd44620da1153fd5(CI). Oracle Pro gaveSAFE TO MERGEfor code safety after a fresh full exact-head review (review). BCE Junior's low path-coverage finding was addressed. Source is unchanged since that approval. Human code-owner approval remains separate.
Core Product pins inference 1.6.13 and does not modify this function. Production adoption requires an inference release and a Core Product dependency-lock update. The isolated candidate source override used for validation is not a release or deployment.
Both task-owned profiling pods were deleted and verified absent. The task tunnel is closed, temporary auto-hibernation protection is removed, and SND1 is confirmed Hibernated. The completed sweep and shared deployments were not modified.
Originating Codex task: 01a14163-0ad4-7920-98a1-6fb0af0a2526.
authored by codex