PR #554 - Luce-Org/lucebox

Summary

add default-off semantic ROCTX ranges for external ROCm tracing
cover monolithic and heterogeneous prefill, speculative decode, and leaf layer ranges
label exact prefill, decode, and verification explicitly across local and remote target shards
dynamically load ROCTX only when enabled and keep non-HIP builds as no-ops
preserve rocprof's process-global ROCTX interposition while retaining the direct libroctx64.so fallback

This replaces the superseded historical in-process HIP-event profiler on current main. The old implementation recorded on a null/default stream and synchronized events, which could fence unrelated streams and distort the two-owner overlap being measured. This rewrite intentionally performs no internal GPU timing.

Behavior contract

DFLASH_DS4_ROCTX=1 enables balanced semantic ranges for correlation with an external rocprof kernel/copy trace. The marker layer:

uses no HIP/CUDA events
performs no stream or device synchronization
calculates no elapsed time
remains silent and does not load ROCTX when unset or false
reports a one-time diagnostic when explicitly enabled but ROCTX cannot be loaded
closes incomplete library handles when required symbols are missing
balances successful pushes through RAII, including early-return paths
avoids recursive layer-range double counting by marking leaf ranges only
carries a validated value-type semantic phase through both target-shard IPC transports
drains safely framed payloads before rejecting missing, extra, or invalid phase metadata

ROCTX span lengths are host-side semantic intervals. They are not GPU kernel-duration measurements.

Demonstrated loader defect and fix

Real rocprofv3 validation found that the previous handle-scoped dlopen("libroctx64.so") plus dlsym(handle, ...) path bypassed rocprof's preloaded ROCTX interposer: a normally linked ROCTX smoke test traced, while the equivalent dynamic-loader smoke test and server did not.

Commit e028bb22b136695bf4ef89f1677ed4670eae95ee adds a regression that fails with the old lookup, then prefers a complete process-global roctxRangePushA/roctxRangePop pair through dlopen(nullptr). It preserves the original library fallback and does not change Windows behavior or add synchronization.

Current-main build and test validation

Validated as a no-commit synthetic merge with upstream/main 4fa8937c69aa68658a14ec3cab16e1da22886dd0.

strict C++17 -Wall -Wextra -Werror ROCTX/protocol test: passed
ASan/UBSan ROCTX/protocol test: passed
full HIP/gfx1151 RelWithDebInfo build: passed
full HIP CTest: 364/364 passed
full CUDA/sm_86 RelWithDebInfo build with CUDA 12.0.140: passed
final CUDA-configured host-only ROCTX regression: passed
malformed target-shard phase followed by a valid request: passed for missing, extra, and invalid values
IPC serialization/value validation and next-request synchronization tests: passed
git diff --check: passed
fresh independent final review of the three-file fix: no actionable findings
review follow-up 0aec3941088ae348df62c8a5fc8d9eedbb4599fd re-executes the real loader regression in an isolated child process so neither DFLASH_DS4_ROCTX nor the process-scoped callback cache leaks into the parent suite; HIP-defined strict, HIP ASan/UBSan, non-HIP strict, and the CUDA-configured repository test pass
wait hardening 7c87686bad02bb1a1c5b96c2986ad37d3c27fa63 bounds the child to approximately five seconds with WNOHANG, terminates and reaps only that owned child on timeout, and reports signal termination explicitly; the same strict, sanitizer, non-HIP, and repository tests pass

Representative commands:

cmake -S . -B build-pr554-hip-gfx1151 -G Ninja \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DGGML_HIP=ON -DGGML_CUDA=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DDFLASH27B_TESTS=ON \
  -DDFLASH27B_ENABLE_BSA=OFF
cmake --build build-pr554-hip-gfx1151 -j 4
ctest --test-dir build-pr554-hip-gfx1151 --output-on-failure --timeout 180

cmake -S . -B build-pr554-cuda-sm86 -G Ninja \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DGGML_CUDA=ON -DGGML_HIP=OFF \
  -DCMAKE_CUDA_ARCHITECTURES=86 -DDFLASH27B_TESTS=ON \
  -DDFLASH27B_ENABLE_BSA=ON
cmake --build build-pr554-cuda-sm86 -j 4
ctest --test-dir build-pr554-cuda-sm86 \
  -R '^test_deepseek4_roctx$' --output-on-failure --timeout 60

Real gfx1151 rocprof capture

Environment:

Radeon 8060S, gfx1151
ROCm 7.2.4 / HIP 7.2.53211
rocprofv3 1.1.0
final HIP dflash_server SHA-256: 2e545e9d4205773c679bb88b2f1058bac9768f241331d63d816023130fdbacd6

The enabled speculative request captured 21 balanced marker intervals:

one ds4.prefill mode=exact enclosing 11 exact leaf ds4.layer_range intervals
one ds4.spec_decode mode=batched enclosing 8 verify leaf ds4.layer_range intervals

A separate target-only request captured 27 intervals:

one exact prefill plus 11 exact leaf layer ranges
15 mode=decode leaf layer ranges

All intervals had sane ordering and intelligible request nesting. Unset and explicit DFLASH_DS4_ROCTX=0 real-request controls produced no DS4 marker rows and no marker CSV.

Profiler command form:

rocprofv3 --marker-trace --minimum-output-data 0 -f csv json \
  -d <output-directory> -o trace -- env DFLASH_DS4_ROCTX=1 \
  <dflash_server model and request configuration>

Alternating disabled/enabled perturbation

The comparison used fresh processes, cache slots disabled, the same binary/models/prompt/placement, two warmups per mode, then five measured off,on pairs.

Mode	Server total median	MAD	Range
Disabled	3165.8 ms	2.4 ms	3160.4-3204.6 ms
Enabled	3188.6 ms	13.4 ms	3157.5-3217.0 ms

Measured median change: +0.72% in this bounded sample.

Correctness parity was exact across all ten measured requests:

identical generated-content SHA-256: 39bcfb8d666171e8f833154fef4a6f70918ffb7458e4c8b32813b865e94abb67
identical (prompt, completion, total) token counts: (11, 28, 39)
acceptance rate 1.0 throughout

Clocks were not administratively pinned; alternating pairs were used to limit temporal drift. This result is instrumentation-perturbation evidence for this setup, not a general performance claim.

Remaining merge gates

repository CI on exact head 7c87686bad02bb1a1c5b96c2986ad37d3c27fa63
maintainer review/approval

Disclosed hardware follow-ups

rerun the full local CUDA runtime suite after the RTX 3090 device handle is restored
capture a real mixed CUDA/HIP target-shard trace after that device is restored; protocol-level phase transport is already covered

These unavailable local cells are not assumed to pass and are not presented as evidence. They are follow-up hardware coverage rather than blockers for this HIP-only marker-loader correction: the CUDA build and CUDA-configured host regression pass, while exact-head repository CI remains the cross-platform merge gate.

The PR is locally hardware-qualified for the available gfx1151 paths, but it is not genuinely merge-ready until exact-head CI and maintainer approval complete.

Stack

Independent current-main PR. It does not depend on #548 or the #569/#575/#577 performance stack. #575 may provide a useful external rocprof collection wrapper, but it is not a code prerequisite.