Restart rebooted harts on a coroutine stack

Merged
JS
13d ago
+910-51

A guest reboot used to longjmp from the SBI reset handler back to a setjmp in rv_step(), which set the jump point up on every step and jumped across whatever interpreter, JIT or T2C frames lay in between. Native system builds now run the hart on a coroutine (src/coro.c: a short context switch on x86-64 and Arm64, ucontext elsewhere). A reboot abandons the coroutine stack, and the next resume starts the hart's run loop from the top. The coroutine returns to rv_run() only when the hart halts or restarts; under gdbstub it yields after every instruction. The Emscripten build keeps a direct-step path. The coroutine stack is 8 MiB of lazily committed memory with a 64 KiB guard region, and an overflow into it prints a diagnostic instead of corrupting the heap. All of this is gated by a new RV32_HAS_HART_CORO (SYSTEM_MMIO and not Emscripten); user mode, the ELF loader and wasm are unaffected.

Validation, on x86-64 (Threadripper 2990WX) unless noted:

  • Builds with gcc and clang, all passing: defconfig, jit_defconfig (T2C), system_defconfig, system_interpreter_defconfig, system_jit_defconfig (T2C), system with ELF_LOADER, and system and user mode with GDBSTUB.
  • .ci/boot-linux-test.sh, built as CI does (INITRD_SIZE=32 ENABLE_MOP_FUSION=0, then make artifact), on system_defconfig (gcc), system_interpreter_defconfig (clang) and system_jit_defconfig (clang): 26/26 each, covering boot, cold and warm reboot, virtio-blk, and virtio-net over TAP and SLIRP.
  • make check on defconfig, jit_defconfig and system with ELF_LOADER: all pass.
  • Arm64 (Ampere eMAG, clang, ENABLE_SDL=0): system_defconfig and system_interpreter_defconfig build; cold and warm reboot pass on both.
  • A standalone test linking src/coro.c keeps integer and floating-point values live across a million switches in both directions, and checks restart, the overflow diagnostic, and that other faults are passed on. It passes at -O0, -O2 and -O3 with gcc and clang, on x86-64 (asm and ucontext) and Arm64.
  • Boot time to the login prompt is unchanged: x86-64 system_defconfig takes 3.07 to 3.10 s on this branch against 3.08 s on master, and Arm64 takes 8.50 s against 8.58 s. On x86-64, identical source built in two trees differed by up to 6%, so only differences beyond that were treated as real.

Not covered: the wasm build (no emcc here), and gdbstub beyond the first few dozen stepi. Master already hangs in system-mode gdbstub at the same point (pc 0x100), and this change does not alter that.

Summary by cubic

Reboots used to longjmp from the SBI reset handler to a setjmp point set up on every step. Native system builds now run each hart on a coroutine, so a guest reboot abandons the whole host call stack and restarts the hart's run loop from the top on the next resume.

  • Adds src/coro.c: assembly context switches on x86-64 and Arm64, ucontext elsewhere; 8 MiB lazily committed stack with a 64 KiB guard region that prints a diagnostic on overflow.
  • Gated by RV32_HAS_HART_CORO (system mode, no Emscripten); user mode, the ELF loader, and wasm are unaffected.
  • The coroutine returns to rv_run() only on halt or restart; under gdbstub it yields after every instruction.
  • The Emscripten build keeps the direct-step path and resets dispatcher state after a reboot instead.
  • rv_debug() now releases the breakpoint map and gdbstub connection when a session ends.
  • Boot time to the login prompt is unchanged on x86-64 and Arm64.