feat: richer results, run controls and a language availability check

Merged
SH
10d ago
+628-9

Gives the platform more to work with from each run, lets callers set a few more limits, and stops the engine accepting languages its sandbox does not have.

Richer results

Each test in the VERDICT event and the webhook now carries exit_status, sandbox_status and message (the sandbox's explanation, such as the signal name, capped at 512 bytes) beside cpu_time_ns, run_time_ns and memory_bytes.

Run controls

POST /api/v1/execute accepts stack_limit_kb (1024 to 1048576), max_processes (1 to 512) and args (up to 16 command-line arguments, 4 KB in total, passed as separate argv entries and never through a shell; not for sqlite). They are validated at the gateway, stored with the job, re-validated when read back, and applied in the sandbox command.

Language availability

At startup, and every five minutes, the engine asks the sandbox whether each registered language's compiler and runtime exist. GET /api/v1/languages reports available, and POST /api/v1/execute answers 400 for a language that is not installed (also for the checker and interactor languages) instead of failing the job later. Until the first probe succeeds, every language is accepted; a failed probe keeps the last answer.

The sandbox image now installs Rust, Mono (C#), PHP and Ruby too. Kotlin and TypeScript (bun) have no Debian package and stay unavailable until a pinned release is added.

Testing

  • go vet clean; go test -race ./... passes for every package except internal/db, which needs Docker (testcontainers) and fails the same way on development without it.
  • New tests: argument validation, the command carrying args/stack/processes, queue round-trip and corrupt-args rejection, gateway rejection of bad controls, result fields, the language probe, and rejection of an unavailable language.
  • Not run: the sandbox image build (Docker was not available here), so the added package names are unverified; the startup probe reports any that are missing.

Summary by cubic

Adds sandbox exit details to per-test results, lets callers set stack, process, and argument limits per run, and stops the engine accepting languages that are not installed in its sandbox.

Richer results and run controls

  • Each test in the VERDICT event, the webhook, and stored results now carries exit_status, sandbox_status, and message (the sandbox's explanation, capped at 512 bytes) beside cpu_time_ns, run_time_ns, and memory_bytes.
  • POST /api/v1/execute accepts stack_limit_kb (1024–1048576), max_processes (1–512), and args (up to 16 arguments, 4 KB total, passed as separate argv entries and never through a shell; not for sqlite). Controls are validated at the gateway, stored with the job, re-validated when read back, and applied in the sandbox command.

Language availability

  • At startup and every five minutes, the engine probes the sandbox for each language's compiler and runtime; GET /api/v1/languages reports available, and POST /api/v1/execute answers 400 for unavailable languages (including checker and interactor languages) instead of failing the job later.
  • The sandbox image now installs Rust, Mono (C#), PHP, and Ruby; Kotlin and TypeScript (bun) have no Debian package and remain unavailable until a pinned release is added.

Summary by CodeRabbit

  • New Features
    • Execution requests can set per-test stack size, process/thread limits, and command-line arguments, within documented limits.
    • Test results now include runtime duration, exit status, sandbox status, and an explanatory message.
    • Language availability is reported through the languages endpoint; execution requests for unavailable languages receive an HTTP 400 response.
    • The documented language list now includes Bash, Rust, C#, PHP, Ruby, Kotlin, and TypeScript. Availability may vary by environment.
  • Documentation
    • Updated the README and API documentation with execution controls, result details, and language availability behavior.