fix(startup): make STOP/START safe and add per-protocol startup controls

RE
2d ago
+1644-96

PROXYSQL STOP can deadlock while teardown waits for the issuing Admin client, and stopped-state runtime reads can dereference absent processors. This PR fixes those lifecycle failures and adds independent MySQL/PostgreSQL startup controls on current v3.0.

  • Lifecycle: drain the STOP acknowledgement before initiating shutdown, with a one-second bound for stalled peers; retain existing client draining and normal statistics refresh. Guard absent-processor snapshots, preserve stopped-state LOAD SERVERS and Prometheus output, and report explicit unavailable-runtime errors for SAVE QUERY RULES/FIREWALL. Fix PostgreSQL TLS retry accounting so accepted plaintext is not sent twice.
  • Startup: add mysql-workers, pgsql-workers, mysql-admin, pgsql-admin, mysql-monitor, and pgsql-monitor as CLI/top-level configuration permissions, defaulting to true. Validate every explicit value, support separate/equal forms and true/false/1/0, apply CLI precedence, and retain permissions through STOP/START. Disabled workers have zero active Worker status and no worker/idle threads or listeners; configured positive thread counts survive runtime/disk SAVE and enabled relaunch.

Shared authentication, hostgroup managers and monitor data objects remain available. Core LOAD MYSQL/PGSQL SERVERS stays successful. Monitor permission remains distinct from persisted monitor_enabled. New data fields are appended to preserve existing member offsets used by plugin readers.

STOP acknowledges the request asynchronously; existing Admin clients disconnect during reload and must reconnect before configuration or START.

This combines the lifecycle work previously reviewed in #6558 with the startup refresh in this PR, using one CI matrix against v3.0. Fresh implementation of the remaining ideas in #5217 and #4960, with context from #5222 and issues #5186, #4951 and #5218. Prior implementation/report/review by JavierJF and Rene Cannao are retained as references. The original PRs remain open.

Validation:

  • DEBUG Stable and v3.1 AddressSanitizer builds; seven lifecycle and twelve startup process tests pass, including all 16 worker/Admin combinations, all four monitor combinations, STOP/START, graceful shutdown, CLI validation/precedence, and saved nondefault worker counts through physical relaunch.
  • Twenty real-stream drain checks pass on both builds: oversized packet staging, full sockets, partial writes, stalled readers, real TLS 1.3 and exact decrypted output. Layout regression passes.
  • Latest standalone-runner changes were rechecked on Stable: prerequisite/non-Linux skips, Linux/Darwin/FreeBSD dependency probes, and genuine probe failures remaining failures. Coverage CI installs the tools and executes the probes.
  • All eight changed production translation units, relevant unit-test sources and a layout probe pass v4.0 compilation checks. Installed-plugin and full backend integration are covered by remote CI rather than claimed as local runs.
  • Final head 907e218b1: repository CI lint, all 65 product test checks, the selected-tier gate (66/66 including the build), ThreadSanitizer, Sonar and both Codecov checks pass. The v4.0 AddressSanitizer unit job reports 234 PASS / 0 FAIL, including the lifecycle/startup probes and response-drain/layout regressions.
  • CI follow-up: the startup probe tolerates a transient task exiting during /proc enumeration while retaining its worker assertions. A controlled real task exit reproduced the original failure and passed after the fix; all twelve startup cases also pass locally. Confirmed backend setup timeouts were retried at failed-job scope.
  • Sonar's quality gate passes with four nonblocking code-smell findings: parser complexity/nesting and two deliberate offsetof checks in the ABI layout regression.

Probes: python3 test/infra/control/test_module_lifecycle.py src/proxysql and python3 test/infra/control/test_module_startup.py src/proxysql. Linux, Python 3 and mysql/psql clients are required; each test owns its private daemon, datadir and ports. Sanitizer execution uses ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:halt_on_error=1.

Operator details: doc/features/module-startup-controls.md. Plan and verification record: docs/superpowers/plans/2026-10-28-module-lifecycle-refresh.md.

Summary by CodeRabbit

  • New Features
    • Added startup controls for MySQL and PostgreSQL workers, Admin listeners, and monitor threads. All are enabled by default; settings can be supplied in configuration or on the command line and take effect after relaunch.
    • Configured worker-thread counts remain available even when worker startup is disabled.
  • Bug Fixes
    • Improved Admin shutdown response handling and behavior when modules are stopped or unavailable, including clearer errors when saving runtime state.
    • Improved reliability of encrypted PostgreSQL responses and availability of metrics when components are not running.
  • Documentation
    • Documented startup settings, precedence, and runtime behavior.