fix(admin): make STOP/START and stopped-state queries safe

Closed
RE
6d ago
+793-59

Superseded by #6559, which now includes these lifecycle fixes and the startup controls in one PR against v3.0. Consolidated at the maintainer's request to avoid duplicate CI work.

PROXYSQL STOP can deadlock because the issuing Admin client waits for teardown while teardown waits for that client to exit. With --no-start (including after STOP), runtime query-rule reads can also dereference absent query processors and crash, reproducing #5186 on current v3.0.

This change drains the STOP acknowledgement before initiating shutdown (bounded to one second for a stalled peer), releases the Admin SQL lock, and lets existing client draining finish safely. It also fixes duplicate PostgreSQL TLS plaintext on a ciphertext-write retry, matching MySQL's existing accounting. It guards processor-dependent snapshots without disabling normal statistics refresh, makes Prometheus output available while stopped, and constructs monitor data objects before Admin accepts LOAD SERVERS.

Behavior: STOP acknowledges the request asynchronously. Existing Admin clients disconnect during reload; reconnect to the rebuilt Admin interface before continuing with configuration or START. LOAD MYSQL/PGSQL SERVERS remains usable while stopped. Explicit SAVE QUERY RULES/FIREWALL from unavailable runtime state returns an error and preserves configuration; snapshot reads still use the actual empty schemas. Existing processor-required LOAD errors remain in place.

Fresh replacement for the remaining lifecycle ideas in #5217, with context from #5222 and #5218. Prior work and the original report by JavierJF and Rene Cannao are preserved as references. The old SSL initialization change is not carried forward: the current startup/TLS ownership path is used.

Validation:

  • Fresh DEBUG v3.1 baseline reproduced eight query-processor crashes across both Admin wire protocols, STOP timeouts on both protocols, and NULL stopped-state metrics.
  • Seven isolated lifecycle regression tests pass on DEBUG Stable and v3.1 with AddressSanitizer: initial no-start, all runtime/statistics table reads, both LOAD SERVERS commands, actual statistics refresh after START, repeated STOP/START through both Admin protocols, concurrent readers, and Prometheus output.
  • A registered 20-check real-stream drain regression passes on DEBUG Stable and v3.1 with AddressSanitizer: oversized packet staging, full sockets, partial writes, stalled readers, real TLS and exact decrypted output. It reproduced the PostgreSQL retry defect before the fix.
  • Lifecycle and drain probes run in the CI unit/coverage suite; its runtime setup installs the required Python and protocol clients.
  • All modified production translation units pass compilation checks with v4.0 feature flags. This is compile-only; installed-plugin integration and the full backend TAP matrix are left to CI/review.
  • Repository CI lint passes.

Probe: python3 test/infra/control/test_module_lifecycle.py src/proxysql. It launches private daemons with temporary datadirs/loopback ports, and requires Python 3 plus the mysql and psql clients. Sanitizer runs use ASAN_OPTIONS=detect_leaks=0:abort_on_error=1:halt_on_error=1.

Summary by CodeRabbit

  • Bug Fixes
    • Improved reliability of STOP and START operations. Admin clients receive the shutdown acknowledgment without waiting for shutdown to finish.
    • Admin queries and metrics handle unavailable components safely. Save operations return an error rather than clearing existing data when required components are unavailable.
    • Improved stability for runtime reads and maintenance operations when modules are stopped or not started, while preserving statistics refresh behavior.
    • Improved reliability of encrypted network writes during backpressure.
  • Improvements
    • MySQL and PostgreSQL monitor components are initialized earlier during startup.