Fix issue 5186: Prevent admin query crashes after PROXYSQL STOP/START cycles

Draft
RE
10mo ago
+430-19

Summary

Fixes segmentation faults that occur when admin queries access destroyed global modules after executing PROXYSQL STOP. This resolves crashes in MySQL_Query_Processor.cpp:958 and ProxySQL_Admin.cpp:4424 by implementing proper query counting, null pointer protection, and restart sequence fixes.

Problem

After PROXYSQL STOP command execution, admin queries could crash when accessing global modules (GloMyQPro, GloMyAuth, etc.) that were destroyed during shutdown. Multiple crash scenarios were identified:

  1. Admin queries running after module destruction during STOP
  2. SSL context corruption during thread destruction from multiple SSL initializations
  3. Race conditions during PROXYSQL START where admin threads query before Query Processors are initialized
  4. No proper restart capability from STOP state

Solution

Implemented comprehensive crash prevention with layered fixes:

Query Counting and Drain Timing

  • Fixed admin query counting to properly subtract the current PROXYSQL STOP query
  • Added wait mechanism for admin queries to complete before module destruction
  • Enhanced debugging to identify bottlenecks in thread shutdown sequence

PROXYSQL START Restart Sequence

  • Fixed PROXYSQL START to trigger proper module reinitialization
  • Set GloVars.global.nostart = 1 to simulate startup conditions
  • Ensured all global modules are properly reinitialized from STOP state

SSL Context Corruption Prevention

  • Made ProxySQL_Main_init_SSL_module() idempotent to prevent duplicate SSL contexts
  • Prevents OpenSSL thread-local storage corruption during thread destruction

Null Pointer Protection

  • Added null checks in four save_*_from_runtime functions:
    • save_mysql_query_rules_from_runtime()
    • save_mysql_query_rules_fast_routing_from_runtime()
    • save_mysql_users_from_runtime()
    • save_mysql_servers_from_runtime()
  • Gracefully handles queries when Query Processor modules are not initialized

Query Handling During STOP State

  • COUNT(*) queries work normally with null pointer protection
  • LOAD MYSQL USERS TO RUNTIME succeeds (MySQL Auth module is loaded)
  • LOAD MYSQL QUERY RULES TO RUNTIME fails with proper error message
  • Basic queries continue to work: arithmetic, version, prometheus metrics

Files Modified

  • lib/Admin_Handler.cpp: Query counting, STOP/START handling, debug logging (38 lines)
  • lib/ProxySQL_Admin.cpp: Null pointer protection in save functions (12 lines)
  • src/main.cpp: SSL context idempotent initialization (8 lines)
  • test/tap/tests/test_proxysql_stop_query_handling-t.cpp: Comprehensive test coverage (167 lines)
  • test/tap/groups/groups.json: Test configuration (1 line)

Test Coverage

Added comprehensive TAP test suite that verifies:

  • PROXYSQL STOP command succeeds and completes in reasonable time
  • COUNT(*) queries work normally during STOP state with null protection
  • LOAD MYSQL USERS TO RUNTIME succeeds (Auth module loaded)
  • LOAD MYSQL QUERY RULES TO RUNTIME fails (Query Processor not started)
  • Basic queries work during STOP state (arithmetic, version, prometheus)
  • PROXYSQL START successfully restarts from STOP state
  • Queries continue working after START

Key Technical Insights

  • Root Cause: Multiple crash modes during STOP/START cycles requiring layered protection
  • Critical Fix: Null pointer protection more effective than query blocking
  • SSL Issue: Duplicate SSL contexts caused thread-local storage corruption
  • Race Condition: Admin threads querying before module initialization completed
  • Query Behavior: Different module availability during STOP state

Verification

  • ProxySQL binary compiles successfully with all fixes
  • TAP tests validate crash prevention and proper functionality
  • Manual testing confirms no segmentation faults during STOP/START
  • Test coverage validates all critical paths and edge cases

Impact

This fix ensures:

  • Complete elimination of segmentation faults from admin queries after PROXYSQL STOP
  • Stable PROXYSQL STOP/START functionality without crashes
  • Proper handling of different module states during admin operations
  • Prevention of SSL context corruption during restart cycles
  • Graceful degradation when modules are not available

Resolves issue #5186 completely while maintaining full backward compatibility.