feat(duckdb): DuckDB Server plugin for the v4.0 chassis (MySQL + PostgreSQL)

Merged
RE
2mo ago
+12376-33

Adds a DuckDB Server plugin for the v4.0 Plugin Chassis: an embedded DuckDB instance served over both the MySQL and PostgreSQL wire protocols. Ordinary mysql / psql clients connect, authenticate against mysql_users / pgsql_users, and run DuckDB SQL.

Design: docs/superpowers/specs/2026-08-26-duckdb-server-plugin-design.md

Approach

The plugin writes no wire-protocol code. It reuses core's MySQL_Session / PgSQL_Session, authentication and result serialisers by converting duckdb_result into a SQLite3_result and handing it to SQLite3_to_MySQL / SQLite3_to_Postgres. Sessions use session_type = PROXYSQL_SESSION_SQLITE with thread->gen_args left null.

  • Dependency: DuckDB 1.4.5 vendored as source (git LFS), built from source under PROXYSQL40, linked statically. Nothing new ships in the package beyond the plugin .so.
  • Threading: thread-per-connection, following SQLite3_Server::child_mysql. Unlike it, stop() joins every connection thread before closing the engine — required for clean plugin unload.
  • Tier: v4.0 only. Stable and Innovative are unaffected.

Core changes — please review these first

The design originally promised zero changes outside the plugin. That held for MySQL but not for PostgreSQL. Four core files changed:

FileChangeWhy
include/PgSQL_Protocol.hget_header() private → publicOnly route to the SQL text on this path; CurrentQuery isn't populated before handler_function runs. Reimplementing it would duplicate 60+ lines of wire-format parsing.
lib/PgSQL_Session.cppPROXYSQL_SESSION_SQLITE added to the query-dispatch gateWithout it a 'Q' packet fell through unhandled — leaked, client hung forever. The handler it dispatches to already had a case PROXYSQL_SESSION_SQLITE: arm that was unreachable dead code (the SQLite3 Server is MySQL-only, so nothing had ever driven that path).
include/ProxySQL_Plugin.hDEBUG-aware ABI versionSee below.
lib/ProxySQL_PluginManager.cppLoader refuses a mismatched pluginSee below.

The first two are inert when the plugin isn't loaded: an access-specifier change, and a condition no other session type reaches (verified — the only other assignment of that session type in the tree builds a MySQL_Session, not a PgSQL_Session).

The ABI guard applies to all plugins, not just this one

MySQL_Protocol has a #ifdef DEBUG bool dump_pkt; member and MySQL_Data_Stream holds it by value, so field offsets shift 8 bytes between debug and release builds — measured: sizeof(MySQL_Protocol) 80/88, offsetof(MySQL_Data_Stream, DSS) 768/776. A plugin built with a different -DDEBUG setting than the core silently writes through the wrong offsets. Nothing detected this: the chassis versions only the descriptor struct, while a plugin resolves ~475 undefined symbols against the core binary.

The loader now refuses such a plugin with a clear message instead of corrupting memory. This protects mysqlx and genai equally. Both were verified to still load.

Security

DuckDB's enable_external_access defaults to true, which would let anyone in mysql_users read and write files as the ProxySQL process user. The plugin now exposes it as duckdb_variables.enable_external_access, defaulting to false, with a Security section in the README. DuckDB permits true→false on a running database but throws on false→true, so loosening requires an engine reopen — documented.

Testing

  • 7 unit-test binaries, 111 assertions.
  • End-to-end through the TAP harness (duckdb-e2e-g1): MySQL 1..10, PostgreSQL 1..8, Admin 1..7. Reconciliation declared=3 discovered=3 executed=3 passed=3, ret_rc=[0].

Known open defect

An intermittent assert(0) at lib/MySQL_Protocol.cpp:434 in generate_pkt_ERR aborted the whole process twice early in development. ~500 executions since — stress, a control against core's own SQLite3 Server, and several harness runs — are clean. The ABI-mismatch mechanism above is proven but reproduces as a hang, not that abort, and cannot explain crashes that survived seven prior queries on the same connection. Documented as unexplained, not closed. Details in plugins/duckdb/README.md §7.

Notes for reviewers

  • This is the repository's first git-LFS object (98 MB). git lfs is now required to build v4.0 from source, and GitHub-generated source tarballs will contain pointers rather than content.
  • macOS -Wl,-exported_symbol is unverified locally — first macOS CI run confirms it.
  • The design doc carries a "post-implementation corrections" section retracting several of its own pre-implementation claims.

Summary by cubic

Adds a v4.0-only DuckDB Server plugin that embeds DuckDB and serves SQL over MySQL and PostgreSQL listeners, reusing ProxySQL authentication and result serialization. It also fixes PostgreSQL SQLite-session dispatch and hardens execution, shutdown, and plugin loading so queries no longer hang or run twice, unload cannot block on long queries, and mismatched DEBUG builds are rejected before they can corrupt memory.

Behavior and safety

  • Unsupported result types are detected from a prepared statement and wrapped with SELECT COLUMNS(*)::VARCHAR before execution, so side-effecting statements execute once.
  • Multi-statement and comment-only requests are rejected; Admin/STATS PostgreSQL extended-query handling stays a silent drop, while SQLITE sessions dispatch to the plugin.
  • DuckDB errors use specific error codes instead of generic 1064/42000, including MySQL errno 1451 for referenced-parent foreign key failures; PostgreSQL transaction status is tracked in plugin state, and result values preserve embedded NUL bytes.
  • MySQL clients that set CLIENT_DEPRECATE_EOF no longer receive EOF packets.
  • enable_external_access defaults to false; enabling it requires reopening the DuckDB engine.
  • In-flight queries are interrupted when the listener stops, and SELECT DATABASE() reports the configured path.
  • The build-tier DEBUG mismatch is a proven offset-shift corruption class and the loader now refuses mismatched plugins; the early assert(0) aborts remain plausible but unreproduced, documented with the evidence for each claim separated.

Build and operations

  • DuckDB 1.4.5 is vendored as a Git LFS archive and builds only with PROXYSQL40=1, so source builds require git lfs.
  • Adds Admin configuration and runtime views plus operator documentation under doc/duckdb/; LOAD DUCKDB VARIABLES TO RUNTIME applies only max_connections immediately.
  • Adds unit, lifecycle, plugin-load, and MySQL/PostgreSQL/Admin end-to-end coverage, plus a TAP benchmark comparing native DuckDB, plugin MySQL/PostgreSQL, and SQLite3 Server paths (skipped unless RUN_DUCKDB_BENCH=1).

Summary by CodeRabbit

  • New Features

    • Added DuckDB server support with MySQL and PostgreSQL connectivity.
    • Added configurable database settings, connection limits, read-only mode, and external-access controls.
    • Added Admin commands and runtime views for managing DuckDB variables.
    • Added scalar, NULL, and complex result handling.

  • Bug Fixes

    • Prevented duplicate execution of side-effecting statements.
    • Preserved embedded NUL bytes in results.
    • Improved plugin compatibility validation.

  • Documentation

    • Added DuckDB installation, configuration, operations, security, and usage guides.

  • Tests

    • Expanded unit, integration, protocol, listener, and source-verification coverage.