fix(host-service): harden DB migration startup (busy_timeout + fail-closed)

KI
5mo ago
+322-5

Problem

The host-service runs DB migrations synchronously inside createApp() before the port binds (app.ts → serve.ts). The DB was opened with only WAL + foreign_keys (no busy_timeout), and createDb swallowed any migration error and returned the db anyway. desktop 1.12.0 added the first migration since this code shipped (0005_host_settings_and_project_overrides), exposing two failure modes under write-lock contention on host.db:

  • Half-migrated, served anyway — SQLITE_BUSY swallowed → runtime no such table host_settings.
  • Killed mid-migration — the synchronous stall exceeds the coordinator's 10s health window → SIGKILL → respawn loop, port never binds (terminals dead / 'can't type').

Fix

  • db/db.ts — set busy_timeout = 8000 (wait out transient contention) and fail closed: throw instead of swallowing. drizzle runs all pending migrations in a single BEGIN/COMMIT with ROLLBACK+rethrow, so a failure leaves the DB at its prior version (never half-applied). The throw propagates to serve.ts main().catch(… process.exit(1)), so the coordinator's health check fails instead of serving a broken DB.
  • host-service-utils.ts — raise HEALTH_POLL_TIMEOUT_MS 10s → 20s so a recoverable ~8s migration stall isn't SIGKILLed before the port binds.

Reproduction + verification

New integration test db.contention.node-test.ts: real createDb + real 0005 + a sibling process holding the write lock. Run with bun run test:integration:db (Electron-as-node — better-sqlite3 here is built for the Electron ABI; bun can't load it and plain node is the wrong ABI).

  • ✔ baseline: no contention → 0005 applies, host_settings exists
  • ✔ fail-closed: sustained contention → createDb throws, DB left cleanly unmigrated
  • ✔ recovery: contention clears within the timeout → waits then succeeds

Host-service unit suite 691/0, typecheck + lint clean.

Scope / follow-up

This hardens migration startup (resilience + correctness). It does not address the source of the contention — whether two host-services can race the same host.db — which is a separate, root-cause lifecycle change (reap orphan host-service before spawn / single-writer guarantee) tracked in a follow-up PR. The plan doc (plans/host-service-migration-startup-hardening.md) covers both.

Open in Stage

Summary by cubic

Hardened host-service DB migration at startup by failing closed so we never serve a half-migrated DB. Removed the migration busy_timeout and focused on surfacing failures; added an Electron-as-node integration test for write-lock contention.

  • Bug Fixes

    • src/db/db.ts: removed the try/catch that swallowed migration errors; migrate() now throws. drizzle keeps migrations atomic, so failures roll back cleanly. Dropped the migration busy_timeout.

  • Tests

    • src/db/db.contention.node-test.ts: integration test holds a write lock in a sibling process and asserts fail-closed behavior.
    • scripts/run-db-integration-test.ts + test:integration:db: run under Electron-as-node to load better-sqlite3.

Summary by CodeRabbit

  • Bug Fixes

    • Improved database migration error handling for host service to ensure the application fails cleanly at startup when database contention is detected, rather than silently continuing with an unmigrated database.

  • Tests

    • Added integration test suite to verify database behavior and recovery under write-lock contention scenarios.