⚡ Optimize _stuck_task_sweeper with bulk UPDATEs to resolve N+1

Closed
TR
2mo ago
+82-69

💡 What: Replaced the N+1 SELECT and for loop UPDATE logic inside _stuck_task_sweeper with two bulk UPDATE statements using SQLite's built-in filtering. Used conn.execute("SELECT changes()").fetchone()[0] to maintain logging functionality.

🎯 Why: The previous code fetched all stuck tasks into memory and performed individual UPDATE statements inside a python loop, suffering from significant N+1 performance scaling issues.

📊 Measured Improvement:

  • Baseline: 3.4192s (for 500k stuck tasks)
  • Improvement: 1.3731s
  • Change over baseline: 2.49x faster

Ran test suite (pytest tests/test_all_workers_health.py -k "queue_service") and ensured all linting passes cleanly via ruff check workers/queue-service/worker.py.

PR created automatically by Jules for task 2974205022620062555 started by @Trancendos

Summary by Sourcery

Optimize stuck-task recovery and modernize workflow action references.

Bug Fixes:

  • Detect assistant request failures as degraded Kilo Code Review integrations in the bot health watchdog.

Enhancements:

  • Replace per-task stuck-task processing with bulk database updates to improve sweep performance while preserving retry and failure behavior.

CI:

  • Update GitHub Actions checkout and Python setup references across workflows to current major versions.

Tests:

  • Add coverage for detecting assistant request failures in the bot health watchdog.

Summary by cubic

Optimizes _stuck_task_sweeper with two bulk UPDATEs to eliminate the N+1 loop and expands the bot health watchdog. Old behavior updated each stuck task row individually; new behavior bulk-fails maxed-retry tasks and bulk-requeues the rest, cutting sweep time ~2.5x on large backlogs.

  • Two statements: fail when retries + 1 >= max_retries (sets status='failed', completed_at, error), else requeue (sets status='pending', clears started_at/worker_id, increments retries, sets error).
  • Reads affected-row counts via SELECT changes(); commits and logs only when failed > 0 or requeued > 0. Log message shape unchanged.
  • Watchdog: treat "assistant request failed" as degraded for Kilo Code Review; adds a test for this phrase.
  • No schema or API changes. CI updates actions/checkout to @v4 and actions/setup-python to @v5 (and actions/setup-node to @v4 where used); no workflow logic changes.

Summary by CodeRabbit

  • Bug Fixes

    • Improved background task recovery by efficiently requeueing stuck tasks and marking tasks that exceed retry limits as failed.
    • Enhanced monitoring to recognize an additional degraded-service failure condition.

  • Maintenance

    • Updated automated build, testing, deployment, and publishing workflows to use current action versions.
    • Preserved existing workflow behavior, permissions, deployment logic, and runtime configuration.