feat(account): rotate on quota exhaustion (task 25)

Merged
MP
6mo ago
+1244-16

Summary

Implements automatic account rotation on quota exhaustion (HTTP 429 or traffic limit). When an account hits its quota limit, rotator marks it exhausted with a configurable TTL cooldown, excludes it from selection, and automatically picks the next best account per the selection strategy (BestTraffic/RoundRobin). Enables seamless fallback across premium/debrid/free accounts without manual intervention. Closes task 25, unblocks downstream tasks (38, 51-52, 55-56, 75-76).

Why

Multi-account orchestration requires intelligent failover. Task 24 (autoselector) picks the best account; task 25 (rotator) handles graceful degradation on quota loss. Together they enable:

  • Transparent account rotation on hoster rate limits
  • Cross-service isolation (MediaFire's cooldown doesn't affect Uploaded)
  • Traffic-aware refresh (clear exhaustion only when confirmed available)
  • Zero user intervention — download retries automatically with next account

Rationale for design: in-memory cooldown map (not persisted) because window is short (5-15 min) and upstream re-sends 429 anyway on retry. Exclusive deadline semantics for expiry (now >= deadline = available).

Changes

  • New module AccountRotator in src-tauri/src/application/services/account_rotator.rs (750+ lines):

    • NextAccountOutcome enum: Picked(Account) / NoneAvailable / AllExhausted { next_eligible_at_ms } with error_message() returning PRD-frozen wording
    • mark_exhausted(account_id, service_name, ttl_secs) — inserts cooldown deadline into map, publishes AccountExhausted event
    • next_account(service, strategy) — snapshots exhausted ids (lazy prune expired), calls selector.select_best_excluding(..., &exhausted_ids), distinguishes NoneAvailable (zero accounts exist) from AllExhausted (all on cooldown) by querying repo directly
    • record_traffic_refresh(account_id, traffic_left, threshold) — clears marker ONLY if traffic_left >= threshold (no flip-flop on unknown/below-threshold)
    • is_quota_signal(http_status, traffic_left, threshold) — returns true on 429 OR traffic < threshold (threshold exclusive at equality)
    • 22 unit tests covering quota detection, rotation, all-exhausted state, cooldown expiry (exclusive deadline), cross-service isolation, traffic refresh semantics, TTL edge cases

  • Modified AccountSelector:

    • New select_best_excluding(service, strategy, exclude: &[AccountId]) — identical to select_best() but filters out ids in exclude list
    • select_best() now delegates to select_best_excluding() with empty exclude list
    • 3 new tests: step-by-step filtering, all-excluded → no event, RoundRobin over non-excluded subset

  • Added AccountExhausted event variant to domain/event.rs:

    • Carries: id: AccountId, service_name: String, exhausted_until_ms: u64 (Unix epoch deadline)

  • Added transient exhausted_until: Option<u64> field to Account:

    • Methods: mark_exhausted(until_ms), clear_exhausted(), is_exhausted(now_ms) -> bool, exhausted_until() -> Option<u64>
    • Reset to None on reconstruct() (transient state not persisted)
    • 4 unit tests: new-no-exhaustion, reconstruct resets, mark/clear lifecycle, expiry boundary

  • Wired AccountRotator into CommandBus:

    • Builder: with_account_rotator(arc), accessor: account_rotator()
    • Initialized to None in new()

  • Updated adapter bridges:

    • tauri_bridge.rs: added AccountExhausted → "account-exhausted" with JSON payload (camelCase)
    • download_log_bridge.rs: added AccountExhausted arm to exhaustive match (event not download-scoped, ignored)

  • Updated CHANGELOG.md:

    • Added comprehensive entry under [Unreleased] → Added section (270+ words)
    • Lists all 4 acceptance criteria with test names
    • Explains architecture choice (in-memory, not SQLite)
    • Documents all API methods and semantics

Testing

All tests green: 1172 lib tests, 0 regressions.

cargo test --lib account_rotator       # 22 rotator + 3 selector tests
cargo test --workspace                 # 1172 total

Acceptance criteria verified:

  • ✅ Quota detection (429 + traffic_left check) → rotation to next account — test_quota_detection_to_rotation_full_flow
  • ✅ All accounts exhausted → Error state with PRD-frozen message — test_next_account_returns_all_exhausted_when_every_candidate_is_marked + test_outcome_error_message_uses_prd_wording
  • ✅ Cooldown expiry after refresh confirms availability — test_record_traffic_refresh_clears_cooldown_when_confirms_available + edge cases (None, below-threshold stay marked)
  • ✅ Cross-service isolation tested — cooldown deadline for MediaFire doesn't leak into Uploaded decision

Additional coverage:

  • Exclusive deadline semantics (now >= deadline = available)
  • Lazy pruning of expired entries from map
  • TTL zero edge case (no-op)
  • Idempotent clear_exhausted
  • Traffic threshold boundary (exclusive at equality)

Related Issues

  • Closes task 25
  • Refs task 24 (autoselector — dependency)
  • Unblocks: tasks 38, 51, 52, 55, 56, 75, 76 (all dependent on account rotation)

Notes for Reviewer

Architecture notes:

  1. In-memory state — cooldown map lives in Mutex<HashMap> (not SQLite) because:

    • Cooldown window is short (5-15 min); loss on restart is acceptable
    • Upstream hoster re-sends 429 anyway on retry → no correctness gap
    • Avoids schema migration + background purge job
    • Simpler test story (no SQLite mocking)

  2. NoneAvailable vs AllExhausted distinction — After selector.select_best_excluding(...) returns Ok(None):

    • Call repo.list_by_service() directly (not from map) to check: enabled + not expired
    • Empty set → NoneAvailable (zero credentials exist)
    • Non-empty set → AllExhausted (rotation is blocker, not absence)
    • Extract deadline only from live accounts, not map (prevents cross-service leakage)

  3. Exclusive deadline semantics — now >= deadline = available (not exhausted):

    • Clock doesn't drift; microsecond boundaries don't matter in practice
    • Matches intuition: "expires AT this time, so available AT this time"
    • Pinned with unit test test_mark_exhausted_records_deadline_and_flips_is_exhausted

  4. Traffic refresh only clears on confirmation — record_traffic_refresh() clears marker ONLY if traffic_left >= threshold:

    • If traffic_left == None (unknown): leave marker untouched (can't confirm availability)
    • If traffic_left < threshold (below floor): leave marker untouched (still exhausted)
    • Prevents silent undo for hosters without traffic API; prevents flapping near threshold

Checklist

  • Tests added and passing (22 rotator + 3 selector = 25 new tests, all green)
  • Docs updated (CHANGELOG.md with comprehensive feature description)
  • No secrets, debug prints, commented code
  • Self-reviewed diff
  • CI ready (cargo build, cargo clippy -D warnings, cargo fmt --check all green)

Summary by cubic

Adds automatic account rotation when a hoster quota is hit and hardens race handling so only committed picks emit AccountSelected. Prevents false “no account”/“all exhausted” states, stale selections during races, and noisy telemetry. Fulfills task 25.

  • New Features

    • AccountRotator with mark_exhausted, next_account -> Picked | NoneAvailable | AllExhausted { next_eligible_at_ms }, and error_message(service).
    • Quota detection: is_quota_signal (429 or traffic_left below threshold). record_traffic_refresh only clears cooldown when traffic_left >= threshold.
    • New DomainEvent::AccountExhausted forwarded by the Tauri bridge as account-exhausted (download log ignores it).
    • Account gains transient exhausted_until (reset on reconstruct).
    • CommandBus wiring: with_account_rotator(...) and account_rotator().

  • Bug Fixes

    • Selector: select_best_excluding(service, strategy, exclude_ids); RoundRobin respects exclusions; emit NoAccountAvailable only when the pre-exclusion eligible set is empty.
    • New select_best_excluding_quiet(...) used by the rotator; rotator emits AccountSelected only for a pick that survives the under-lock recheck.
    • Rotator race-guards: re-check picked id under lock and retry with it excluded if a concurrent mark_exhausted lands; when selection returns none, re-snapshot cooldowns and retry if any id from the full current exclude list (including race-added ids) was just cleared, avoiding false AllExhausted.
    • Preserve longer cooldowns on re-mark: mark_exhausted keeps max(existing, proposed) and publishes the committed deadline to avoid shortening an active window.

Summary by CodeRabbit

  • New Features

    • Quota-aware account rotation with cooldowns, earliest-deadline handling, automatic recovery and exhaustion notifications per service
    • Automatic account selection with exclusion-aware strategies, round-robin behavior and strict strategy validation
    • Accounts management UI: import/export, enable/disable, validate, edit, delete, IPC wiring and i18n
    • Server-side account queries returning secret-safe results and stable selection semantics

  • Tests

    • Extensive tests covering rotation, selection, cooldown/refresh behavior, events, and query shapes