fix(rank): retry postings that could not be fetched instead of expiring them

Merged
NI
25d ago
+330-21

What changed and why

Fixes #534 (reported by @ai-tvang). It's built to @MadsLorentzen's decisions on the issue, which also take in @ai-tvang's points about the time floor and the give-up listing. The PR is kept to rank.md, rank_state.py, their tests and the CHANGELOG, as asked.

A posting /rank could not fetch and a posting that is really closed both became expired. expired is terminal: candidates only selected new. So a 403, a login wall or a timeout on ranking day removed a live job from the pipeline, and the report presented it as closed.

1. unreachable (.claude/commands/rank.md Step 2).

  • unreachable: the agent's verdict after exhausting the 09-web-research.md escalation order, with a short "fetch_error". That covers 401/403, a login wall, a 404, a timeout, a 5xx or an empty page.
  • expired: stays for positive evidence of closure: a closed page, a 410, a redirect to a listing page, or a passed deadline.
  • A 404 is unreachable first, because of the #538 case.

2. Bounded retry (tools/rank_state.py).

  • candidates selects unreachable alongside new.
  • apply:
    • increments fetch_attempts;
    • keeps first_unreachable, the time of the first failure as a UTC timestamp, so the floor is measured in real hours;
    • stores last_fetch_error: a short reason, no page text, whitespace-collapsed and capped at 200 characters.
  • Giving up: after 3 attempts with at least 48 hours between the first and the last, apply writes expired with expired_reason: "unreachable". The retry fields stay as history, and the job is returned once under a new gave_up output key.
  • The report: Step 5 prints these jobs under "Gave up, check manually" with the URL, so a login-walled live posting is still seen by a human. A "Couldn't fetch - will retry" list shows the jobs still being retried, with attempt N/3.
  • Clearing: a later successful fetch, an explicit expired, or a passed deadline (the #560 path) clears the retry fields.
  • /rank --all keeps reviving everything, unchanged. Rule 6's note now says it covers jobs given up as unreachable.
  • --now: a new option (default: the current UTC time) makes the 48-hour floor testable, just as --today does for dates.

3. /scrape dedup. No change was needed. /scrape Step 3 already skips any URL or company+title present in seen_jobs.json, whatever its status, so unreachable counts as seen.

4. One source of truth. rank_state.py owns the status set as AGENT_STATUSES = ("scored", "expired", "unreachable"). apply rejects any other status as an error, so a misspelling doesn't surface as a misleading "missing score"; a result without a status is still scored, as before. In tests/test_rank_command.py, test_agent_status_contract_matches_the_state_helper loads AGENT_STATUSES and asserts that rank.md's JSON contract names exactly that set.

5. CHANGELOG under [Unreleased] / Changed. No migration is needed, since no existing entry carries the new status. rank.md's step text is updated together: Step 2's rules and contract (old lines 58 and 67), rule 6's --all note (old line 100), Step 4's fields, Step 5's report, and Important Rule 1 (old line 190).

09-web-research.md step 4 still says that in /rank, "declaring it unavailable" means marking the entry expired. I left it out to keep the PR to the files you named. It's a one-line follow-up (plus a framework_version bump) whenever you'd like it.

Failing case / reproduction (for fixes)

These are the documented tools/rank_state.py invocations against a seen_jobs.json holding one new posting. Each run is fed the result its spec version tells a scoring agent to return for a 403 that survives the curl retry.

On master, rank.md says to mark it expired:

apply 2026-10-14 -> expired: ['acme_data-engineer']
candidates on 2026-10-15 -> []
candidates on 2026-10-20 -> []          # never looked at again

On this branch, three failures on one day (a scheduled run plus manual ones) keep retrying:

apply 2026-10-14T07:00:00Z -> unreachable: [(1, '403 after curl retry')]  gave_up: []
apply 2026-10-14T12:00:00Z -> unreachable: [(2, 'timeout')]  gave_up: []
apply 2026-10-14T19:00:00Z -> unreachable: [(3, 'timeout')]  gave_up: []
next candidates -> ['acme_data-engineer']

Three failures with 48 hours from the first to the last give up, and the job is reported once:

apply 2026-10-14T09:00:00Z -> unreachable: [(1, '403 after curl retry')]  gave_up: []
next candidates -> ['acme_data-engineer']
apply 2026-10-15T09:00:00Z -> unreachable: [(2, 'timeout')]  gave_up: []
next candidates -> ['acme_data-engineer']
apply 2026-10-16T09:00:00Z -> unreachable: []  gave_up: [(3, 'login wall')]
next candidates -> []
stored: {'status': 'expired', 'expired_reason': 'unreachable', 'fetch_attempts': 3,
         'first_unreachable': '2026-10-14T09:00:00Z', 'last_fetch_error': 'login wall'}

Verification

New tests in tests/test_rank_state.py, class Unreachable:

  • a failed fetch is stored as unreachable with its retry fields;
  • unreachable entries are candidates again;
  • three failures with 47h59m between first and last keep retrying, while three exactly 48 hours apart give up and produce the gave_up row;
  • a later success clears the retry fields;
  • an explicit expired still expires at once and clears them;
  • fetch_error is bounded;
  • an unknown status is an error, and the entry is left untouched.

Plus the contract test in tests/test_rank_command.py, and the existing guard test_step4_preserves_every_field_step2_declares, which covers the new fetch_error.

Against master's rank_state.py and rank.md, all 8 Unreachable tests fail and the contract test errors (AGENT_STATUSES doesn't exist there). On the branch, rebased on today's master (including #560):

python3 tools/lint_skills.py                 lint_skills: OK (9 skills, 12 commands, settings.json)
python3 tools/check_framework_version.py     Framework Version Check: OK
python3 tools/security_guards.py             security_guards: OK
python3 -m unittest discover -s tests        Ran 583 tests, OK (skipped=10)

All of these were run on Windows 11 (10.0.26200) with Python 3.12, with PyYAML installed so the lint integration test runs. The constants (UNREACHABLE_MAX_ATTEMPTS = 3, UNREACHABLE_MIN_SPAN = timedelta(hours=48)) sit at the top of rank_state.py.

I worked on this with Claude Code (Claude Opus 5.5): it wrote the change and tests and ran the checks and the demonstration above.