feat(hubspot): re-fetch failed documents by id (Resolver)

Merged
EV
3d ago
+641-35

Description

HubSpotConnector now implements Resolver. A targeted reindex re-reads the failed HubSpot records by id. It does not run a crawl or use a checkpoint.

How it works

  • reindex parses each hubspot_<noun>_<id> document id (parse_hubspot_document_id). It groups the ids by object type.
  • It reads each group with the CRM batch-read API, 100 ids for each call. All calls go through the gateway's rate limiter.
  • It builds each document with _build_document. I moved this helper out of the crawl loop, so the crawl and the reindex use the same code. The yielded Document.id is the input id exactly.
  • HubSpot has no hierarchy nodes, so reindex yields none. The crawl also yields none.

Covered document types: tickets, companies, deals and contacts. These are all the types the connector writes. Notes are not documents. They are sections in their parent record, so a reindex of the parent reads them again.

Not covered: none.

Failures

  • An id that does not parse, or names an unknown type, gives a ConnectorFailure for that id.
  • An id whose type is not in the connector's object_types gives a ConnectorFailure (out of scope).
  • An id that HubSpot does not return (deleted, archived, unreadable, or merged into another record) gives a ConnectorFailure.
  • Entity failures are skipped, as in Confluence. The HubSpot crawl never yields them.
  • A 401, a missing credential, or an exhausted rate limit ends the run. The crawl also raises on these errors. Other errors fail only the targets of that batch.

Permissions: with include_permissions=True (perm-synced cc-pair), reindex sets external_access with HubSpotPermissionReader. The doc sync uses the same reader. It asks the permitted-users API for viewers, 20 records for each call. If the viewer read fails for a record, reindex yields a failure for it. It does not index the record without access.

Filter parity with the crawl. I compared each crawl filter with the by-id read:

  • object_types config: applied. An id of another type is a failure and is not read.
  • Archived records: the crawl's listing and search return unarchived records only. The batch read now sends archived=False explicitly. Reindex also drops a record that HubSpot marks archived (new archived field on HubSpotRecord).
  • Merged records: HubSpot can return a merged record under its new id. Reindex yields only the ids it was asked for. The crawl indexes the new id on its own.
  • Code-side skips: the crawl has none.
  • The time window is not an access filter, so reindex does not apply it.

I also updated the HubSpot row in .agents/feature-map/components/permission-sync.md. It now says that a targeted reindex also sets access.

How Has This Been Tested?

  • New daily test backend/tests/daily/connectors/hubspot/test_hubspot_resolver.py runs against the CI HubSpot portal (HUBSPOT_ACCESS_TOKEN). It covers: one record, records of each type, a bogus id that is well-formed, ids that do not parse, an out-of-scope type, empty input, entity-only input, and include_permissions=True with external_access set. 8 passed.
  • The test portal has no tickets. The "each type" test covers companies, deals and contacts, plus any type the portal has records of.
  • New unit test backend/tests/unit/onyx/connectors/hubspot/test_hubspot_reindex.py: an archived record, a merged record, an unconfigured type, a 401 that ends the run, and a 500 that fails only its batch.
  • Existing tests: backend/tests/daily/connectors/hubspot/ 28 passed; HubSpot unit tests 61 passed.

Additional Options

  • [Optional] Please cherry-pick this PR to the latest release version.
  • [Optional] Override Linear Check

🤖 Generated with Claude Code

Summary by cubic

HubSpotConnector now implements Resolver, so a targeted reindex re-reads failed records by id through the batch-read API instead of running a crawl.

  • Parses each hubspot_<noun>_<id> id, groups ids by object type, and batch-reads 100 records at a time through the gateway's rate limiter. The batch read sends archived=False explicitly, and reindex drops records HubSpot marks archived.
  • Unparseable ids, out-of-scope types, and records HubSpot doesn't return (deleted, archived, or merged) yield a ConnectorFailure. A 401, missing credential, or exhausted rate limit ends the run; other errors fail only their batch.
  • The portal id is read once per run, and a target whose associations or notes cannot be read fails rather than indexing partial content — the crawl still skips those reads.
  • With include_permissions=True, reindex sets external_access through HubSpotPermissionReader; a viewer-read failure prevents indexing that record.
  • Added daily and unit tests covering reindex behavior, failures, and permission sync; entity failures are skipped, as in Confluence.

Summary by CodeRabbit

  • New Features

    • Added targeted reindexing for individual HubSpot records across configured object types.
    • Supports permission-aware access while preserving document metadata.
    • Continues processing eligible records while reporting failures for invalid, missing, inaccessible, archived, or unsupported records.

  • Bug Fixes

    • Archived records are consistently excluded from synchronization and batch reads.
    • Improved handling of association and record-read failures during reindexing.

  • Documentation

    • Updated HubSpot connector documentation to describe targeted reindexing and permission behavior.