fix(documents): count the hash-id sequence number per page, not per contiguous run

EA
2d ago
+26-4

Summary

Fixes #4620

assign_and_map_hash_ids() counts the "sequence number on page" per contiguous run of equal page numbers, so an element's id depends on which elements precede it. When the elements of two partitions of the same document are combined, page numbers can come back later in the list, and two distinct elements on the same page that share their text then receive the same id — old_to_new_mapping silently merges their parent_ids.

This counts per page number instead, matching the comment on the block, Element.id_to_hash() ("sequence number on page"), and the sibling _assign_hash_ids() in unstructured/partition/common/metadata.py.

Testing

import copy
from unstructured.documents.elements import ElementMetadata, Text, assign_and_map_hash_ids

elements = [
    Text(text="Section", metadata=ElementMetadata(page_number=1)),
    Text(text="Interlude", metadata=ElementMetadata(page_number=2)),
    Text(text="Section", metadata=ElementMetadata(page_number=1)),
]
ids = [e.id for e in assign_and_map_hash_ids(copy.deepcopy(elements))]
  • before: 3 elements, 2 unique ids (both "Section" elements hash alike)
  • after: 3 unique ids

test_unstructured/documents/test_elements.py with the new test only: 3 failed / 79 passed. With the change: 2 failed / 80 passed. The two remaining failures are pre-existing fixture gaps in this checkout (example-docs/test_evaluate_files/unstructured_output/form.json is absent) and fail identically without the change.

Ids for ascending page numbers are unchanged — test_hash_ids_are_deterministic (which pins exact ids) and the other hash-id tests still pass.

Notes

groupby is no longer used anywhere in the module, so the import goes away with it.