fix(staging): keep element embeddings when rehydrating serialized elements
Summary
Text.to_dict() writes an element's vector to a top-level "embeddings" key, and the embedders in unstructured-ingest write the same key (EMBEDDINGS_KEY in unstructured_ingest/embed/interfaces.py). elements_from_dicts() rebuilds each element from text, element_id and metadata only, so every deserializer that goes through it silently drops the vectors:
from unstructured.documents.elements import NarrativeText, Title
from unstructured.staging.base import elements_from_json, elements_to_json
elements = [
Title("Intro", embeddings=[0.1, 0.2, 0.3]),
NarrativeText("Hello there.", embeddings=[0.4, 0.5, 0.6]),
]
serialized = elements_to_json(elements) # both element dicts contain "embeddings"
[e.embeddings for e in elements_from_json(text=serialized)]
# main: [None, None]
# this PR: [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]
elements_from_json(text=serialized) == elements
# main: False (Text.__eq__ compares embeddings); this PR: True
The same happens with elements_from_dicts() itself, with partition_json() and partition_ndjson() rehydration (and so in partition() on a .json file of elements), and to a chunk's orig_elements when its metadata goes through ElementMetadata.from_dict(). Loading embedded output, whether written by elements_to_json() or by an ingest pipeline with an embedder, returns elements without their vectors and without an error. On main, in a JSON round trip of elements with every metadata field set, embeddings is the only value lost.
elements_from_dicts() now passes embeddings=item.get("embeddings") in the two places it builds a Text subclass: the TYPE_TO_TEXT_ELEMENT_MAP branch and the TableChunk branch. A dict without the key still gives embeddings=None, and CheckBox, which has no embeddings, is unchanged. This continues #4504, which made partition_json() rehydration keep element ids and stored metadata.
One visible effect: when rehydrated elements are chunked again with include_orig_elements=True (the default), the chunks' orig_elements now carry the vectors, the same as when the in-memory elements are chunked. This can make the serialized chunks much larger; callers who do not want the vectors carried can pass include_orig_elements=False. The rehydrated element gets its own copy of the vector list, as ElementMetadata.from_dict() does for metadata.
Validation
Five new tests fail on main and pass with this change:
test_unstructured/staging/test_base.py::test_elements_from_dicts_keeps_element_embeddings:Title,NarrativeTextandTableChunkkeep their vectors, aCheckBoxstill has none, and re-serializing gives back the original dicts.test_unstructured/staging/test_base.py::test_elements_from_json_keeps_element_embeddings:elements_from_json(text=elements_to_json(elements)) == elements.test_unstructured/staging/test_base.py::test_orig_elements_keep_their_embeddings_when_chunk_metadata_is_rehydrated:ElementMetadata.from_dict(metadata.to_dict()).orig_elements == orig_elements.test_unstructured/partition/test_json.py::test_partition_json_preserves_element_embeddings_during_rehydrationtest_unstructured/partition/test_ndjson.py::and_it_keeps_the_embeddings_of_rehydrated_elements
# main source + the new tests
FAILED test_base.py::test_elements_from_dicts_keeps_element_embeddings - assert [None, None, None, None] == [[0.1, 0.2, 0...8, 0.9], None]
FAILED test_base.py::test_elements_from_json_keeps_element_embeddings - At index 0 diff: <...Title object at 0x...> != <...Title object at 0x...>
FAILED test_base.py::test_orig_elements_keep_their_embeddings_when_chunk_metadata_is_rehydrated - At index 0 diff: <...Title object at 0x...> != <...Title object at 0x...>
FAILED test_json.py::test_partition_json_preserves_element_embeddings_during_rehydration - assert [None, None] == [[0.1, 0.2, 0....4, 0.5, 0.6]]
FAILED test_ndjson.py::and_it_keeps_the_embeddings_of_rehydrated_elements - assert [None, None] == [[0.1, 0.2, 0....4, 0.5, 0.6]]
5 failed
# this PR
5 passed
Wider run on Windows (Python 3.12) over test_unstructured/chunking, test_unstructured/staging, documents/test_elements.py, documents/test_ontology_to_unstructured_parsing.py, embed/test_mixedbreadai.py, embed/test_voyageai.py, partition/common/test_json_partitioning.py, partition/test_api.py, partition/test_json.py, partition/test_ndjson.py, partition/test_xml.py and test_runtime_telemetry.py:
mainsource with the new tests: 12 failed, 1363 passed, 10 skipped.- This PR: 7 failed, 1368 passed, 10 skipped.
The difference is the five new tests. The other 7 failures are the same on both and come from the Windows environment: UTF-8 fixtures read as cp1252 (the two test_elements_to_md_conversion cases and the ontology fixture), a backslash path separator in test_elements.py, the JSON decoder reaching the recursion limit first in the two too_deep_to_pretty_print tests, and file-type detection without libmagic in it_rehydrates_on_a_detect_then_partition_sequence_over_the_same_file_handle.
Also checked with a script: each of these returns embeddings=None on main and the original vectors with this PR: partition(filename=...) on a .json file written by elements_to_json(), elements_from_json(filename=...), element dicts given an "embeddings" key in place the way an embedder does, and the orig_elements of partition_json(..., chunking_strategy="basic", include_orig_elements=True). An element serialized without embeddings still rehydrates with embeddings=None.
ruff check . and ruff format --check . pass (ruff 0.15.10, the version in uv.lock).
Release note (optional)
elements_from_json(), elements_from_dicts(), partition_json() and partition_ndjson() now keep each element's embeddings when they load serialized elements, so embedded output reads back with its vectors instead of None.
Checklist
- The PR title is a user-facing release-note entry with a conventional commit prefix.
- Apply a category label (
enhancement,bug, or an appropriate existing label), or ask a maintainer to apply it. This is abugfix; could a maintainer please apply thebuglabel? I can't set labels on this repository. - For breaking changes, use
!in the title and explain impact and migration above. (Not a breaking change.)