fix: preserve element fields when flattening DataFrame metadata

AF
10h ago
+112-3

Summary

convert_to_dataframe() flattens metadata into each element dictionary with dict.update(). Dynamic metadata named text, type, element_id or embeddings therefore replaces the actual element value. Columns that only appear on another row, such as a checkbox's checked, can also receive unrelated metadata values.

Reserve element columns across the whole batch before flattening. Give conflicting metadata fields a unique metadata_ prefix, keeping both values. Existing nonconflicting metadata names, including already-prefixed names, keep their columns. Iterables, dtype conversion, empty-column handling and the nested record-locator value retain their existing behavior.

This fixes the DataFrame export path. The CSV exporter changes in #4549 are separate and are not included here.

Validation

  • The 13 new cases produce 11 failures and 2 passing controls on the original source.
  • Covers each core column with and without dtype conversion, multiple existing prefix levels, cross-row checkbox metadata in both input orders, generator input, ordinary metadata and empty input.
  • A combined fork validation branch first runs the new regressions against the pinned original production files: 23 failures, 9 passing controls. Restoring the final source files yields 463 passed across the new cases and the existing staging base, HTML parser and HTML partition suites on Python 3.12.
  • Repository-configured Ruff check and format check pass; source-integrity checks pass after the before/after run.
  • Validation run, exact commands and JUnit artifacts.

Checklist

  • User-facing conventional-commit title.
  • Maintainer: please apply the bug category label.