fix(html): leave ruby text out of the paragraph text
GA
7h ago+11+11+01234567890123456789-0-0-0123456789
Summary
partition_html() glues ruby text to the base text it annotates. Japanese pages use ruby for furigana, and Chinese pages use it for pinyin:
from unstructured.partition.html import partition_html
partition_html(text="<p>首都は<ruby>東京<rt>とうきょう</rt></ruby>です</p>")[0].text
# before: '首都は東京とうきょうです'
# after: '首都は東京です'
<rt> and <rp> had no element class, so DefaultElement treated them as transparent. Their text was then merged into the sentence, which corrupts search, embeddings and chunk text for 東京. This change maps both to RemovedPhrasing, like other content that isn't rendered as body text. The tail after each tag is kept. <rp> holds the fallback parentheses that browsers hide when they support ruby.
Validation
- Added
and_it_is_used_for_ruby_text_so_it_is_not_glued_to_the_base_texttoDescribeRemovedPhrasing. It covers<rp>and two ruby pairs in one element. It fails onmainand passes with this change. pytest test_unstructured/partition/html test_unstructured/documents: 667 passed.ruff checkandruff format --checkpass on the changed files.
Checklist
- The PR title is a user-facing release-note entry with a conventional commit prefix.
- Apply a category label (
bug): I can't add labels, so please apply it. - Not a breaking change.