fix(html): leave ruby text out of the paragraph text

GA
7h ago
+11-0

Summary

partition_html() glues ruby text to the base text it annotates. Japanese pages use ruby for furigana, and Chinese pages use it for pinyin:

from unstructured.partition.html import partition_html

partition_html(text="<p>首都は<ruby>東京<rt>とうきょう</rt></ruby>です</p>")[0].text
# before: '首都は東京とうきょうです'
# after:  '首都は東京です'

<rt> and <rp> had no element class, so DefaultElement treated them as transparent. Their text was then merged into the sentence, which corrupts search, embeddings and chunk text for 東京. This change maps both to RemovedPhrasing, like other content that isn't rendered as body text. The tail after each tag is kept. <rp> holds the fallback parentheses that browsers hide when they support ruby.

Validation

  • Added and_it_is_used_for_ruby_text_so_it_is_not_glued_to_the_base_text to DescribeRemovedPhrasing. It covers <rp> and two ruby pairs in one element. It fails on main and passes with this change.
  • pytest test_unstructured/partition/html test_unstructured/documents: 667 passed.
  • ruff check and ruff format --check pass on the changed files.

Checklist

  • The PR title is a user-facing release-note entry with a conventional commit prefix.
  • Apply a category label (bug): I can't add labels, so please apply it.
  • Not a breaking change.