Feat/refactor layoutelement textregion to vectorized data structure

Merged
BA
2y ago
+647-429

This PR refactors the data structure for list[LayoutElement] and list[TextRegion] used in partition pdf/image files.

  • new data structure replaces a list of objects with one object with numpy array to store data
  • this only affects partition internal steps and it doesn't change input or output signature of partition function itself, i.e., partition still returns list[Element]
  • internally list[LayoutElement] -> LayoutElements; list[TextRegion] -> TextRegions
  • current refactor stops before clean up pdfminer elements inside inferred layout elements -> the algorithm of clean up needs to be refactored before the data structure refactor can move forward. So current refactor converts the array data structure into list data structure with element_array.as_list() call. This is the last step before turning list[LayoutElement] into list[Element] as return
  • a future PR will update this last step so that we build list[Element] from LayoutElements data structure instead.

The goal of this PR is to replace the data structure as much as possible without changing underlying logic. There are a few places where the slicing or filtering logic was simple enough to be converted into vector data structure operations. Those are refactored to be vector based. As a result there is some small improvements observed in ingest test. This is likely because the vector operations cleaned up some previous inconsistency in data types and operations.