Feat/refactor layoutelement textregion to vectorized data structure
Merged
BA
2y ago+647+647+012345678901234567890123456789-429-429-012345678901234567890123456789
This PR refactors the data structure for list[LayoutElement] and list[TextRegion] used in partition pdf/image files.
- new data structure replaces a list of objects with one object with
numpyarray to store data - this only affects partition internal steps and it doesn't change input or output signature of
partitionfunction itself, i.e.,partitionstill returnslist[Element] - internally
list[LayoutElement]->LayoutElements;list[TextRegion]->TextRegions - current refactor stops before clean up pdfminer elements inside inferred layout elements -> the algorithm of clean up needs to be refactored before the data structure refactor can move forward. So current refactor converts the array data structure into list data structure with
element_array.as_list()call. This is the last step before turninglist[LayoutElement]intolist[Element]as return - a future PR will update this last step so that we build
list[Element]fromLayoutElementsdata structure instead.
The goal of this PR is to replace the data structure as much as possible without changing underlying logic. There are a few places where the slicing or filtering logic was simple enough to be converted into vector data structure operations. Those are refactored to be vector based. As a result there is some small improvements observed in ingest test. This is likely because the vector operations cleaned up some previous inconsistency in data types and operations.