Handle non-trivially-copyable types in Loop/Scan output concatenation

Merged
YU
4mo ago
+343-13

This pull request improves the handling of string tensors in the CPU implementation of the Loop operator, ensuring correct memory management and copy semantics for non-trivially-copyable types like std::string. It also adds comprehensive unit tests to verify these behaviors, especially for cases involving string scan outputs and loop-carried variables.

Enhancements for string tensor support:

  • Updated ConcatenateCpuOutput in loop.cc to properly detect string tensors and use std::copy for concatenation, ensuring correct handling of heap-allocated string payloads and avoiding unsafe byte-wise copying. [1] [2]
  • Modified OutputIterator::ZeroOutCurrent in scan_utils.h to skip zeroing for string tensors, as their elements are already default-constructed and cannot be safely set with memset.

New and extended unit tests for string tensor scenarios:

  • Added tests to loop_test.cc covering:
    • String scan outputs, including long strings that exceed the small-string-optimization threshold and multi-element outputs.
    • Loop-carried string variables to ensure they are copied correctly.
    • Zero-iteration cases to confirm empty string scan outputs are handled without errors.