fix: preserve reference CSV cell text in table accuracy evaluation

AF
8h ago
+74-1

Summary

eval_table_transformer_for_file() lets Pandas infer the ground-truth CSV's cell types before converting them back to strings. That changes reference text: a blank cell becomes the literal string nan, identifiers such as 007 lose leading zeroes, N/A/NULL become missing values, and numeric spellings change. Even an exact prediction can then receive an inaccurate score.

Read reference cells directly as strings with default NA inference disabled. Keep empty cells empty, preserve their literal content and leave the prediction-side normalization and scoring functions unchanged.

Validation

  • The 18 new cases produce 11 failures and 7 passing controls on the original source.
  • New tests plus neighboring table evaluation/alignment/format suites: 38 passed, 3 deselected on Python 3.12. The three deselected existing tests perform model inference; no inference-quality claim is made here.
  • Tests write and read real CSV files and use the installed unstructured-inference comparison functions with both ratio and token_ratio. Only the expensive image-to-table prediction step is replaced with a supplied DataFrame.
  • Coverage includes empty cells, zero-prefixed identifiers, literal NA tokens, decimal/exponent notation, quoted commas, embedded newlines, Unicode, prediction-side NaN and incorrect predictions that must still be penalized. Input prediction frames remain unchanged.
  • Repository-configured Ruff check, format check and source-integrity checks pass.
  • Validation run, dependency identities and JUnit artifacts.

Checklist

  • User-facing conventional-commit title.
  • Maintainer: please apply the bug category label.