fix: handle encoding for text file checks
Merged
MT
3y ago+35+35+01234567890123456789-9-9-0123456789
Summary
Closes #705. Updates the text file checks for JSON and CSV structured in detect_filetype to handle documents with different encodings.
Testing
This code should now run without error
from unstructured.file_utils.filetype import _is_text_file_a_csv, _is_text_file_a_json
filename = "example-docs/fake-text-utf-32.txt"
_is_text_file_a_csv(filename=filename)
_is_text_file_a_json(filename=filename)