File Magic Numbers

There are certain points in Haystack where we perform file type checks based on their extensions (e.g. .txt .pdf). As pointed out in #708 by @lalitpagaria, it would be better to do this using file magic numbers. We could implement this in 2 places:

• When performing file conversion
• When downloading a file using fetch_archive_from_http

TODO:

• implement double check for docx (e.g. if file_type == "xml" and extension ==".docx")
• Investigate behaviour on compressed files, compare when the uncompressed param is set to True/False

[brandenchan]

Actually docx is nothing but compressed collection of xml files. https://docs.fileformat.com/word-processing/docx/

Ahhh good to know! Thanks

I ran the above experiment again using this python magic object with some new params and actually got much better and more interpretable results.

f = magic.Magic(mime=False, uncompress=False)

"""
Results

bert
PDF document, version 1.5

classics
UTF-8 Unicode text, with very long lines

everything.tar.gz
gzip compressed data, from Unix, original size modulo 2^32 189440000 gzip compressed data, reserved method, ASCII, has CRC, extra field, has comment, encrypted, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 189440000

heavy_metal
Microsoft OOXML

nq-dev-small.json
ASCII text, with very long lines, with no line terminators

nq-dev-small.json.gz
gzip compressed data, from Unix, original size modulo 2^32 109076480

nq-dev-small.zip
Zip archive data, at least v2.0 to extract

test_file_magic.py
Python script, ASCII text executable
"""

I think it would be worth going ahead with incorporating this package! @lalitpagaria would you be interested in opening a PR?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 21 days if no further activity occurs.