Parse Excel from in-memory file object

We came across a situation where we had a file object representing Excel data (came from HTTP POST but I’m thinking it could also come from MongoDB for example), and would’ve liked to pass it directly to Pandas to parse (vs saving it to disk and passing path to Pandas).

Could this be possible?

I saw that xlrd had file_contents as a possible argument of open_workbook: https://github.com/python-excel/xlrd/blob/master/xlrd/__init__.py#L385

Maybe ExcelFile in Pandas could take path_or_buffer as argument, and pass along the correct one to xlrd. https://github.com/pydata/pandas/blob/master/pandas/io/parsers.py#L1133

Don’t know if that could work for openpyxl also.

Thoughts?

Thanks! Nicolas

[nicolashery]

Haha well done! I didn’t know we were racing 😉

Yeah didn’t get a chance to work on it as much as I wanted. But I did spot:

• xlrd accepts path or bytes (coming from file.read()) https://github.com/python-excel/xlrd/blob/master/xlrd/__init__.py#L385
• openpyxl accepts path or file object (not file-like, has to be an instance of file, which is kind of limitting…) https://github.com/chronossc/openpyxl/blob/master/openpyxl/reader/excel.py#L43

I also saw that xlrd was going to support Excel 2007 in future versions, I don’t know what you want to do about that, ie keep using both, or switch to only xlrd.

I didn’t know about tempfile (nice trick!). I was going to just feed xlrd the bytes from file.read(), but that wouldn’t work for openpyxl. I also had put together a function to check what type of Excel file it is, inspired with how the master branch of xlrd does it:

def _excel_type(filepath_or_buffer):
    # Thanks to xlrd for this
    peeksz = 4
    if isinstance(filepath_or_buffer, str):
        f = open(filepath_or_buffer, "rb")
        peek = f.read(peeksz)
        f.close()
    elif hasattr(filepath_or_buffer, 'read') \
            and hasattr(filepath_or_buffer, 'seek'):
        f = filepath_or_buffer
        peek = f.read(peeksz)
        f.seek(0)
    else:
        raise TypeError("You must provide the path to a file "
                        "or a file-like object")
    # Check if ZIP file
    if peek == "PK\x03\x04" \
        or peek == "PK\x03\x04".encode('latin1'): # Python 3
        return 'xlsx'
    else:
        return 'xls'

Then I would’ve checked the type (and I like your way of checking for file-like, ie needs a read method, possibly a seek too, this way it works with “file objects” coming from Flask HTTP uploads, or MongoDB GridFS…), and done:

wb = xlrd.open_workbook(filename=filename)
# or
f = open(filename, 'rb')
bytes = f.read()
f.close()
wb = xlrd.open_workbook(file_contents=bytes)

I guess the only advantage there is it saves having to use a tempfile and an I/O trip to the disk. But your solution has the advantage that it just works, and also is compatible with openpyxl.

Thanks for taking the time!

[ludaavics]

sweet