Bug: Object of type int64 is not JSON serializable - introduced by Version 3.1.1

Hey @robin900 , we (@AndreCimander and me) might have found a bug which will lead to the following error.

The error is caused by this commits (https://github.com/robin900/gspread-dataframe/commit/00d2de44b81da78721879633368b9f37db42b5c5) changes in gspread_dataframe.py:53.

As the default pandas DataFrame will cast an integer to a numpy.int64, your changes will cause the JSON parsing to fail with the following set of information to reproduce the error.

Rolling back to version 3.1.0 fixes the issue, as numpy.int64 is properly casted to a str before insertion.

Code:

test_dataframe = pd.DataFrame({"column1": [6, 6, 7], "column2": [4, 4, 4]})
[...]
gspread_dataframe.set_with_dataframe(
    worksheet, dataframe, row=start_row, col=1, include_column_header=use_header
)

Error message: TypeError: Object of type int64 is not JSON serializable

Stack trace:

google_sheets_api/sheet.py:85: in append_pandas_dataframe_to_sheet
    gspread_dataframe.set_with_dataframe(
venv/lib/python3.8/site-packages/gspread_dataframe.py:299: in set_with_dataframe
    resp = worksheet.update_cells(cells_to_update, value_input_option='USER_ENTERED')
venv/lib/python3.8/site-packages/gspread/models.py:906: in update_cells
    data = self.spreadsheet.values_update(
venv/lib/python3.8/site-packages/gspread/models.py:235: in values_update
    r = self.client.request('put', url, params=params, json=body)
venv/lib/python3.8/site-packages/gspread/client.py:61: in request
    response = getattr(self.session, method)(
venv/lib/python3.8/site-packages/requests/sessions.py:590: in put
    return self.request('PUT', url, data=data, **kwargs)
venv/lib/python3.8/site-packages/google/auth/transport/requests.py:464: in request
    response = super(AuthorizedSession, self).request(
venv/lib/python3.8/site-packages/requests/sessions.py:516: in request
    prep = self.prepare_request(req)
venv/lib/python3.8/site-packages/requests/sessions.py:449: in prepare_request
    p.prepare(
venv/lib/python3.8/site-packages/requests/models.py:317: in prepare
    self.prepare_body(data, files, json)
venv/lib/python3.8/site-packages/requests/models.py:467: in prepare_body
    body = complexjson.dumps(json)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py:231: in dumps
    return _default_encoder.encode(obj)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py:199: in encode
    chunks = self.iterencode(o, _one_shot=True)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py:257: in iterencode
    return _iterencode(o, 0)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.encoder.JSONEncoder object at 0x7f9038667790>, o = 1

    def default(self, o):
        """Implement this method in a subclass such that it returns
        a serializable object for ``o``, or calls the base implementation
        (to raise a ``TypeError``).

        For example, to support arbitrary iterators, you could
        implement default like this::

            def default(self, o):
                try:
                    iterable = iter(o)
                except TypeError:
                    pass
                else:
                    return list(iterable)
                # Let the base class default method raise the TypeError
                return JSONEncoder.default(self, o)

        """
>       raise TypeError(f'Object of type {o.__class__.__name__} '
                        f'is not JSON serializable')
E       TypeError: Object of type int64 is not JSON serializable

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py:179: TypeError

If you need further information or assistance, please let me know.

[robin900]

@bastianwegge @AndreCimander I’m going to close this issue until we have some way to reproduce the problem. Let me know if you have that information and want the issue re-opened…

[robin900]

@AndreCimander Yes, I looked at that numpyencoder code, and we could adapt something like it to be a pre-encoding step. But I think that first we need to reach a consensus on the most reliable way to extract values from a DataFrame before sending them to Sheets API. The current reliance on DataFrame.values seemed like the right away forward, but now I’m puzzled by the behavior you’re seeing. I’ll test with your versions of pandas and numpy and see what I can find out…