Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

to_dict doesn't convert np.int64 to python integers

See original GitHub issue

Code Sample, a copy-pastable example if possible

# Your code here
data = [{'id': 11, 'text': 'Osv1wbZoL'},
        {'id': 0, 'text': 'KQpPReW3S9nZOS3'},
        {'id': 0, 'text': 'cbqLhjrb0B2Ah6E'},
        {'id': 3, 'text': 'qu1Jlnyba'},
        {'id': 14, 'text': 'aJUv5DBjbcGc3'},
        {'id': 12, 'text': 'Yobf9'},
        {'id': 4, 'text': 'awzZCV'},
        {'id': 4, 'text': '3NvBAVL'},
        {'id': 11, 'text': '80sPCxIf9s5wmEZ1'},
        {'id': 5, 'text': 'afrPD0X6mIzFK'}]

df = pd.DataFrame(data)
# out:
# id       int64
# text    object
# dtype: object

type(df[['id', 'text']].to_dict(orient='records')[0]['id'])
# out: int

type(df[['id']].to_dict(orient='records')[0]['id'])
# out: numpy.int64

Problem description

depending on the count of output columns, numpy integers getting converted to python integers

afterwards both json.dumps and ujson.dumps fails to encode

Expected Output

int for both cases

Output of `pd.show_versions()`

INSTALLED VERSIONS ------------------ commit: None python: 3.6.1.final.0 python-bits: 64 OS: Darwin OS-release: 16.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2 nose: 1.3.7 pip: 9.0.1 setuptools: 33.1.1.post20170320 Cython: 0.25.2 numpy: 1.12.1 scipy: 0.19.0 statsmodels: 0.8.0 xarray: None IPython: 5.3.0 sphinx: None patsy: 0.4.1 dateutil: 2.6.0 pytz: 2017.2 blosc: None bottleneck: None tables: 3.3.0 numexpr: 2.6.2 matplotlib: 2.0.0 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: 0.999 httplib2: None apiclient: None sqlalchemy: 1.1.9 pymysql: 0.7.10.None psycopg2: 2.6.2 (dt dec pq3 ext lo64) jinja2: 2.9.5 boto: None pandas_datareader: None

Issue Analytics

State:
Created 6 years ago
Comments:8 (3 by maintainers)

Top GitHub Comments

3reactions

kumudrajcommented, Jun 7, 2020

use: from pandas.io.json import dumps import json data_dictionary = df.iloc[0].to_dict() data_dictionary=json.loads(dumps(data_dictionary,double_precision=0))

1reaction

TomAugspurgercommented, Apr 18, 2017

afterwards both json.dumps and ujson.dumps fails to encode

If you’re only dumping to JSON, then df.to_json might work for you. It knows how to serialize numpy types.

Top Results From Across the Web

python: converting an numpy array data type from int64 to int

If it's a pandas serise, you can first convert it to Dataframe, then use df.to_dict(), then the numpy.int64 will convert to int

Convert numpy dtypes to native python types - w3resource

Write a NumPy program to convert numpy dtypes to native python types. Sample Solution : Python Code : import numpy as np print("numpy.float32...

pandas.DataFrame.convert_dtypes

Convert columns to best possible dtypes using dtypes supporting pd.NA . ... Whether, if possible, conversion can be done to integer extension types....

convert string to int in python - You.com | The search engine you ...

The int() function takes in any python data type and converts it into a integer. ... ID, errors='coerce').fillna(0).astype(np.int64) print (df) ID 0 ...

How to convert to/from NumPy - Awkward Array

As a generalization of NumPy, any NumPy array can be converted to an Awkward Array ... var * int64'> allow_missing = True )...