BUG: Cannot replace NaNs with None

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Previously, we were able to replace NaN values with None for easier insertion into databases like mysql, as listed here: https://stackoverflow.com/questions/14162723/replacing-pandas-or-numpy-nan-with-a-none-to-use-with-mysqldb

python

df = pd.DataFrame([[1, None, np.NaN], [2, np.NaN, np.NaN]])
df = df.where(pd.notnull(df), None)
df

Output of df:

0	1	2
1	NaN	NaN
2	NaN	NaN

Problem description

Previously, all np.NaN were replaced with None. Now, they still remain as np.NaN, and breaks database insertions

Expected Output

0	1	2
1	None	None
2	None	None

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : c7f7443c1bad8262358114d5e88cd9c8a308e8aa python : 3.8.8.final.0 python-bits : 64 OS : Linux OS-release : 5.4.0-1049-aws Version : #51~18.04.1-Ubuntu SMP Fri May 14 18:38:46 UTC 2021 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.3.1 numpy : 1.19.2 pytz : 2020.1 dateutil : 2.8.1 pip : 20.2.4 setuptools : 50.3.1 Cython : 0.29.21 pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : 2.8.5 (dt dec pq3 ext lo64) jinja2 : None IPython : 7.19.0 pandas_datareader: None bs4 : None bottleneck : None fsspec : None fastparquet : None gcsfs : None matplotlib : 3.2.2 numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 1.0.1 pyxlsb : None s3fs : None scipy : 1.5.2 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None numba : None

[dylancaponi]

Confirmed we see this same issue on 1.3.0 and 1.3.1 but not 1.2.5.

[mzeitlin11]

What’s the best way around this for now? I tried to revert to 1.2.5 but I got an import error

The other issue has a good workaround: https://github.com/pandas-dev/pandas/issues/42423#issuecomment-876890034 Not sure about your 1.2.5 installation issue, I’d just try installing a fresh one (or fresh env if using something like conda)