Incorrect boolean bitwise conversion when dtype=object

Code Sample, a copy-pastable example if possible

import pandas as pd
s = pd.Series([False, True, None])
s[s.isnull()] = False

print('incorrect:')
print((~s).astype(bool))

print('\ncorrect:')
print(s.apply(lambda x: not x))

Problem description

The above code will produce the following unexpected output:

incorrect:
0    True
1    True
2    True
dtype: bool

correct:
0     True
1    False
2     True
dtype: bool

The same issues persist with using np.nan or other NULL-like. NULL-like are commonly encountered in data. It is very convenient and common for data scientists to use bitwise operators like |, &, and ~ to slice and dice pandas Series and DataFrames. Bitwise operator ~ returns unexpected behavior when some data has been NULL in the past (implicitly causing the dtype to become object), even after coercion of the offending elements to bool. This can cause subtle problems in data filtering that will go undetected. If it is not fixed, use of bitwise operators on dtype=object should at least raise a Warning.

Expected Output

0    True
1    False
2    True
dtype: bool

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.5.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.7.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.3
numpy            : 1.17.4
pytz             : 2018.7
dateutil         : 2.7.5
pip              : 19.3.1
setuptools       : 41.6.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10
IPython          : 7.11.1
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.0.3
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.3
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

[TomAugspurger]

I would like to see a Warning when applying ~ to a dtype object

I don’t think that’s possible, since object dtype can include anything, including Python objects where ~ is perfectly sensible.

This would be fixed by https://github.com/pandas-dev/pandas/pull/23313, and implementing BooleanArray.__invert__ on the new boolean array.

[MarcoGorelli]

I would like to see a Warning when applying ~ to a dtype object

Seems reasonable to me, let’s see what the core team thinks (I’m relatively new to issue triage). Could we just add a warning to __invert__ from pandas.core.generic.py in the case that self is an object?