question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[0.24.1] New nullable integer fillna with non-int doesn't coerce to object

See original GitHub issue

Code Sample

import pandas as pd

sample_data = []

sample_data.append({"integer_column":None})
sample_data.append({"integer_column":1})
sample_data.append({"integer_column":2})

df = pd.DataFrame(sample_data)

# Previous type is object
# df.dtypes

df.loc[:,'integer_column'] = df.loc[:,'integer_column'].astype('Int64')

# Check new type is Int64, nullable
# df.dtypes

df.fillna('null_string')

Problem description

Using the new nullable type Int64, it is not possible to fill “NaN” values with other value.

Error raised

TypeError: <U11 cannot be converted to an IntegerDtype

Expected Output

The new dataframe should have replaced it’s NaN values with the desired input of .fillna() method.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.6.4.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 85 Stepping 4, GenuineIntel byteorder: little LC_ALL: None LANG: en LOCALE: None.None

pandas: 0.24.1 pytest: 3.3.2 pip: 9.0.1 setuptools: 38.4.0 Cython: 0.27.3 numpy: 1.14.0 scipy: 1.0.0 pyarrow: None xarray: None IPython: 6.2.1 sphinx: 1.6.6 patsy: 0.5.0 dateutil: 2.6.1 pytz: 2017.3 blosc: None bottleneck: 1.2.1 tables: 3.4.2 numexpr: 2.6.4 feather: None matplotlib: 2.1.2 openpyxl: 2.5.12 xlrd: 1.1.0 xlwt: 1.3.0 xlsxwriter: 1.0.2 lxml.etree: 4.1.1 bs4: 4.6.0 html5lib: 1.0.1 sqlalchemy: 1.1.18 pymysql: None psycopg2: None jinja2: 2.8.1 s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None gcsfs: None

Issue Analytics

  • State:open
  • Created 5 years ago
  • Reactions:3
  • Comments:22 (15 by maintainers)

github_iconTop GitHub Comments

5reactions
gosuto-inzasherucommented, Feb 27, 2020

I would assume that .fillna() would coerce the series into being of type object when I am trying to fill it with an object. Just like for example when adding a float to it coerces it into being of type float:

>>> pd.Series([1, 2, None], dtype='Int64') + 0.5
0    1.5
1    2.5
2    NaN
dtype: float64

However;

>>> pd.Series([1, 2, None], dtype='Int64').fillna('')
TypeError: <U1 cannot be converted to an IntegerDtype
2reactions
gosuto-inzasherucommented, Oct 12, 2021

i brought it up here: https://github.com/pandas-dev/pandas/issues/25288#issuecomment-592917095

i still think it makes more sense to be able to just use .fillna() without explicitly casting first. as i stated earlier in this thread, this is default behaviour for .fillna() on other dtypes, and it also is a logical step when for example adding a float to an Int64.

what you are suggesting from a user perspective, is that now sometimes i can .fillna() directly, and sometimes i will have to cast + fill. as a user i would feel more for consistent behaviour of .fillna().

Read more comments on GitHub >

github_iconTop Results From Across the Web

Convert Pandas column containing NaNs to dtype `int`
In version 0.24.+ pandas has gained the ability to hold integer dtypes with missing values. Nullable Integer Data Type.
Read more >
What's new in 1.5.0 (September 19, 2022) - Pandas
With Pyarrow installed, users can now create pandas objects that are backed by a pyarrow.ChunkedArray and pyarrow.DataType .
Read more >
Python | Pandas DataFrame.fillna() to replace Null values in ...
Just like pandas dropna() method manage and remove Null values from a data ... Input can be 0 or 1 for Integer and...
Read more >
Pandas DataFrame fillna() Method - W3Schools
Definition and Usage. The fillna() method replaces the NULL values with a specified value. The fillna() method returns a new DataFrame object unless...
Read more >
Handling Missing Data in Pandas: NaN Values Explained
The date column is not changed since the integer 1 is not a date. Copy. df=df.fillna(1).
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found