question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BUG: Enlarging DataFrame with 2d indexer converts ints to floats

See original GitHub issue

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

ct_arr = np.array([[70, 150], [66, 81]])
counts = pd.DataFrame(
    data=ct_arr,
    columns=["Outstanding", "Not Outstanding"],
    index=["Bank", "Credit Union"],
)
ctot = counts.copy()
# everything above this line is integers

ctot.loc["Total", :] = ctot.sum(axis=0)
ctot.loc[:, "Total"] = ctot.sum(axis=1)
print(ctot)

Issue Description

I generate a dataframe from an np.array() of integers. I want to calculate all the totals: by row, by column, and the grand total.

The code works, but the output is the float type.

Expected Behavior

Since all inputs are integers and I only add the numbers, I expected the output dataframe to also be only integers.

Installed Versions

INSTALLED VERSIONS

commit : e8093ba372f9adfe79439d90fe74b0b5b6dea9d6 python : 3.9.9.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : English_United States.1252

pandas : 1.4.3 numpy : 1.23.0 pytz : 2021.3 dateutil : 2.8.2 setuptools : 62.6.0 pip : 22.1.2 Cython : 0.29.30 pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.7.1 html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.4.0 pandas_datareader: None bs4 : 4.11.1 bottleneck : None brotli : None fastparquet : 0.8.1 fsspec : 2021.11.1 gcsfs : None markupsafe : 2.1.1 matplotlib : 3.5.2 numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 8.0.0 pyreadstat : None pyxlsb : None s3fs : None scipy : 1.8.1 snappy : None sqlalchemy : None tables : None tabulate : 0.8.10 xarray : 0.20.2 xlrd : 2.0.1 xlwt : None zstandard : None

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:8 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
simonjayhawkinscommented, Jul 5, 2022

@jreback - why should expanding an integer column with an integer value coerce to float?

In case the examples are not clear, does not coerce to float with 1d indexer

df = pd.DataFrame([70, 150], dtype="i8")
df.loc["Total"] = df.sum(axis=0)
print(df.dtypes)
# 0    int64
# dtype: object

does coerce to float with 2d indexer

df = pd.DataFrame([70, 150], dtype="i8")
df.loc["Total", :] = df.sum(axis=0)
print(df.dtypes)
# 0    float64
# dtype: object

and only happens with axis=0, i.e. adding rows.

looks like a bug to me. These should both produce the same output?

1reaction
simonjayhawkinscommented, Jun 25, 2022

Thanks @FlorinAndrei for the report.

Note that the row expansion with a 2d indexer for .loc is causing the cast (the first total) and in the code sample this results in the floats in the row totals (second total and added column) and the DataFrame copy is also not relevant.

# single column DataFrame

df = pd.DataFrame([70, 150], dtype="i8")
df.loc["Total"] = df.sum(axis=0)
print(df.dtypes)
# 0    int64
# dtype: object

df = pd.DataFrame([70, 150], dtype="i8")
df.loc["Total", :] = df.sum(axis=0)
print(df.dtypes)
# 0    float64
# dtype: object

df = pd.DataFrame([70, 150], dtype="i8")
df.loc[:, "Total"] = df.sum(axis=1)
print(df.dtypes)
# 0        int64
# Total    int64
# dtype: object


# # multi-column DataFrame

df = pd.DataFrame([[70, 150], [66, 81]], dtype="i8")
df.loc["Total"] = df.sum(axis=0)
print(df.dtypes)
# 0    int64
# 1    int64
# dtype: object

df = pd.DataFrame([[70, 150], [66, 81]], dtype="i8")
df.loc["Total", :] = df.sum(axis=0)
print(df.dtypes)
# 0    float64
# 1    float64
# dtype: object

df = pd.DataFrame([[70, 150], [66, 81]], dtype="i8")
df.loc[:, "Total"] = df.sum(axis=1)
print(df.dtypes)
# 0        int64
# 1        int64
# Total    int64
# dtype: object

The result dtype from df.sum() is correct, so the title of this issue is misleading and the MRE could be reduced to

df = pd.DataFrame([70, 150], dtype="i8")
df.loc["Total", :] = pd.Series([220], dtype="i8")
df.dtypes
# 0    float64
# dtype: object

so it appears to be an indexing issue i.e. setting values with .loc with row expansion. (might be worth looking for similar issues)

contributions and PRs to fix welcome

Read more comments on GitHub >

github_iconTop Results From Across the Web

Stop Pandas from converting int to float due to an insertion in ...
I understand that if I insert NaN into the int column, Pandas will convert all the int into float because there is no...
Read more >
What's new in 1.4.0 (January 22, 2022) - Pandas
Added rank function to Rolling and Expanding . The new function supports the method , ascending , and pct flags of DataFrame.rank() ....
Read more >
Indexing and Selecting Data — pandas 0.13.1 documentation
Selection By Position. Pandas provides a suite of methods in order to get purely integer based indexing. The semantics follow closely python and...
Read more >
Indexing and Selecting Data — pandas 0.14.0 documentation
.iloc is strictly integer position based (from 0 to length-1 of the axis), will raise IndexError if an indexer is requested and it...
Read more >
Indexing and selecting data — pandas 0.25.3 documentation
Indexing on an integer-based Index with floats has been clarified in 0.18.0, for a ... -0.861849 2000-01-05 -0.424972 2000-01-07 0.404705 Freq: 2D, Name:...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found