question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

.loc on DataFrame returning coerced dtype for single rows

See original GitHub issue

xref #14205

The .loc method of DataFrame with different dtypes yields coerced type even if the resulting slice does only contain elements from one type. This happens only when selecting a single row. I can guess that this might be intended because the implementation of loc seems to first lookup the row as a single Series, doing the coercion and then applying the second (column) indexer.

However, when the column indexer narrows down the selection such that the upcasting would not have been necessary in the first place, it can be very surprising and may even cause bugs (on user-side) if it goes unnoticed. (Like, “I was sure that those column was int64”).

>>> import pandas as pd

>>> d = pd.DataFrame(dict(a=[1.23]))
>>> d["b"] = 666  # adding column with int

>>> d.info()  # info as expected (column b is int64 - fine)
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1 entries, 0 to 0
Data columns (total 2 columns):
a    1 non-null float64
b    1 non-null int64
dtypes: float64(1), int64(1)
memory usage: 24.0 bytes

>>> d.loc[0,"b"]  # UNEXPECTED: returning a single float
666.0

>>> d.ix[0, "b"]  # OK: returns a single int
666

>>> d.loc[[0], "b"]  # OK
0    666
Name: b, dtype: int64

Feel free to close if the behavior s intended. Maybe this this a “bug” or an suggested API change. I dunno.

Perhaps related to #10503, #9519, #9269, #11594 ?

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: Darwin
OS-release: 15.0.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

pandas: 0.17.0
[...]

Issue Analytics

  • State:closed
  • Created 8 years ago
  • Reactions:2
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

2reactions
jrebackcommented, May 4, 2016

I suppose a:

df.loc(coerce=False)[0] might be ok (with a default of True) for back compat.

2reactions
mao-liucommented, May 4, 2016

I think I will submit a separate issue. I currently require a way of retrieving a row of a DataFrame that preserves numerical dtypes, which is separate to this issue, but very related.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Preserving dtypes when extracting a row from a pandas ...
Is there a good way to extract a row without incurring this type conversion? I could imagine e.g. returning a numpy structured array,...
Read more >
pandas.DataFrame.loc — pandas 1.5.2 documentation
Returns a cross-section (row(s) or column(s)) from the Series/DataFrame. Series.loc ... df.loc['viper'] max_speed 4 shield 5 Name: viper, dtype: int64.
Read more >
The pandas DataFrame Object - Cheat Sheet
Iterating over DataFrame rows for (index, row) in df.iterrows(): # pass. Trap: row data type may be coerced. Sorting DataFrame rows values df...
Read more >
The Pandas DataFrame: Make Working With Data Delightful
iat[] accepts the zero-based indices of rows and columns and returns a single data value. Of these, .loc[] and .iloc[] are particularly powerful....
Read more >
How to Convert String to Integer in Pandas DataFrame?
dtype : Data type to convert the series into. (for example str, float, int). ... One of the most effective approaches is Pandas...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found