question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Replacing multiple columns (or just one) with iloc does not work

See original GitHub issue

Code Sample, a copy-pastable example if possible

import pandas

columns = pandas.DataFrame({'a2': [11, 12, 13], 'b2': [14, 15, 16]})
inputs = pandas.DataFrame({'a1': [1, 2, 3], 'b1': [4, 5, 6], 'c1': [7, 8, 9]})

inputs.iloc[:, [1]] = columns.iloc[:, [0]]

print(inputs)

Problem description

I have a code which is replacing a set of columns with another set of columns, based on column indices. To make things done without a special case, I assumes I could just use iloc to both select and set columns in a DataFrame. But it seems that this not work and fails in strange ways.

Expected Output

   a1  b1  c1
0   1  11   7
1   2  12   8
2   3  13   9

But in reality, you get:

    a1  b1   c1
0  1.0 NaN  7.0
1  2.0 NaN  8.0
2  3.0 NaN  9.0

See how values converted to float and how column is NaNs?

But, if I do the following I get expected results:

inputs.iloc[:, [1]] = [[11], [12], [13]]

This also works:

inputs.iloc[:, [1]] = columns.iloc[:, [0]].values

So if it works with lists and ndarrays, one would assume it would also work with DataFrames themselves. But it does not.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None python: 3.6.3.final.0 python-bits: 64 OS: Linux OS-release: 4.13.0-46-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.23.3 pytest: None pip: 18.0 setuptools: 40.0.0 Cython: None numpy: 1.15.0 scipy: None pyarrow: None xarray: None IPython: None sphinx: None patsy: None dateutil: 2.7.3 pytz: 2018.5 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: None openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:11 (11 by maintainers)

github_iconTop GitHub Comments

1reaction
jorisvandenbosschecommented, Jul 25, 2018

I can understand that this happens with .loc, but how can it happen with .iloc? .iloc should be position-based so it should not care about column names, no?

Yes, in principle iloc should not care about column names, so this certainly seems a bug to me.

0reactions
mitarcommented, Nov 19, 2020

Thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Change column values using iloc doesn't works - Stack Overflow
The correct way of using it to use df.columns.get_loc('istrain') get the column location as integer and so use it directly inside iloc and ......
Read more >
Pandas iloc[] Usage with Examples
pandas.DataFrame.iloc[] is a property that is used to select rows and columns by position/index. If the position/index does not exist, it gives an...
Read more >
Indexing and selecting data - Pandas
pandas aligns all AXES when setting Series and DataFrame from .loc , and .iloc . This will not modify df because the column...
Read more >
How to Select Rows and Columns in Pandas Using [ ], .loc ...
How to Select Rows and Columns in Pandas Using [ ], .loc, iloc, ... To select multiple columns, you can pass a list...
Read more >
Views and Copies in pandas - Practical Data Science
An Aside: No, the problem doesn't only emerge when you change the data type of a column¶ ; 0 · 2 · 4...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found