can't drop duplicated on dask dataframe index

please note my stackoverflow question

I am using dask dataframe with python 2.7 and want to drop duplicated index values from my df.

When using pandas i would use

df = df[~df.index.duplicated(keep = "first")]

And it works

When trying to do the same with dask dataframe i get

AttributeError: ‘Index’ object has no attribute ‘duplicated’

I could reset the index and than use the column that was the index to drop duplicated but I would like to avoid it if possible

I could use df.compute() and than drop the duplicated index values but this df is too big for memory.

I tried using the following code as suggested by jezrael in stackoverflow rxTable[~rxTable.index.to_Series().duplicated()] and got

AttributeError: ‘Index’ object has no attribute ‘to_Series’

It worked a few days ago and just stopped, i can’t find any difference in the code and data.

How can i drop the duplicated index values from my dataframe using dask dataframe?

Thanks

[jsignell]

@3ggaurav you can read about split_every and split_out here:

    split_every : int, optional
        Group partitions into groups of this size while performing a
        tree-reduction. If set to False, no tree-reduction will be used,
        and all intermediates will be concatenated and passed to ``aggregate``.
        Default is 8.
    split_out : int, optional
        Number of output partitions. Split occurs after first chunk reduction.

@Demirrr this is a very old issue. I think you’d be better off setting your index as a column and sing drop_duplicates:

import pandas as pd
import dask.dataframe as dd

a = pd.DataFrame({"A": [1, 2, 2, 3, 2, 2]},
                 index=[0, 0, 1, 1, 2, 2])
b = dd.from_pandas(a, 2)
out = b.reset_index().drop_duplicates(["index"]).set_index("index")
out.compute()

I am closing this. But if you are still having issues, please reopen.

[TomAugspurger]

Reminder, it’s helpful to have reproducible examples 😃

This may be your best shot right now:

import pandas as pd
import dask.dataframe as dd

a = pd.DataFrame({"A": [1, 2, 3, 4]},
                 index=[0, 0, 1, 1])
b = dd.from_pandas(a, 2)

a.groupby(a.index).first()

def chunk(x):
    return x.first()


def agg(x):
    return x.first()

A PR implementing first and last on groupby objects would be helpful if you have time.