Repartition randomly dropping some rows

Bug report

When I called set_index(..., sorted=True), I got a warning saying partition indices have overlap, which is fine. From here, after calling repartition, I lost a row. This must be a bug in the repartition code since I don’t see why repartition would make me lose rows.

>>> import pandas as pd
>>> import dask.dataframe as dd
>>> A = pd.DataFrame({'key': [1,2,3,4,4,5,6,7], 'value': list('abcd'*2)})
>>> a = dd.from_pandas(A, npartitions=2)
>>> a = a.set_index('key', sorted=True)
/Users/ctj/Documents/dask/dask/dataframe/shuffle.py:600: UserWarning: Partition indices have overlap.
  warnings.warn("Partition indices have overlap.")
>>> a.compute()
    value
key      
1       a
2       b
3       c
4       d
4       a
5       b
6       c
7       d
>>> a = a.repartition(divisions=a.divisions)
>>> a.compute()
    value
key      
1       a
2       b
3       c
4       a
5       b
6       c
7       d

[mrocklin]

Is ddf.divisions necessarily a strictly increasing sequence

Yes

how would Dask handle when a ton of the rows have the same index?

Poorly

On Mon, Jul 1, 2019 at 3:27 PM Cody Johnson [e-mail] wrote:

In that case, how would Dask handle when a ton of the rows have the same index? Is ddf.divisions necessarily a strictly increasing sequence? (Except the last one)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dask/dask/issues/4860?email_source=notifications&email_token=AACKZTEZNKN2WU4RYMWGCMLP5IH4NA5CNFSM4HRBGJV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY6JUCA#issuecomment-507288072, or mute the thread https://github.com/notifications/unsubscribe-auth/AACKZTB7STM5I4OAPNYRW6LP5IH4NANCNFSM4HRBGJVQ .

[codercody]

On second thought, it does cause the strictness of the upper bound to be violated. If you deem that error-worthy then sure.