New Feature Request: Add support for drop_duplicates()

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 19.04
• Modin installed from (source or binary): binary, pip install modin
• Modin version: 0.5.0
• Python version: 3.7.3
• Exact command to reproduce: Use drop_duplicates()

Describe the problem

drop_duplicates() is not supported today resulting in the following message when used:

UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
UserWarning: `DataFrame.duplicated` defaulting to pandas implementation.
To request implementation, send an email to [e-mail].
UserWarning: Distributing <class 'pandas.core.series.Series'> object. This may take some time.
UserWarning: `Series.__array__` defaulting to pandas implementation.

Source code / logs

[devin-petersohn]

We may want to put some of this logic into the PandasQueryCompiler so that it can be used by other implementations.

What about this:

in dataframe.py:

    def duplicated(self, subset=None, keep="first"):
        df = self[subset] if subset is not None else self
        # if the number of columns we are checking for duplicates is larger than 1, we must
        # hash them to generate a single value that can be compared across rows.
        if len(df.columns > 1):
            hashed = df.apply(lambda s: hash(s.to_numpy().data.tobytes()), axis=1)).to_frame()
        else:
            hashed = self
        return hashed.apply(lambda s: s.duplicated(keep=keep)).squeeze(axis=1)

in series.py:

    def duplicated(self, keep="first"):
        return self.to_frame().duplicated(keep=keep)

The base.py code in your answer assumed a dataframe input, so that code probably belongs in dataframe.py.

[devin-petersohn]

Feature added via #892