BUG: pandas.core.ops.dispatch_to_extension_op fails with UnboundLocalError

Code Sample, a copy-pastable example if possible

This bug requires an ExtensionArray that is not backed with a numpy array to occur.

import pandas as pd
import fletcher as fr

s = pd.Series(fr.FletcherArray(TEST_ARRAY))
assert (s.T == s).all()

Problem description

Looking at the code the value of new_right is only set when the the values of the ExtensionArray are a np.ndarray. With fletcher, this is not the case.

https://github.com/pandas-dev/pandas/blob/8bb2cc19ca3aa347d086b4fb5704ca9a3723172f/pandas/core/ops.py#L1142-L1156

Error message

>       assert (s.T == s).all()

tests/test_pandas_integration.py:139:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../pandas/pandas/core/ops.py:1433: in wrapper
    return dispatch_to_extension_op(op, self, other)
../pandas/pandas/core/ops.py:1163: in dispatch_to_extension_op
    res_values = op(new_left, new_right)
../pandas/pandas/core/ops.py:1433: in wrapper
    return dispatch_to_extension_op(op, self, other)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

op = <built-in function eq>, left = 0      Test
1    string
2      None
dtype: fletcher[string], right = <fletcher.base.FletcherArray object at 0x112ce7518>

    def dispatch_to_extension_op(op, left, right):
        """
        Assume that left or right is a Series backed by an ExtensionArray,
        apply the operator defined by op.
        """

        # The op calls will raise TypeError if the op is not defined
        # on the ExtensionArray
        # TODO(jreback)
        # we need to listify to avoid ndarray, or non-same-type extension array
        # dispatching

        if is_extension_array_dtype(left):

            new_left = left.values
            if isinstance(right, np.ndarray):

                # handle numpy scalars, this is a PITA
                # TODO(jreback)
                new_right = lib.item_from_zerodim(right)
                if is_scalar(new_right):
                    new_right = [new_right]
                new_right = list(new_right)
            elif is_extension_array_dtype(right) and type(left) != type(right):
>               new_right = list(new_right)
E               UnboundLocalError: local variable 'new_right' referenced before assignment

../pandas/pandas/core/ops.py:1154: UnboundLocalError

[xhochy]

How to best to incorporate support for non-ndarray is TBD

Yes, this is a thing I need to write an example ExtensionArray for in the Pandas tests. I’ve been reporting these bugs but it is always hard to make reproducible tests. I hope to find enough time in a single block soon to write such an example ExtensionArray. Then it will be simpler to fix them with a reproducing test.

[jorisvandenbossche]

@xhochy BTW, in the meantime, there is a non-ndarray backed example ExtensionArray (based on a pyarrow array) that is used for testing (since this was a problem for you to write a test).