Series.str.decode() turns arrays of strings to NaN and fails on byte strings

Hello,

this looks like a bug:

x = np.array(['x','y'])
pd.Series(x).str.decode(encoding='UTF-8',errors='strict')
0   NaN
1   NaN
dtype: float64

the line above is also used in pytables.py:

data = Series(data).str.decode(encoding, errors=errors).values

… and leads to an error when reading hdf-files written with pandas <0.23 In some cases the text data was stored as byte string (i.e. ‘x = np.array([b’x’, b’y’]) ) which raises the following:

AttributeError: Can only use .str accessor with string values, which use np.object_ dtype in pandas

[Jywscher]

Hello, is there any update on the issue? Since i ran into the same problem it would be great if someone could help to solve it…

[abrakababra]

finally done a little research, it’s taking the “except-route” here:

def _map(f, arr, na_mask=False, na_value=np.nan, dtype=object):
    if not len(arr):
        return np.ndarray(0, dtype=dtype)

    if isinstance(arr, ABCSeries):
        arr = arr.values
    if not isinstance(arr, np.ndarray):
        arr = np.asarray(arr, dtype=object)
    if na_mask:
        mask = isna(arr)
        try:
            convert = not all(mask)
            result = lib.map_infer_mask(arr, f, mask.view(np.uint8), convert)
--->  except (TypeError, AttributeError) as e:**
            # Reraise the exception if callable `f` got wrong number of args.
            # The user may want to be warned by this, instead of getting NaN
            if compat.PY2:
                p_err = r'takes (no|(exactly|at (least|most)) ?\d+) arguments?'
            else:
                p_err = (r'((takes)|(missing)) (?(2)from \d+ to )?\d+ '
                         r'(?(3)required )positional arguments?')

            if len(e.args) >= 1 and re.search(p_err, e.args[0]):
                raise e

            def g(x):
                try:
                    return f(x)
------->   except (TypeError, AttributeError):
                    return na_value

            return _map(g, arr, dtype=dtype)
        if na_value is not np.nan:
            np.putmask(result, mask, na_value)
            if result.dtype == object:
                result = lib.maybe_convert_objects(result)
        return result
    else:
return lib.map_infer(arr, f)

the first except is triggered by lib.map_infer_maskwhich in my installation is this file \python3\lib\site-packages\pandas_libs\lib.cp37-win_amd64.pyd