API: is_string_dtype is not strict

https://github.com/pandas-dev/pandas/issues/15533#issuecomment-284270591

pandas.types.common.is_string_dtype is not strict, just checking if its a fixed-width numpy string/unicode type or object dtype. It could be made strict via something like this.

def is_string_dtype(arr_or_dtype):
    dtype = _get_dtype(arr_or_dtype)
    if isinstance(arr_or_dtype, np.ndarray):
        if not lib.infer_dtype(arr_or_dtype) in ['string', 'unicode']:
            return False

    return dtype.kind in ('O', 'S', 'U') and not is_period_dtype(dtype)

this would need performance checking to see if its a problem. Further this then changes the API a tiny bit (as we allow an object OR a dtype to be passed in). Which is probably ok as long as its documented a bit.

[ivirshup]

Any updates on this? It seems like behavior changed in v0.24.0, but not necessarily for the better.

import pandas as pd
from pandas.api.types import is_string_dtype

is_string_dtype(pd.Series(pd.Categorical([1,2,3])))
# False in v0.23.4
# True in v0.24.1

# Still:
is_string_dtype(pd.Series([(0,1), (1,1)]))
# True in v0.23.4
# True in v0.24.1

This was causing issues in a project where we were expecting the results of this check not to change between versions. On further investigation, it seems like this function is unreliable (tuples aren’t strings, categoricals as strings feels very R).

[jreback]

@ResidentMario you shouldn’t need this fyi