MAINT: Numpy uses `PyUnicode_AS_DATA` which will be deprecated eventually

I originally raised this as https://github.com/pandas-dev/pandas/issues/21758 thinking it was a pandas issue, but when I delved more deeply I found that it was numpy.

Using np.array changes the return value of sys.getsizeof on the first single character string that is passed to it.

Reproducing code example:

Pasting this into a Python REPL:

import sys
sys.getsizeof('a')
sys.getsizeof('b')
import numpy as np
sys.getsizeof('a')
sys.getsizeof('b')
np.array(['b'], dtype=(np.str_, 1))
sys.getsizeof('a')
sys.getsizeof('b')
sys.getsizeof('c')

I get output like this:

$ python
Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getsizeof('a')
50
>>> sys.getsizeof('b')
50
>>> import numpy as np
>>> sys.getsizeof('a')
50
>>> sys.getsizeof('b')
50
>>> np.array(['b'], dtype=(np.str_, 1))
array(['b'], dtype='<U1')
>>> sys.getsizeof('a')
50
>>> sys.getsizeof('b')
58
>>> sys.getsizeof('c')
50
>>>
>>>

Notice how sys.getsizeof('b') has increased from 50 to 58 following the call to np.array. This is very much unexpected behaviour.

Numpy/Python version information:

Output from import sys, numpy; print(numpy.__version__, sys.version):

>>> import sys, numpy; print(numpy.__version__, sys.version)
1.15.0 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0]

I’ve seen this with a mixture of Python versions and numpy versions. I’m using Ubuntu 18.04LTS 64-bit, but I’ve also seen this on Ubuntu 16.04LTS 64-bit.

[seberg]

I think in most cases, we still use API which causes the unicode object to cache the result. (E.g. as utf8 string) but I am not convinced this is a bad thing in most cases, as opposed to just surprising. Also, I expect we have to use utf8 in most of those cases, since we probably need ascii compatibility.

So, I will close it. I am not quite sure, but it would seem more helpful to open a new issue that is more to the point, and personally, I am not sure there is an issue at all.

[eric-wieser]

The original behavior also no longer occurs:

In [2]: import sys
   ...: print(sys.getsizeof('a'))
   ...: print(sys.getsizeof('b'))
   ...: import numpy as np
   ...: print(sys.getsizeof('a'))
   ...: print(sys.getsizeof('b'))
   ...: np.array(['b'], dtype=(np.str_, 1))
   ...: print(sys.getsizeof('a'))
   ...: print(sys.getsizeof('b'))
   ...: print(sys.getsizeof('c'))
50
50
50
50
50
50
50