Cannot allocate very large (global ~1km) ESMF grid

@sdeastham failed to use xESMF to regrid global ~1 km resolution data. The issue is that ESMPy cannot create very large grid object. A minimal example is:

import numpy as np
import ESMF

# not-so-large grid works
grid = ESMF.Grid(np.array([20000, 10000]), 
                 staggerloc=ESMF.StaggerLoc.CENTER, 
                 coord_sys=ESMF.CoordSys.SPH_DEG)

# larger grid breaks
grid = ESMF.Grid(np.array([20000, 15000]), 
                 staggerloc=ESMF.StaggerLoc.CENTER, 
                 coord_sys=ESMF.CoordSys.SPH_DEG)

The last command leads to TypeError: buffer is too small for requested array:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-e629ff74bc4d> in <module>()
      1 grid = ESMF.Grid(np.array([20000, 15000]), 
      2                  staggerloc=ESMF.StaggerLoc.CENTER,
----> 3                  coord_sys=ESMF.CoordSys.SPH_DEG)

~/Research/Computing/miniconda3/envs/sci/lib/python3.6/site-packages/ESMF/util/decorators.py in new_func(*args, **kwargs)
     62 
     63         esmp = esmpymanager.Manager(debug = False)
---> 64         return func(*args, **kwargs)
     65     return new_func
     66 

~/Research/Computing/miniconda3/envs/sci/lib/python3.6/site-packages/ESMF/api/grid.py in __init__(self, max_index, num_peri_dims, periodic_dim, pole_dim, coord_sys, coord_typekind, staggerloc, filename, filetype, reg_decomp, decompflag, is_sphere, add_corner_stagger, add_user_area, add_mask, varname, coord_names, tilesize, regDecompPTile, name)
    451         # Add coordinates if a staggerloc is specified
    452         if staggerloc is not None:
--> 453             self.add_coords(staggerloc=staggerloc, from_file=from_file)
    454 
    455         # Add items if they are specified, this is done after the

~/Research/Computing/miniconda3/envs/sci/lib/python3.6/site-packages/ESMF/api/grid.py in add_coords(self, staggerloc, coord_dim, from_file)
    797 
    798                 # and now for Python
--> 799                 self._allocate_coords_(stagger, from_file=from_file)
    800 
    801                 # set the staggerlocs to be done

~/Research/Computing/miniconda3/envs/sci/lib/python3.6/site-packages/ESMF/api/grid.py in _allocate_coords_(self, stagger, localde, from_file)
   1022         if (self.ndims == self.rank) or (self.ndims == 0):
   1023             for xyz in range(self.rank):
-> 1024                 self._link_coord_buffer_(xyz, stagger, localde)
   1025         # and this way if we have 1d coordinates
   1026         elif self.ndims < self.rank:

~/Research/Computing/miniconda3/envs/sci/lib/python3.6/site-packages/ESMF/api/grid.py in _link_coord_buffer_(self, coord_dim, stagger, localde)
   1072         lb, ub = ESMP_GridGetCoordBounds(self, staggerloc=stagger, localde=localde)
   1073 
-> 1074         gridCoordP = ndarray_from_esmf(data, self.type, ub-lb)
   1075 
   1076         # alias the coordinates to a grid property

~/Research/Computing/miniconda3/envs/sci/lib/python3.6/site-packages/ESMF/util/esmpyarray.py in ndarray_from_esmf(data, dtype, shape)
     37 
     38     esmfarray = np.ndarray(tuple(shape[:]), constants._ESMF2PythonType[dtype],
---> 39                            buffer, order="F")
     40 
     41     return esmfarray

TypeError: buffer is too small for requested array

@bekozi Any idea about this?

[JiaweiZhuang]

As for TypeError: buffer is too small for requested array, I am still not sure what’s happening.

From the source code addon/ESMPy/src/ESMF/util/esmpyarray.py, the function ndarray_from_esmf() is trying to create numpy array from ESMF Fortran array:

import ESMF.api.constants as constants
import numpy as np
import numpy.ma as ma
import ctypes as ct
import sys

def ndarray_from_esmf(data, dtype, shape):
    '''
    :param data: buffer of fortran allocated ESMF array
    :type data: ctypes void_p
    :param dtype: the type of the esmf buffer
    :type dtype: ESMF.TypeKind
    :param shape: N-D Python shape corresponding to 1D ESMF allocation
    :type shape: list or tuple
    :return: numpy array representing the data with dtype and shape
    '''
    # find the size of the local coordinates
    size = np.prod(shape[:]) * \
           np.dtype(constants._ESMF2PythonType[dtype]).itemsize

    # create a numpy array to point to the ESMF data allocation
    if sys.version_info[0] >= 3:
        buffer = ct.pythonapi.PyMemoryView_FromMemory
        buffer.restype = ct.py_object
        buffer = buffer(data, ct.c_int(size), 0x200)
    else:
        buffer = np.core.multiarray.int_asbuffer(
            ct.addressof(data.contents), size)


    esmfarray = np.ndarray(tuple(shape[:]), constants._ESMF2PythonType[dtype],
                           buffer, order="F")

    return esmfarray

The error happens in the call:

    esmfarray = np.ndarray(tuple(shape[:]), constants._ESMF2PythonType[dtype],
                           buffer, order="F")

The minimal code to get the same error is:

np.ndarray(shape=3, buffer=np.array([0.0, 1.0]), dtype=np.float64) 
# buffer size is smaller than the array size, leads to "buffer is too small for requested array"

It is probably because the earlier call buffer = buffer(data, ct.c_int(size), 0x200) doesn’t allocate enough memory. buffer is a PyMemoryView_FromMemory() function, which takes three arguments:

• data is the memory location allocated upstream and passed to ndarray_from_esmf(). It was created by a data = ESMP_GridGetCoordPtr(...) call from addon/ESMPy/src/ESMF/api/grid.py. Then it goes into deep Fortran code which I haven’t had time digging into.
• ct.c_int(size) is the requested memory in bytes. For the large grid shape (20000, 15000) used here, size == 20000*15000*8 == 2400000000
• 0x200 (512) is simply the constant PyBUF_WRITE.

Maybe @bekozi can elaborate more on this. Probably some internal memory constraints in ESMF?

[ManyAngledOne]

Just ran into this today when trying to regrid high resolution land use and vegetation density data.