High memory usage

The memory usage of xESMF seems quite high, higher than standalone ESMF CLI, and I wonder if this is due to a possible leak, as loading from a saved weights file uses much less RAM? Additionally, perhaps it can be improved by using lower precision (32bit vs 64bit). I also note it doesn’t seem possible to destroy a Regridder object to free memory?

import xesmf
import numpy as np
import gc
@profile
def test():
    src_ds = {'lat': np.arange(29.5,70.5,0.05), 'lon': np.arange(-23.5,45.0,0.05)} 
    dst_ds = xesmf.util.grid_2d(29, 70, 0.03, -23, 45, 0.03)
    regridder = xesmf.Regridder(src_ds, dst_ds, 'bilinear')
    regridder = None
    gc.collect()
    print("collected")
test()

python3 -m memory_profiler memtest.py
Line #    Mem usage    Increment   Line Contents
================================================
     4   65.500 MiB   65.500 MiB   @profile
     5                             def test():
     6   65.500 MiB    0.000 MiB       src_ds = {'lat': np.arange(29.5,70.5,0.05), 'lon': np.arange(-23.5,45.0,0.05)} 
     7  160.461 MiB   94.961 MiB       dst_ds = xesmf.util.grid_2d(29, 70, 0.03, -23, 45, 0.03)
     8 1246.199 MiB 1085.738 MiB       regridder = xesmf.Regridder(src_ds, dst_ds, 'bilinear')
     9 1246.203 MiB    0.004 MiB       regridder = None
    10 1246.203 MiB    0.000 MiB       gc.collect()
    11 1246.203 MiB    0.000 MiB       print("collected")

[mohoch1]

Hi.

Is there any news regarding this issue? We are experiencing similar problems.

Our application needs to perform regriding many times, and we have tracked that each usage of the Regridder causes a massive increase in memory usage, which is not released.

Has this issue come to any resolution?

Thanks

[rokuingh]

I just became aware of this issue, and thought I would chime in from the ESMPy perspective (ESMPy is the engine behind xESMF). The ESMF 8.1.0 release, expected at the end of March '21, will include a fix for a memory leak in the search algorithm of the regridding code. This may resolve the memory issues discussed in this thread. There should be a new conda package version of ESMPy 8.1.0 by the first of April.