Handle weight files more elegantly

(continuing #2 and #9)

In the just-released v0.1.1, the default behavior is to write weights to disk, and there is a regridder.clean_weight_file() method to optionally remove it (demo).

This exactly matches how ESMPy works (directly dump weights to disk, not returning a numpy array) but feels a bit odd. A more intuitive workflow is only getting in-memory weights by default, and have a regridder.save() method to optionally save it to disk (like the model.save() method in Keras).

With current ESMPy (up to 7.1.0dev41) I need to do very weird things to implement this new design:

1. Use ESMPy to write weights to disk
2. Read it back to memory
3. Then automatically delete the weight file by default (what a waste if you actually need it)
4. Finally, have a method to optionally re-write the weight file to disk. (duplicates step1)

This incurs a lot of unnecessary disk I/O. If ESMPy has the plan to return in-memory weights I will definitely use the new design. Otherwise I’d rather keep the current approach.

@bekozi Any plans on that?

[bekozi]

@JiaweiZhuang, the in-memory weights branch is ready for testing. A Dockerfile for building ESMF from that branch is available here.

For reference, here are links to the docstrings for the new interfaces. ESMF doesn’t build docs for branches…

• get_factors
• get_weights_dict
• test_field_regrid_factor_retrieval

In our testing, the factor arrays contain zero elements with no spatial overlap (disjoint geometries) and unmapped action is ignore. Again, sorry for the delay on this. Hopefully we won’t encounter too many issues during your implementation.

[bekozi]

This slows down the sparse MM by ~40%. See this demo. It can be fixed by copying and rearranging the memory layout, anyway.

This is a downer. I think what we’ll need is a follow on issue where we adapt the regrid store interface in ESMF to return vectors for the index lists directly.

Looks good me if you don’t mind adding & maintaining one more function.

Don’t mind.

Even though the old ESMF.Regrid object is not referenced by any variable identifiers, the memory will not be freed (in contrast to the automatic garbage collection for normal numpy arrays).

It’s hard to guarantee how the Python garbage collection will run. As usual, SO provides a good overview. This is why we introduced the destroy method. The memory will be cleaned up eventually, it may just not happen when you really need it. destroy will be called via __del__ / the destructor on the ESMF.Regrid object when gc runs.

In summary

Thanks for this.

My mistake – field.get_area() is to compute the area and add to field.data, not directly returning the area array.

You are correct, and you are forgiven. 😄 Not like I caught it anyway.

I’m really hoping to ship these features soon.