Incremental writes with scipy.io.netcdf_file

scipy.io.netcdf_file is great, but although it supports memory mapping for incremental reads of data from disk, it doesn’t support any form of incremental writes. The flush() method writes an entire file from scratch.

This would be nice to have for writing datasets larger than fit into memory (e.g., with the 64-bit offset variant).

Possible levels of support, ordered by roughly ascending difficulty / decreasing importance:

1. Modifying data in existing variables without new memory allocation (perhaps setting mmap.ACCESS_WRITE would suffice?)
2. Appending new entries for record variables (only adds data to the end of the file)
3. Modifying metadata of existing variables. In general this requires rewriting the file to make room for new and/or enlarged metadata entries, but we could do this copy in a streaming fashion.

[shoyer]

To be clear, even though I think most users would be better suited by using netcdf4-python, scipy’s pure Python netcdf3 reader is quite valuable. Xarray users (including myself) use it all the time because it often has better performance and is easier to install.

There are a few longstanding issues, but this does not detract from it’s usefulness. The module is basically feature complete, even if it could be improved.

I think it would be fine to mark netcdf_file as “legacy” code (per Ralf’s recent suggestion for categorization on the developer list), but I would strongly object to removing it from SciPy.

[orbeckst]

I think https://github.com/scipy/scipy/issues/9157#issuecomment-1144305199 summarized my views.

If getting good write performance (maybe for well-defined subset of ncdf3 files) is relatively easy then that would be convenient to have in a light-weight implementation. However, given scarce resources I very much understand that netcdf I/O is not considered a core competency of scipy — and that’s ok, given the existence of other libraries.