Add a How-To guide on storing and loading array data

Right now the only information on storing data in NumPy arrays on disk is the bare-bones function listing in https://numpy.org/devdocs/reference/routines.io.html. All it mentions is the functions available in NumPy itself.

It would be very useful to have a How-To guide (see NEP 44 for what a How-To guide is) on data storage covering:

• the options: NumPy built-in functionality (mainly the .npy/.npz format), Zarr, HDF5 & co, Bloscpack, pickling (anything else?)
• short summary of the storage model (e.g. Zarr chunked compressed n-D arrays, optionally in groups; HDF5 filesystem-like in one file)
• performance (I/O speed, size on disk)
• portability
• dependencies
• maturity
• recommendations for what to use when

My impression is that we should use .npy for the really simple cases, and direct people to Zarr for pretty much anything else (related: https://github.com/zarr-developers/community/issues/28).

Data source(s) to use:

• preferably use real-world data (we plan to produce a datasets package that we can rely on for the NumPy docs).
• should use a regular dtype (e.g. float64), but can also show a structured dtype and an object dtype (solutions may be different for those).

TBD: how much example code should it have, and if we do add example code with Zarr and perhaps also Pytables, should we run the code in CI so we’re sure that code remains working.

My tentative answer: yes let’s add example code, yes let’s run CI for any code we add, TBD can we do this sensibly with Sphinx or is this a good time to start with notebooks?

[melissawm]

Hello, all!

I have a draft for this How To at the numpy-tutorials repo. I’d appreciate if you could comment and give me any feedback there. The reason why I’m not adding this as a part of the main NumPy documentation is the dependencies and the time required to build the document; I think this is much more suitable to be built separately from the main documentation.

If you have any considerations, please comment here or over at the numpy-tutorials PR so we can move this forward. Thanks!

[Irio]

In this how-to guide, I’d wish to read about:

• common problems that I might run into and how to tackle e.g., pickle an object and later change the source code of the class that generated it.
• when to use NumPy, when to approach from other libraries such as Pandas e.g., there’s a .xls file with the numbers
• when to consider using a database for storage (not necessarily tell how to do it, but give a direction of what to google for finding details or link to a secondary how-to/tutorial)
• the possibility of losing information between saving & loading from a file

On formats, possibly: Apache Parquet, Protocol Buffers.

On storage model: it can be really helpful to use images here. One could create an image showing a simplified model of how NumPy maintains the data in memory; from that, how each format will store the information. It would be helpful for building a better mental model of what happens behind the scenes and understand the trade-offs.

Finally: depending on the title/keywords used, I bet people wanting to save this data in S3 or Cloud Storage may get to this how-to.