local file operations for subsetting and file conversion

adding another layer of in-memory bytesIO objects for performing local operations on subsetted files from icepyx

• subsetting to valid data points using provided or calculated quality flags
• converting to different file formats not available from NSIDC (such as zarr)

basic addition to https://github.com/icesat2py/icepyx/blob/master/icepyx/core/granules.py#L390 will be like this (with the file operations coming after):

for zfile in z.filelist:
    # Remove the subfolder name from the filepath
    zfile.filename = os.path.basename(zfile.filename)
    fileID = io.BytesIO(z.read(zfile))
    fileID.seek(0)
    # open in-memory HDF5 file and perform operations
    with h5py.File(fileID,'r') as source:

[fspaolo]

Another use case we might want to think about is “what happens if the user finds out later it needs another variable”? Will the user need to download the data again (since we have not persisted the original files)?

[JessicaS11]

This is a great discussion and a critical one for where icepyx goes next. It will be important to have a way for people to use/interact with data locally that is not dependent on them having just downloaded it, which raises a few questions about where/when some of these subsetting and conversion operations should happen and what files are ultimately stored for the user. The modus operandi I’ve been using can be summarized as “make most of these decisions automatically for the user based on best practices and recommendations from the science team, assuming users just want some basic data without having to make many decisions, but implement those defaults in a way (i.e. with flags and keywords) that make it easy for the heavy-data user to choose something different”. For instance, this is the idea behind the default automatic use of the NSIDC subsetter for spatial and temporal subsetting - most people don’t need full granules if they’ve already created a region of interest, so we only give them data where they’ve asked for it, but if they really want full granules, it’s easy to get them.