Supporting range queries

Not even sure if this makes sense. Still thinking through this myself. So maybe this is just a starting point for this discussion. Though this has been coming up in a few places.

Is there a way with fsspec to perform range queries? Or could there be?

Basically thinking about this from the Zarr side where we are increasingly interested in being able to select out portions of chunks. For this range queries would be useful for selecting out this portion.

cc @joshmoore @rabernat

Some related discussion in these issues:

• https://github.com/zarr-developers/zarr-python/issues/713
• https://github.com/zarr-developers/zarr-python/pull/667
• https://github.com/zarr-developers/zarr-python/issues/623
• https://github.com/zarr-developers/zarr-python/issues/595
• https://github.com/zarr-developers/zarr-python/issues/40

[rjzamora]

it sounds like you have done similar things with tabular data loading on GCP recently IIUC

Yes - cudf#9265 was recently merged as a temporary workaround for the fact that cudf cannot seek/read from an fsspec file-like object. Before that PR was merged, cudf would always read the entire remote file into a host memory buffer, even for partial IO. The “simple” workaround was to transfer only the necessary byte ranges into the local buffer (in parallel). Martin’s cat_ranges PR was not used in the cudf change, but it probably will be in the near future. The new cat_ranges API makes it easy to efficiently transfer a specific set of byte ranges with a single line of code. The only logic that the downstream library needs to worry about is the calculation of the specific byte ranges to pass to cat_ranges.

If you are working with a library that is able to read/seek from an fsspec file-like object, then the best approach is likely to gather known bytes ranges with cat_ranges, and then to open the remote file with the new ”parts” caching strategy. Note that I plan to add this optimization to Dask for read_parquet and byte_range-based read_csv.

[martindurant]

related, @jakirkham : zarr cannot currently read portions of a key, specifically for the case where the storage target is not compressed. I believe it can read selective blosc blocks (and zstd, in particular, would be very doable). Such functionality would be very helpful in a number of access patterns.

@rabernat : I don’t know how h5py achieves this either, exactly, but I assume it must be compiled against the python interpreter and really asks it to call the (dynamic) methods on the objects passed. A similar issue in rasterio: https://github.com/mapbox/rasterio/pull/2141