Avoiding metadata bloat caused by many long URLs

I woke up this morning wondering whether it would be possible to allow a variable to be defined in the referencefile spec. I’m worried about long s3 (or other) urls bloating the metadata.

If we could do something like:

prefix001='first/superlongurl/that/keeps/going/on/and/on/for/ever/to/some/dir'
prefix002='second/superlongurl/that/keeps/going/on/and/on/for/ever/to/some/dir'

"key1": {
    ["s3://$prefix001/data001.nc", 10000, 100]
  }
"key2": {
    ["s3://$prefix001/data001.nc", 10100, 100]
  }
"key3": {
    ["s3://$prefix002/data001.nc", 10000, 100]
  }
"key4": {
    ["s3://$prefix002/data001.nc", 10100, 100]
  }

we could make the bloat a lot smaller

[martindurant]

To be sure, I have a bias against arrow, in that the installation in python is enormous, and in many zarr/xarray or other array-based workloads would only be doing this one job.

Context: I work on the js implementation of zarr.

So exactly, zarr could be the target, I suspect it’s much smaller in code than arrow and/or parquet - but less well known, of course. You can also store arrow in zarr https://github.com/zarr-developers/zarr-python/issues/515

[martindurant]

I’m curious if Arrow would be of interest/suitable binary format.

What do you mean? Arrow is not a storage format.

In the context of the usage case of this repo, zarr would be a nice option maybe (it does have a JS implementation).