Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Issue with TFRecord from URL

See original GitHub issue

I’m building a plugin for CML using TFRecord files. When I point the MultiURL method at https://storage.ecmwf.europeanweather.cloud/MAELSTROM_AP3/TFR/TripCloud0.0.tfrecord the file is successfully downloaded and loaded. When I point it at https://storage.ecmwf.europeanweather.cloud/MAELSTROM_AP3/TFR/TripCloud0.1.tfrecord the call to cml.load_source(…) fails with

Unknown file type /hugetmp/tmp-climetlab/url-63428528b56eb9c50418511287b38d1c7c53e6e203671861fb71faf86e6cbb72.man (b'PP\x00\x00\x00\x00\x00\x00'), ignoring

Examining the cache directory, the first file has been downloaded as a .tfrecord file and the second has been downloaded as a .man file. Thanks. Mat

Issue Analytics

State:
Created 2 years ago
Comments:7 (7 by maintainers)

Top GitHub Comments

1reaction

floriankrbcommented, Jul 22, 2021

Works out-of-the box with version 0.8.9. No need for the “extension” argument.

1reaction

floriankrbcommented, Jul 21, 2021

This issue definitely comes from the http server, but we can try some workarounds.

we could add an argument “extension” to load_source for urls. This would work as:

t = cml.load_source(
    "url",
    "https://storage.ecmwf.europeanweather.cloud/MAELSTROM_AP3/TFR/TripCloud0.1.tfrecord",
    extension="tfrecord",
)
t.to_tfdataset()

This would be experimental and may not work in future release (we will keep the functionaly if it is needed)