Trouble with streaming frgfm/imagenette vision dataset with TAR archive

Link

https://huggingface.co/datasets/frgfm/imagenette

Description

Hello there 👋

Thanks for the amazing work you’ve done with HF Datasets! I’ve just started playing with it, and managed to upload my first dataset. But for the second one, I’m having trouble with the preview since there is some archive extraction involved 😅

Basically, I get a:

Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.

I’ve tried several things and checked this issue https://github.com/huggingface/datasets/issues/4181 as well, but no luck so far!

Could you point me in the right direction please? 🙏

Owner

Yes

[frgfm]

Hi @albertvillanova 👋

Thanks, since my last message, I went through the repo of https://huggingface.co/datasets/food101/blob/main/food101.py and managed to get it to work in the end 🙏

Here it is: https://huggingface.co/datasets/frgfm/imagenette

I appreciate you opening an issue to document the process, it might help a few!

[albertvillanova]

Hi @frgfm, streaming a dataset that contains a TAR file requires some tweaks because (contrary to ZIP files), tha TAR archive does not allow random access to any of the contained member files. Instead they have to be accessed sequentially (in the order in which they were put into the TAR file when created) and yielded.

So when iterating over the TAR file content, when an image file is found, we need to yield it (and not keeping it in memory, which will require huge RAM memory for large datasets). But when yielding an image file, we also need to yield with it what we call “metadata”: the class label, and other textual information (for example, for audio files, sometimes we also add info such as the speaker ID, their sex, their age,…).

All this information usually is stored in what we call the metadata file: either a JSON or a CSV/TSV file.

But if this is also inside the TAR archive, we need to find this file in the first place when iterating the TAR archive, so that we already have this information when we find an image file and we can yield the image file and its metadata info.

Therefore:

• either the TAR archive contains the metadata file as the first member when iterating it (something we cannot change as it is done at the creation of the TAR file)
• or if not, then we need to have the metadata file elsewhere
  • in these cases, what we do (if the dataset license allows it) is:
    • we download the TAR file locally, we extract the metadata file and we host the metadata on the Hub
    • we modify the dataset loading script so that it first downloads the metadata file (and reads it) and only then starts iterating the content of the TAR archive file

See an example of this process we recently did for “google/fleurs” (their metadata files for “train” were at the end of the TAR archives, after all audio files): https://huggingface.co/datasets/google/fleurs/discussions/4

• we uploaded the metadata file to the Hub
• we adapted the loading script to use it