Discussion using datasets in offline mode

datasets.load_dataset("csv", ...) breaks if you have no connection (There is already this issue https://github.com/huggingface/datasets/issues/761 about it). It seems to be the same for metrics too.

I create this ticket to discuss a bit and gather what you have in mind or other propositions.

Here are some points to open discussion:

• if you want to prepare your code/datasets on your machine (having internet connexion) but run it on another offline machine (not having internet connexion), it won’t work as is, even if you have all files locally on this machine.
• AFAIK, you can make it work if you manually put the python files (csv.py for example) on this offline machine and change your code to datasets.load_dataset("MY_PATH/csv.py", ...). But it would be much better if you could run ths same code without modification if files are available locally.
• I’ve also been considering the requirement of downloading Python code and execute on your machine to use datasets. This can be an issue in a professional context. Downloading a CSV/H5 file is acceptable, downloading an executable script can open many security issues. We certainly need a mechanism to at least “freeze” the dataset code you retrieved once so that you can review it if you want and then be sure you use this one everywhere and not a version dowloaded from internet.

WDYT? (thks)

[ZizhenWang]

here is my way to load a dataset offline, but it requires an online machine

1. (online machine)

import datasets
data = datasets.load_dataset(...)
data.save_to_disk(/YOUR/DATASET/DIR)

1. copy the dir from online to the offline machine
2. (offline machine)

import datasets
data = datasets.load_from_disk(/SAVED/DATA/DIR)

HTH.

[geniki]

Requiring online connection is a deal breaker in some cases unfortunately so it’d be great if offline mode is added similar to how transformers loads models offline fine.

@mandubian’s second bullet point suggests that there’s a workaround allowing you to use your offline (custom?) dataset with datasets. Could you please elaborate on how that should look like?