[api request] API to obtain "dataset_module" dynamic path?

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is.

This is an awesome library.

It seems like the dynamic module path in this library has broken some of hyperparameter tuning functionality: https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/34

This is because Ray will spawn new processes, and each process will load modules by path. However, we need to explicitly inform Ray to load the right modules, or else it will error upon import.

I’d like an API to obtain the dynamic paths. This will allow us to support this functionality in this awesome library while being future proof.

Describe the solution you’d like A clear and concise description of what you want to happen.

datasets.get_dynamic_paths -> List[str] will be sufficient for my use case.

By offering this API, we will be able to address the following issues (by patching the ray integration sufficiently):

https://github.com/huggingface/blog/issues/106 https://github.com/huggingface/transformers/issues/11565 https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/34 https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/35

[lhoestq]

I like the idea as well ! thanks @albertvillanova

[richardliaw]

Hi @albertvillanova, the default value proposal seems great 😃 Looking forward to this!