"no-token" is passed to `huggingface_hub` when token is `None`

Describe the bug

In the 2 lines listed below, a token is passed to huggingface_hub to get information from a dataset. If no token is provided, a “no-token” string is passed. What is the purpose of it ? If no real, I would prefer if the None value could be sent directly to be handle by huggingface_hub. I feel that here it is working because we assume the token will never be validated.

https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L753 https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L1121

Expected results

Pass token=None to huggingface_hub.

Actual results

token="no-token" is passed.

Environment info

huggingface_hub v0.10.0dev

[albertvillanova]

Hi @Wauplin, thanks for the warning about the deprecation of token in favor of use_auth_token.

Indeed, in datasets we use internally use_auth_token, which in this case was transformed to token to call HfApi.dataset_info: https://github.com/huggingface/datasets/blob/1a9385d7cc8a3241b44015145ef56a230fdadc51/src/datasets/load.py#L747

Therefore, for the new hfh release, the fix will be trivial: we will pass directly use_auth_token.

As discussed during our meeting yesterday, due to the fact that at datasets we support multiple hfh versions, I think we should handle passing token or use_auth_token depending on the hfh version.

[lhoestq]

As soon as token is deprecated and hfh has a new release, we’ll update datasets to use the new argument instead. Does it sound good to you ?