Issue with downloading Wikipedia data for low resource language

Hi, I tried to download Sundanese and Javanese wikipedia data with the following snippet

jv_wiki = datasets.load_dataset('wikipedia', '20200501.jv', beam_runner='DirectRunner')
su_wiki = datasets.load_dataset('wikipedia', '20200501.su', beam_runner='DirectRunner')

And I get the following error for these two languages: Javanese

FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/jvwiki/20200501/dumpstatus.json

Sundanese

FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/suwiki/20200501/dumpstatus.json

I found from https://github.com/huggingface/datasets/issues/577#issuecomment-688435085 that for small languages, they are directly downloaded and parsed from the Wikipedia dump site, but both of https://dumps.wikimedia.org/jvwiki/20200501/dumpstatus.json and https://dumps.wikimedia.org/suwiki/20200501/dumpstatus.json are no longer valid.

Any suggestions on how to handle this issue? Thanks!

[lhoestq]

Thanks for reporting I created a PR to make the custom config work (language=“zh”, date=“20201120”).

[kellymarchisio]

For posterity, here’s how I got the data I needed: I needed Bengali, so I had to check which dumps are available here: https://dumps.wikimedia.org/bnwiki/ , then I ran:

load_dataset("wikipedia", language="bn", date="20211101",
                        beam_runner="DirectRunner")