Unable to download IWSLT datasets

🐛 Bug

Describe the bug Unable to download IWSLT2016 or IWSLT2017 datasets.

To Reproduce Steps to reproduce the behavior:

from torchtext.datasets import IWSLT2016
train, valid, test = IWSLT2016()
src, tgt = next(iter(train))

The same error occurs when trying to use IWSLT2017.

Expected behavior The program returns the next src, tgt pair in the training data.

Screenshots Full error logs are in this gist.

Environment Included in gist above.

Additional context No additional context.

[adzcai]

As a temporary fix, I’m just downloading the datasets manually via the links in the documentation:

• https://wit3.fbk.eu/2016-01 for IWSLT2016
• https://wit3.fbk.eu/2017-01 for IWSLT2017

Then you can put the downloaded .tgz file into the proper directory: ~/.torchtext/cache/IWSLT2016/ for 2016 and similar for 2017.

Then torchtext will recognize the files and not download from GDrive.

[lolzballs]

@Nayef211 thanks, it does sound like exactly what I’m observing with IWSLT.

But I tried what is suggested in #1735 with (note the order of end_caching here and in the original code):

def _filter_clean_cache(cache_decompressed_dp, full_filepath, uncleaned_filename):

    cache_inner_decompressed_dp = cache_decompressed_dp.on_disk_cache(
        filepath_fn=partial(_return_full_filepath, full_filepath)
    )
    cache_inner_decompressed_dp = cache_inner_decompressed_dp.open_files(mode="b").load_from_tar()
    cache_inner_decompressed_dp = cache_inner_decompressed_dp.end_caching(mode="wb", same_filepath_fn=True)
    cache_inner_decompressed_dp = cache_inner_decompressed_dp.filter(partial(_filter_filename_fn, uncleaned_filename))
    cache_inner_decompressed_dp = cache_inner_decompressed_dp.map(partial(_clean_files_wrapper, full_filepath))
    return cache_inner_decompressed_dp

I still get the same behaviour: the inner load_from_tar() never gets iterated over.