Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

feat(dl_manager): add support for ftp downloads

See original GitHub issue

I am working on a new dataset (#302) and encounter a problem downloading it.

# This is the official download link from https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/
_URL = "ftp://wasserstoff.informatik.rwth-aachen.de/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz"

dl_manager.download_and_extract(_URL)

I get an error:

ValueError: unable to parse ftp://wasserstoff.informatik.rwth-aachen.de/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz as a URL or as a local path

I checked, and indeed you don’t consider ftp as a remote file. https://github.com/huggingface/datasets/blob/4c2af707a6955cf4b45f83ac67990395327c5725/src/datasets/utils/file_utils.py#L188

Adding ftp to that list does not immediately solve the issue, so there probably needs to be some extra work.

Issue Analytics

State:
Created 3 years ago
Comments:11 (8 by maintainers)

Top GitHub Comments

1reaction

AmitMYcommented, Oct 24, 2020

The dataset loader is not yet ready, because of that issue. If you want to just download the dataset the old-fashioned way, just go to: https://www-i6.informatik.rwth-aachen.de/ftp/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz (the ftp link is now broken, and its available over https)

1reaction

AmitMYcommented, Oct 24, 2020

@hoanganhpham1006 yes. See pull request https://github.com/huggingface/datasets/pull/722 , it has a loader for this dataset, mostly ready. There’s one issue that delays it being merged - https://github.com/huggingface/datasets/issues/741 - regarding memory consumption.