Archive Method: Wget: unrecognized option '--compression=auto'

Hey there, I just pulled the master and poked around and then I realized that the wget command isn’t working correctly. Running via docker-compose.

Describe the bug

wget is being executed with the following standard parameters:

wget --no-verbose --adjust-extension --convert-links --force-directories --backup-converted --span-hosts --no-parent --compression=auto -e robots=off --restrict-file-names=unix --timeout=60 --warc-file=warc/1553663964 --page-requisites "--user-agent=ArchiveBox/58c9b47d4 (+https://github.com/pirate/ArchiveBox/) wget/1.18" https://www.theatlantic.com/health/archive/2014/05/why-medicine-is-cheaper-in-germany/371418/

If I run this outside of the script, I get a wget: unrecognized option '--compression=auto' which makes wget basically stopp working.

Steps to reproduce

Run the above mentioned wget command

My workaround was to edit archivebox/archive_methods.py manually inside the container and comment out the compression=auto there.

[bradparks]

I get this error as well, and I just cloned this now, March 30 at 9:12 PM AST

[pirate]

This should be fixed on the latest django branch. If you’re still seeing any issues comment back here and I’ll reopen the ticket.

git checkout django
git pull
docker build . -t archivebox
docker run -v $PWD/output:/data archivebox init
docker run -v $PWD/output:/data archivebox add 'https://example.com'