Cannot download package possibly because of urllib defaults

I’m trying to import fontawesome to a project using Peru 1.3.0.

imports:
  fontawesome: deps/font-awesome

curl module fontawesome:
  url: https://use.fontawesome.com/releases/v5.15.4/fontawesome-free-5.15.4-web.zip
  unpack: zip

But it errors:

$ peru sync
In target "fontawesome":
  Error fetching https://use.fontawesome.com/releases/v5.15.4/fontawesome-free-5.15.4-web.zip
  HTTP Error 403: Forbidden

I think it’s a urllib problem:

from urllib.error import HTTPError
import urllib.request

url = "https://use.fontawesome.com/releases/v5.15.4/fontawesome-free-5.15.4-web.zip"

try:
  req = urllib.request.urlopen(url)
except HTTPError as e:
  print(e.headers)
# this will fail and print something

# this succeeds
import urllib3
http = urllib3.PoolManager()
resp = http.request("GET", "https://use.fontawesome.com/releases/v5.15.4/fontawesome-free-5.15.4-web.zip")
print(resp.status)
# succeeds

The urllib request fails but urllib3 succeeds. I don’t think urllib’s HTTPError gives you access to the request headers, so I can’t really tell what’s wrong here. I’m able to retrieve the file with curl, wget, aria2c, and python via urllib3.

Running peru --verbose sync gave me a stack trace that simply emphasizes “yeah, that’s a 403” and the headers I get back from the request don’t really say what’s wrong. I assume it’s some kind of bot protection on Cloudflare…?

[colindean]

I tested #219 with the example config and it works! It will fix the problem.

[colindean]

#219 goes with № 3 in that list and pulls the urllib UA directly from a core class that won’t go away. urllib doesn’t — IMO — sufficiently abstract this value 👎