DNS lookup error exception

I am using requests to check domain status for k8s.io domains (https://github.com/kubernetes/k8s.io/pull/30) and I can’t find a good way to check for DNS lookup error like ERR_NAME_NOT_RESOLVED in Chrome. Exceptions output seems platform specific:

    Traceback (most recent call last):
      File "check.py", line 11, in <module>
        status = requests.head('http://' + site)
      File "C:\Python27\lib\site-packages\requests\api.py", line 93, in head
        return request('head', url, **kwargs)
      File "C:\Python27\lib\site-packages\requests\api.py", line 53, in request
        return session.request(method=method, url=url, **kwargs)
      File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
        resp = self.send(prep, **send_kwargs)
      File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
        r = adapter.send(request, **kwargs)
      File "C:\Python27\lib\site-packages\requests\adapters.py", line 437, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPConnectionPool(host='jenkins.k8s.io', port=80):
        Max retries exceeded with url: / (Caused by
        NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at
        0x029E43B0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))

So I was redirected here from https://github.com/kennethreitz/requests/issues/3630#issuecomment-255031285 to see if urllib3 could expose more fine-grained DNSLookupError exception and maybe remove MaxRetryError wrapper.

[Lukasa]

Ok, so.

We’re certainly not going to remove the MaxRetryError: that’s not a winner. But we can probably be more clear about the specific error by hoisting the getaddrinfo call on line 75 of urllib3/util/connection.py out of the for statement and shoving it in a try...except that raises an appropriate urllib3 exception.

That would at least give you a specific exception to try to haul out. MaxRetryError has a reason attribute which you can use to fetch the underlying exception, so you can from that point take the exception.

Note, however, that this change will take a while to propagate into a released version of Requests. In the interim, you may find it more helpful to check for DNS errors by simply calling socket.getaddrinfo yourself before making the request.

[pquentin]

So NameResolutionError only triggers when there is no DNS record, right? And if DNS server is not responding, a TimeoutError is raised. Is that it?

Right! (Well technically there are other possible errors: see “Return Value” at https://linux.die.net/man/3/gai_strerror.) This was the original request from psf/requests#3630 which can now be solved. Well, or at least if could if it was not in feature freeze. Sorry for the five years delay.

I wonder if urllib3 should support detecting if DNS server is down vs main server is down.

I’m not sure how useful that would be, but in any case that would be a different issue.