Issues with HTTP proxy and accessing localhost - does requests ignore no_proxy?

Hi,

I have a Django app, exposing a REST API using TastyPie.

I have it running locally using Django’s runserver, and I’m trying to access it using Requests.

For some reason though, accessing it with Curl or Firefox works fine:

curl "http://localhost:8000/api/v1/host/?name__regex=&format=json"

And on the Django runserver console, I see:

[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&format=json HTTP/1.1" 200 2845

However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
<Response [404]>
>>> r.url
u'http://localhost:8000/api/v1?format=json'

Also, on the Django runserver console, I see:

[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072

Notice that runserver prints the whole path in this case, rather than just the part after localhost:8000 - not sure why that is.

We have a HTTP proxy setup in this environment (Microsoft ISA, I believe), which uses authentication (I’ve put the username/password in the variable itself).

I’ve set the no_proxy environment variable and put localhost and 127.0.0.1 in there.

However, requests seems to still be doing something funky with the proxy.

If I actually unset http_proxy - requests works fine:

$ unset http_proxy
$ python
Python 2.7.3 (default, Sep  7 2012, 12:28:31) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> hostname = ''
>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex={}&format=json'.format(hostname))
>>> r
<Response [200]>

So firstly, I’m curious why requests doesn’t work with the proxy variable set? And secondly, is there any any way to get it to respect no_proxy?

Cheers, Victor

[tanmay9]

Hi @Lukasa , I was having this issue with Requests 1.2. After reading this thread, I thought it will go away on upgrading to 2.0.1 (which contains your fix). However, it did not fix it for me. The only way I got around it was to add the following code:

import os
os.environ['no_proxy'] = '127.0.0.1,localhost'

[timsavage]

I have a similar problem where I just want to disable the proxy for certain requests (connecting to internal WebServices that are not available outside our DMZ). After digging through the requests code to find out how proxies are handled and how proxy configuration is loaded I have come up with a work-around to disable the proxy when needed:

NO_PROXY = {
    'no': 'pass',
}

r = requests.get(uri, proxies=NO_PROXY)

How this works, if the proxies parameter is passed to the get (or other http method) it is has any empty entries removed before checking if there are any values still in the dictionary. If the dictionary is empty then the variables are loaded from the environment using the method requests.utils.get_environ_proxies. By including a value in the ‘no’ key then the proxies parameter is used which includes empty http and https entries thereby disabling the proxy. The method is a bit hacky but does solve the issue.