Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Configuring web-requests to use a proxy

See original GitHub issue

I think there is something preventing CurlAsyncHTTPClient from accepting defaults.

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient", defaults=defaults) didn’t work from config.yaml.

I had to do this hack:

  extraConfig: |
    import pycurl
    from tornado.httpclient import HTTPRequest

    def configure_proxy(curl):
        # we only want google oauth to use the proxy
        if "google" in curl.getinfo(pycurl.EFFECTIVE_URL):
            logging.error("adding proxy")
            curl.setopt(pycurl.PROXY, "")
            curl.setopt(pycurl.PROXYPORT, 8080)

    # never do this
    HTTPRequest._DEFAULTS['prepare_curl_callback'] = configure_proxy

I don’t know why this doesn’t work:

    import certifi
    from tornado.httpclient import AsyncHTTPClient

    defaults2 = dict(ca_certs=certifi.where())
    defaults2['proxy_host'] = ''
    defaults2['proxy_port'] = 8080
    AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient", defaults=defaults2)

Actually, I do have one guess. Maybe this overwrite the defaults?

Issue Analytics

  • State:open
  • Created 5 years ago
  • Reactions:2
  • Comments:7 (2 by maintainers)

github_iconTop GitHub Comments

cblomartcommented, Oct 16, 2019

this feels much like hacking…

seems that the issue is with tornado that doesn’t respect http_proxy environment variable for CurlAsyncHttpClient:

The conclusion for the tornado issue is:

OK. In that case you must tell tornado that you want to use a proxy by setting the proxy_host and proxy_port arguments.

Even for simple_httpclient i don’t know if it supports no_proxy


looks like hacking into pycurl is the only solution for now…

gsemetcommented, May 15, 2019

Hi. Would be so cool if everybody simply respected the “http_proxy” environment variables…

Here is my hack to make the hub happy with our proxy (our Gitlab instance is behind our proxy). It also support no_proxy with “*”, so that we can have finer proxy tuning:

    GITLAB_HOST: "http://external.gitlab.server"
    no_proxy: localhost,,*,10.*,localdomain,cluster.local
    NO_PROXY: localhost,,*,10.*,localdomain,cluster.local
  extraConfig: |
    # HACK: consume HTTP?_PROXY and NO_PROXY environment variables
    # so Hub can connect to external Gitlab.
    import pycurl
    import os
    import logging
    from tornado.httpclient import HTTPRequest
    from urllib.parse import urlparse
    from fnmatch import fnmatch

    def get_proxies_for_url(url):
        http_proxy = os.environ.get("HTTP_PROXY", os.environ.get("http_proxy"))
        https_proxy = os.environ.get("HTTPS_PROXY", os.environ.get("https_proxy"))
        no_proxy = os.environ.get("NO_PROXY", os.environ.get("no_proxy"))
        p = urlparse(url)
        netloc = p.netloc
        _userpass,_, hostport = p.netloc.rpartition("@")
        url_hostname, _,  _port = hostport.partition(":")
        proxies = {}
        if http_proxy:
            proxies["http"] = http_proxy
        if https_proxy:
            proxies["https"] = https_proxy
        if no_proxy:
            for hostname in no_proxy.split(","):
                # Support "*" and "10.*"
                if fnmatch(url_hostname, hostname.strip()):
                    proxies = {}
                # Support ""
                elif hostname.strip().replace("*", "").endswith(url_hostname):
                    proxies = {}
                # TODO: support network mask:
        return proxies

    def configure_proxy(curl):
        logging.error("URL: {0}".format(curl.getinfo(pycurl.EFFECTIVE_URL)))
        # we only want google oauth to use the proxy
        proxies = get_proxies_for_url(curl.getinfo(pycurl.EFFECTIVE_URL))
        if proxies:
            host, _, port = proxies["https"].rpartition(":")
            logging.error("adding proxy: https={0}:{1}".format(host, port))
            curl.setopt(pycurl.PROXY, host)
            if port:
                curl.setopt(pycurl.PROXYPORT, int(port))

    # never do this
    HTTPRequest._DEFAULTS['prepare_curl_callback'] = configure_proxy
Read more comments on GitHub >

github_iconTop Results From Across the Web

WebRequest.Proxy Property (System.Net) | Microsoft Learn
The Proxy property identifies the network proxy that the request uses to access the Internet resource. The request is made through the proxy...
Read more >
How to use PowerShell Invoke-WebRequest behind corporate ...
Let's see how you can set proxy for Invoke-WebRequest for example. Other commands usually support proxy settings similarly.
Read more >
C# Connecting Through Proxy - Stack Overflow
config. This sets a default proxy that all http requests will use. Depending upon exactly what you need to achieve, you may or...
Read more >
Send web requests through an upstream proxy in WAN
Configure Sophos Firewall to use the upstream proxy in WAN. In this example, the upstream proxy is in the WAN zone. The network...
Read more >
Enabling HTTP proxy for .NET webservice client
You can start tweaking your Web.config file to set this proxy the right way, ... Proxy = WebRequest.DefaultWebProxy; request.Proxy.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found