question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Proxy-Authorization header not working in Chromium

See original GitHub issue

Hello there, thanks for this great project.

I’m having trouble making requests though a proxy. Consider the following snippet:

import asyncio
from playwright import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={
                "server": "PROXY_URL",
                "username": "PROXY_APIKEY",
                "password": "",
            },
        )
        context = await browser.newContext(ignoreHTTPSErrors=True)
        page = await context.newPage()
        await page.goto("https://httpbin.org/ip")
        print(await page.content())
        await page.screenshot(path="ip_address.png")
        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())

If I run it, I get:

$ DEBUG=pw:api python proxy.py 
  pw:api navigating to "https://httpbin.org/ip", waiting until "load" +0ms
  pw:api   navigated to "https://httpbin.org/ip" +1s
  pw:api   "load" event fired +7ms
  pw:api   "domcontentloaded" event fired +8ms
<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">Internal Server Error</pre></body></html>

I can confirm the proxy URL and credentials are correct with cURL (the resulting IP address is not my own, as expected):

$ curl --insecure --proxy $PROXY_URL --proxy-user $PROXY_APIKEY: https://httpbin.org/ip
{
  "origin": "xxx.xxx.xxx.xxx"
}

Could this be caused by the fact that the proxy server expects an empty password and Playwright sends some default value? For the record, I’m using Python 3.8.2 and Playwright 0.171.1 on GNU/Linux.

Looking forward to your answers, thanks again for the hard work.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:11 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
elacuestacommented, Jan 20, 2021

Hi, thanks for looking into this.

The proxy provider has instructed me to manually set the Proxy-Authorization header (the article is about Puppeteer, I suppose the concepts should still apply). Unfortunately, I’m still having issues, even with a local proxy (mitmproxy --proxyauth "user:pass"):

import asyncio
from base64 import b64encode
from playwright import async_playwright

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={"server": "localhost:8080"},
            # args=[
            #     "--proxy-server=localhost:8080",
            # ],
        )
        context = await browser.newContext(ignoreHTTPSErrors=True)
        page = await context.newPage()
        await page.setExtraHTTPHeaders({
            "Proxy-Authorization": "Basic " + b64encode(b"user:pass").decode("utf8"),
        })
        await page.goto("https://httpbin.org/ip")
        print(await page.content())
        await browser.close()

if __name__ == "__main__":
    asyncio.run(main())
$ DEBUG=pw:api python proxy.py 
  pw:api navigating to "https://httpbin.org/ip", waiting until "load" +0ms
Traceback (most recent call last):
  File "proxy.py", line 23, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "proxy.py", line 18, in main
    await page.goto("https://httpbin.org/ip")
  File "/.../lib/python3.8/site-packages/playwright/async_api.py", line 4269, in goto
    await self._impl_obj.goto(
  File "/.../lib/python3.8/site-packages/playwright/_page.py", line 452, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
  File "/.../lib/python3.8/site-packages/playwright/_frame.py", line 114, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/.../lib/python3.8/site-packages/playwright/_connection.py", line 36, in send
    return await self.inner_send(method, params, False)
  File "/.../lib/python3.8/site-packages/playwright/_connection.py", line 47, in inner_send
    result = await callback.future
playwright._types.Error: net::ERR_INVALID_ARGUMENT at https://httpbin.org/ip
=========================== logs ===========================
navigating to "https://httpbin.org/ip", waiting until "load"
============================================================
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.

What seems strange to me is that I’m also having the same issue with plain Playwright:

const { chromium } = require('playwright');

(async () => {
    const browser = await chromium.launch({
        proxy: {
            server: 'localhost:8080',
            // username: 'user',
            // password: 'pass',
        }
    });
    const context = await browser.newContext({ignoreHTTPSErrors: true})
    const page = await context.newPage();
    await page.setExtraHTTPHeaders({
        'Proxy-Authorization': 'Basic ' + Buffer.from('user:pass').toString('base64'),
    });
    try {
        await page.goto('https://httpbin.org/ip');
        console.log(await page.content());
    } catch(err) {
        console.log(err);
    }
    await browser.close();
})();
$ node proxy.js 
page.goto: net::ERR_INVALID_ARGUMENT at https://httpbin.org/ip
=========================== logs ===========================
navigating to "https://httpbin.org/ip", waiting until "load"
============================================================
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.
    at Connection.sendMessageToServer (/.../node_modules/playwright/lib/client/connection.js:69:15)
    at Proxy.<anonymous> (/.../node_modules/playwright/lib/client/channelOwner.js:44:61)
    at /.../node_modules/playwright/lib/client/frame.js:80:71
    at Frame._wrapApiCall (/.../node_modules/playwright/lib/client/channelOwner.js:72:34)
    at Frame.goto (/.../node_modules/playwright/lib/client/frame.js:78:21)
    at /.../node_modules/playwright/lib/client/page.js:267:60
    at Page._attributeToPage (/.../node_modules/playwright/lib/client/page.js:202:20)
    at Page.goto (/.../node_modules/playwright/lib/client/page.js:267:21)
    at /.../proxy.js:17:20

Is there a chance this might be an upstream issue that should be solved in https://github.com/microsoft/playwright instead?

0reactions
pavelfeldmancommented, Mar 30, 2021

Closing as per above!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Regarding "Proxy-Authorization" header in chrome ...
I need to use "Proxy-Authorization" header in "onBeforeSendHeaders" event. but in documentation I see that this Header is currently not provided.
Read more >
Allow setting Proxy-Authorization in chrome.webRequest API
It should be possible to set the Proxy-Authorization header in the chrome.webRequest API. chrome.webRequest. ... proxyAuthValue}); // Facing problem here
Read more >
Chrome not setting Proxy-Authorization header while testing ...
Hi all,I'm doing testing from API Explorer in Portal. I have a swagger file which has apikey,apikeysecret and Proxy-Authorization passed as ...
Read more >
Refused to set unsafe header "Proxy-Authorization" Chrome
enter image description here. We're having the same problem. I'm using axios inside electron it seems that axios is trying to override the ......
Read more >
Why does Chrome not allow the modification of these headers ...
Proxy-AuthorizationThe HTTP Proxy-Authorization request header contains the credentials to authenticate a user agent to a proxy server, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found