question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Proxy server don't get UA that was set with page.setUserAgent

See original GitHub issue

Steps to reproduce

import * as puppeteer from 'puppeteer';
import * as proxyChain from 'proxy-chain';
import * as http from 'http';

const proxyServer = new proxyChain.Server({
    port: 8000,
    prepareRequestFunction: ({ request }: { request: http.IncomingMessage; }) => {
        console.log('headers on proxy', request.headers);

        // no proxy
        return {
            upstreamProxyUrl: undefined
        };
    }
});


const main = async () => {
    proxyServer.listen()
    console.log('listening');

    const browser = await puppeteer.launch({
        args: ['--proxy-server=http://localhost:8000']
    })
    const page = await browser.newPage()
    await page.setUserAgent('test user agent')
    await page.setRequestInterception(true)
    page.on('request', req => {
        console.log('request headers', req.headers());
        req.continue()
    });
    await page.goto('https://www.whatsmyua.info/api/v1/ua?ua')
    const pageContent = await page.evaluate(() => {
        return document.body.textContent
    }).then(JSON.parse)
    console.log({ result: pageContent[0].ua.rawUa });
}

main();

// 1.13.0
// request headers { 'upgrade-insecure-requests': '1', 'user-agent': 'test user agent' }
// headers on proxy {
//   host: 'www.whatsmyua.info:443',
//   'proxy-connection': 'keep-alive',
//   'user-agent': 'test user agent'
// }
// { result: 'test user agent' }

// 1.15.0
// request headers { 'upgrade-insecure-requests': '1', 'user-agent': 'test user agent' }
// headers on proxy {
//   host: 'www.whatsmyua.info:443',
//   'proxy-connection': 'keep-alive',
//   'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
//     '(KHTML, like Gecko) HeadlessChrome/75.0.3765.0 Safari/537.36'
// }
// { result: 'test user agent' }

Tell us about your environment:

  • Puppeteer version: 1.13.0 - OK, 1.15.0 - NOT OK
  • Platform / OS version: Windows 10
  • Node.js version: 12

What is the expected result? Proxy server should get user-agent assigned to page, instead it gets default puppeteer user-agent

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Reactions:4
  • Comments:13

github_iconTop GitHub Comments

1reaction
MeiK2333commented, Nov 14, 2019

https://www.chromestatus.com/feature/4931699095896064 I think this issue is related to this feature. The https proxy needs to send connect, but it was limited.

1reaction
xuelliucommented, May 5, 2019

I also found this problem at puppeteer@1.14.0 version.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Proxy server don't get UA that was set with page.setUserAgent
Steps to reproduce import * as puppeteer from 'puppeteer'; import * as proxyChain from 'proxy-chain'; import * as http from 'http'; ...
Read more >
Web scraping with pyppeteer, but site is blocking me
I'm trying to web scraping to get some coupom values applyed in a product in site, but it keeps blocking me. I've already...
Read more >
Practical Puppeteer: Using proxy to browse a page
Today Puppeteer topic will be related to proxy. Using proxy when browse a page is useful when we want to hide our origin...
Read more >
How to set User-Agent header with Puppeteer JS and not fail
Don't just blindly change your User-Agent header. ... We setup Puppeteer JS, open the browser, load webpage and get the data. Everything looks...
Read more >
Scrapy Beginners Series Part 4 - User Agents and Proxies
In Part 4 we will be exploring how to use User Agents and Proxies to bypass restrictions on sites who are trying to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found