Proxy server don't get UA that was set with page.setUserAgent
See original GitHub issueSteps to reproduce
import * as puppeteer from 'puppeteer';
import * as proxyChain from 'proxy-chain';
import * as http from 'http';
const proxyServer = new proxyChain.Server({
port: 8000,
prepareRequestFunction: ({ request }: { request: http.IncomingMessage; }) => {
console.log('headers on proxy', request.headers);
// no proxy
return {
upstreamProxyUrl: undefined
};
}
});
const main = async () => {
proxyServer.listen()
console.log('listening');
const browser = await puppeteer.launch({
args: ['--proxy-server=http://localhost:8000']
})
const page = await browser.newPage()
await page.setUserAgent('test user agent')
await page.setRequestInterception(true)
page.on('request', req => {
console.log('request headers', req.headers());
req.continue()
});
await page.goto('https://www.whatsmyua.info/api/v1/ua?ua')
const pageContent = await page.evaluate(() => {
return document.body.textContent
}).then(JSON.parse)
console.log({ result: pageContent[0].ua.rawUa });
}
main();
// 1.13.0
// request headers { 'upgrade-insecure-requests': '1', 'user-agent': 'test user agent' }
// headers on proxy {
// host: 'www.whatsmyua.info:443',
// 'proxy-connection': 'keep-alive',
// 'user-agent': 'test user agent'
// }
// { result: 'test user agent' }
// 1.15.0
// request headers { 'upgrade-insecure-requests': '1', 'user-agent': 'test user agent' }
// headers on proxy {
// host: 'www.whatsmyua.info:443',
// 'proxy-connection': 'keep-alive',
// 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
// '(KHTML, like Gecko) HeadlessChrome/75.0.3765.0 Safari/537.36'
// }
// { result: 'test user agent' }
Tell us about your environment:
- Puppeteer version: 1.13.0 - OK, 1.15.0 - NOT OK
- Platform / OS version: Windows 10
- Node.js version: 12
What is the expected result? Proxy server should get user-agent assigned to page, instead it gets default puppeteer user-agent
Issue Analytics
- State:
- Created 4 years ago
- Reactions:4
- Comments:13
Top Results From Across the Web
Proxy server don't get UA that was set with page.setUserAgent
Steps to reproduce import * as puppeteer from 'puppeteer'; import * as proxyChain from 'proxy-chain'; import * as http from 'http'; ...
Read more >Web scraping with pyppeteer, but site is blocking me
I'm trying to web scraping to get some coupom values applyed in a product in site, but it keeps blocking me. I've already...
Read more >Practical Puppeteer: Using proxy to browse a page
Today Puppeteer topic will be related to proxy. Using proxy when browse a page is useful when we want to hide our origin...
Read more >How to set User-Agent header with Puppeteer JS and not fail
Don't just blindly change your User-Agent header. ... We setup Puppeteer JS, open the browser, load webpage and get the data. Everything looks...
Read more >Scrapy Beginners Series Part 4 - User Agents and Proxies
In Part 4 we will be exploring how to use User Agents and Proxies to bypass restrictions on sites who are trying to...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
https://www.chromestatus.com/feature/4931699095896064 I think this issue is related to this feature. The https proxy needs to send
connect
, but it was limited.I also found this problem at puppeteer@1.14.0 version.