Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Pagespeed API] Support custom user agent in pagespeed API

See original GitHub issue

Pagespeed exhibits bot like behavior which can get caught by our bot detection mechanisms, preventing us from measuring pagespeed. I’d like to request the addition of a custom user agent param for pagespeed to use. We can then whitelist the specified user agent to allow pagespeed to run correctly. Alternatively, if you could provide the network blocks from which the pagespeed is run, we can whitelist those.

Provide the steps to reproduce

Visit https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.redfin.com

What is the current behavior?

Page can show an error saying: Lighthouse returned error: ERRORED_DOCUMENT_REQUEST. Lighthouse was u nable to reliably load the page you requested. Make sure you are testing the correct URL and that the server is properly responding to all requests. (Status code: 403) (It may not error out all the time. We’ve temporarily whitelisted the block 66.249.84.0/24, but I believe that block is used for more than just page speed.)

What is the expected behavior?

Pagespeed runs without an error.

Environment Information

Affected Channels: Pagespeed v5 API

Issue Analytics

State:
Created 4 years ago
Reactions:1
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

paulirishcommented, Apr 28, 2020

If you need to identify that its a Lighthouse pageload that you initiated, consider adding a special URL param.

Also, you can always clone Lighthouse and run the CLI locally with whatever changes you want. 😃

0reactions

angelogulinacommented, Jun 13, 2019

PSI does include Chrome-Lighthouse in the request User-Agent. Is that sufficient to filter for?

Jumping in the conversation (hopefully relevant) given it’s still Open.

Thought the Chrome-Lighthouse is provided, it would be nice to be able to add additional User-Agent parts. Now, as far as I could see from here and here it should be actually possible to do that.

But, if I look at the generated json, I cannot find any trace of the additional part I added. To be fair, I still haven’t tested it from the server-side part - but I suppose that additional User-Agent parts should also be displayed in the json from lighthouseResult > environment> networkUserAgent.

Or am I doing / getting something wrong?