Selenium frequently times out while loading test pages during Percy builds on master

This is causing master to go red often.

For example, see https://travis-ci.org/ampproject/amphtml/jobs/346992913#L656

pr-check.js: Running gulp visual-diff --headless --master...
[21:33:59] Using gulpfile ~/build/ampproject/amphtml/gulpfile.js
[21:33:59] Starting 'visual-diff'...
INFO: Started Percy build 550164...
/home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/protocol.rb:176:in `rbuf_fill': Net::ReadTimeout (Net::ReadTimeout)
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/protocol.rb:154:in `readuntil'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/protocol.rb:164:in `readline'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http/response.rb:40:in `read_status_line'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http/response.rb:29:in `read_new'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:1446:in `block in transport_request'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:1443:in `catch'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:1443:in `transport_request'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:1416:in `request'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:1409:in `block in request'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:877:in `start'
	from /home/travis/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/net/http.rb:1407:in `request'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/remote/http/default.rb:121:in `response_for'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/remote/http/default.rb:76:in `request'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/remote/http/common.rb:59:in `call'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/remote/bridge.rb:164:in `execute'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/remote/oss/bridge.rb:579:in `execute'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/remote/oss/bridge.rb:50:in `get'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/selenium-webdriver-3.9.0/lib/selenium/webdriver/common/navigation.rb:30:in `to'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/capybara-2.18.0/lib/capybara/selenium/driver.rb:49:in `visit'
	from /home/travis/.rvm/gems/ruby-2.4.1/gems/capybara-2.18.0/lib/capybara/session.rb:274:in `visit'
	from build-system/tasks/visual-diff.rb:361:in `block in snapshot_webpages'
	from build-system/tasks/visual-diff.rb:349:in `each'
	from build-system/tasks/visual-diff.rb:349:in `snapshot_webpages'
	from build-system/tasks/visual-diff.rb:334:in `block in generate_snapshots'
	from build-system/tasks/visual-diff.rb:323:in `each'
	from build-system/tasks/visual-diff.rb:323:in `generate_snapshots'
	from build-system/tasks/visual-diff.rb:274:in `run_visual_tests'
	from build-system/tasks/visual-diff.rb:496:in `main'
	from build-system/tasks/visual-diff.rb:501:in `<main>'

[danielrozenberg]

No longer relevant as we switched from Selenium to Puppeteer

[timhaines]

@rsimha Perhaps unrelated - but I’m wondering if there’s something in the example pages - some AMP components, or the way they’re being served - which is causing both chromedriver and the integration tests to time out.

And yes, it waited 2 minutes to load that page. You can tell from timestamps in the the command navigate and response navigate lines.

$ irb
2.3.0 :001 > Time.at 1522336662.399
 => 2018-03-29 08:17:42 -0700
2.3.0 :002 > Time.at 1522336792.508
 => 2018-03-29 08:19:52 -0700
2.3.0 :003 >

You could increase the timeout higher than 2 minutes - to 10 minutes or whatever you wanted. It would just be adjusting the ‘120’ in the last PR I submitted. I’m not sure that increasing it is going to help though. If 2 minutes isn’t enough for one page, 10 minutes is unlikely to be.

I think this timeout you’ve linked to is either your components or serving app blocking and not returning a response, or is a bug in either chromedriver or chrome. You can see some similar (but definitely different) bugs have been reported in chromedriver/chrome recently: https://bugs.chromium.org/p/chromedriver/issues/detail?id=2239

This isn’t a bug in Percy at all - it’s occurring during the process of selenium/chromedriver/chrome trying to navigate to a page, to then capture the dom snapshot, and later send to Percy. When you were using phantomJS, chromedriver+chrome weren’t involved, which I think further suggests it’s an issue with chromedriver/chrome. Do you have internal access to talk to the chromedriver team about it?

We don’t have a documented way of using chromedriver/chrome with Percy. What you’re doing, using Capybara to navigate pages, is probably closest to our percy-anywhere documentation, which suggests to use the poltergeist gem and phantomjs, and we haven’t had anyone report this sort of problem with it. Having said that, I can understand phantomjs may not be well suited to what you’re doing with AMP.

Lets finish up the commercial process with Malte, and then we’ll be able to help out in more depth with this.