Firefox not working on AWS EC2 instance

I’m trying to run Firefox with Seleniumwire on an AWS EC2 instance. Here’s my current configuration:

• ubuntu==20.4
• selenium-wire==4.3.0
• python==3.8.10
• geckodriver==0.29.1
• ChromeDriver==94.0.4606.81
• 64-bit (Arm)

I’ve installed ChromeDriver with sudo apt-get install chromium-chromedriver and geckodriver with sudo apt-get install firefox-geckodriver.

Here’s the script I’m trying to run:

from seleniumwire import webdriver

options = webdriver.FirefoxOptions()
options.add_argument("--headless")
options.add_argument('--no-sandbox')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('--ignore-ssl-errors')
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-gpu")

d = webdriver.Firefox(options=options)
d.get("https://google.com")
print(d.requests)

The problem: d.requests returns [], which is unexpected behavior. I know there’s been issues posted about driver.requests returning an empty list, but the recommended solutions haven’t worked on my setup. Also print(d.page_source) works fine, which leads me to think there’s some incorrect proxy configuration. Also, I isolated this problem to Firefox, since the same code works on Chrome (with Firefox swapped out for Chrome); that is, d.requests contains a non-empty list of requests. I would use Chrome, but the scope of my project requires Firefox.

Here’re some steps I’ve taken to debug:

• Setting the port explicitly using selenium_options={"port": 4444}, where 4444 is taken from the localhost output of geckodriver on the command line.
• Adding a proxy using the following format:

options = {
        'proxy': {
            'http': 'http://myusername:[e-mail]:123456', 
            'https': 'http://myusername:[e-mail]:123456',
            'no_proxy': 'localhost,127.0.0.1'
        }  
    }

• Re-installing and re-running all my scripts on a new EC2 instance.
• Specifying a different tmp directory, per [this] issue (https://github.com/wkeeling/selenium-wire/issues/131)

I have yet to try different versions of geckodriver and selenium-wire, but I suspect that’s not the issue.

Any suggestions on debugging would be greatly appreciated.

EDIT:

I enabled debugging and received the following output:

>>> options = webdriver.FirefoxOptions()
>>> options.add_argument("--headless")
>>> options.add_argument('--ignore-certificate-errors-spki-list')
>>> d = webdriver.Firefox(options=options)
INFO:seleniumwire.backend:Created proxy listening on 127.0.0.1:46473
DEBUG:selenium.webdriver.remote.remote_connection:POST http://localhost:32879/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", "acceptInsecureCerts": true, "moz:debuggerAddress": true, "pageLoadStrategy": "normal", "moz:firefoxOptions": {"args": ["--headless", "--ignore-certificate-errors-spki-list"]}}}, "desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": true, "moz:debuggerAddress": true, "pageLoadStrategy": "normal", "moz:firefoxOptions": {"args": ["--headless", "--ignore-certificate-errors-spki-list"]}}}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:32879
DEBUG:urllib3.connectionpool:http://localhost:32879 "POST /session HTTP/1.1" 200 731
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
>>> d.get('https://google.com')
DEBUG:selenium.webdriver.remote.remote_connection:POST http://localhost:32879/session/1f24cf37-0ff3-4b06-a25e-17c405877477/url {"url": "https://google.com"}
DEBUG:urllib3.connectionpool:http://localhost:32879 "POST /session/1f24cf37-0ff3-4b06-a25e-17c405877477/url HTTP/1.1" 200 14
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
>>> print(d.requests)
[]

From my understanding, it seems that a proxy was created on port 46473 but a connection was made on 32879.

[wkeeling]

OK so I was able to reproduce this issue. It was being caused by the recent release of Selenium (v4.0.0) which expects proxy configuration to be passed in a different way to the way Selenium Wire had been using up to now. As a result, the proxy configuration was being ignored causing driver.requests to always be empty.

I’ve fixed the issue in Selenium Wire 4.5.4. The fix maintains backwards compatibility with earlier Selenium versions, but the following setup should now work:

selenium-wire 4.5.4
selenium 4.0.0
geckodriver 0.30.0

Let me know if you hit any issues with that!

[paulcalabro]

@trimbilrepo

I think this is the source of the issue: https://github.com/SeleniumHQ/selenium/blob/42218df526d3c25843eabc6e9b6bc5082d43e7fe/dotnet/CHANGELOG#L40

Geckodriver: error: Found argument '--websocket-port' which wasn't expected, or isn't valid in this context

USAGE:
    geckodriver [FLAGS] [OPTIONS]

For more information try --help

Though when I upgraded to the latest 0.30.0 release, I started seeing issues with driver.requests being empty.