Kubernetes DNS and Remote Webdriver

Thank you for such an amazing tool you have built!

If I understand the functionality correctly, selenium-wire opens up a proxy socket on the addr FQDN (and random port), and any requests from selenium are routed through this proxy. This allows the proxy to capture and process all the requests.

It took me a while to understand that the addr FQDN is used for both - the creation of the proxy socket and from selenium to connect to the proxy.

I have set up a remote selenium grid on my kubernetes cluster, and I am trying to connect to it from another pod within my kubernetes cluster. The services for these pods are of type ClusterIP - meaning that the IP is randomly generated with each deployment. Kubernetes has intelligent DNS resolution where you can specify http://service-name:port, and it will resolve it to the IP address. So I should be able to open a port with service-name in the addr option, however when I try to do that, I get the following error: seleniumwire.thirdparty.mitmproxy.exceptions.ServerException: Error starting mitmproxy server: gaierror(-5, 'No address associated with hostname')

I tried using 127.0.0.1 or 0.0.0.0, and although I am able to create the proxy server, selenium is of course unable to connect to it and it fails with Message: unknown error: net::ERR_PROXY_CONNECTION_FAILED

It would be beneficial if I could define, for example, an addr to start the proxy server (where I could use 127.0.0.1) and another option to be able to reach that proxy server from selenium (where I can leverage the kubernetes DNS resolution).

[wkeeling]

Thanks for raising this. You’re right, Selenium Wire sends all browser traffic through an internal proxy it spins up in the background, and it uses the same address for both the proxy server and for Selenium itself when it configures the browser. There isn’t the ability to separate these currently in Selenium Wire itself, but I think there may be a workaround.

If you’re using the latest version of Selenium Wire, then there’s an option called auto_config. This tells Selenium Wire to configure the browser - via Selenium - with the IP/port of it’s internal proxy. The option is set to True by default but if you set this to False then Selenium Wire won’t configure the browser and will assume you will do it manually. You can configure the browser by passing a browser specific option (I’m assuming Chrome here) and specify the Kubernetes service-name at this point.

Here’s some code that demonstrates how to do it:

from seleniumwire import webdriver

sw_options = {
    'auto_config': False,  # Ensure this is set to False
    'addr': '0.0.0.0',  # The address the proxy will listen on
    'port': 8087,
}

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=service-name:8087')  # Specify your Kubernetes service-name here
chrome_options.add_argument('--ignore-certificate-errors')

driver = webdriver.Remote(
    desired_capabilities=chrome_options.to_capabilities(),
    seleniumwire_options=sw_options,
)

driver.get(...)

Would that allow you to take advantage of the Kubernetes DNS resolution?

[wkeeling]

Great news that the workaround works! I may now look at adding the above example to the readme, as it may provide useful for other people running a container setup.

Regarding printing the requests, you just need to make sure that you print them before calling browser.quit(). Quitting the browser will shutdown Selenium Wire and clear out all captured requests. So switching the statements around should fix:

finally:
    print(browser.requests)
    browser.quit()  # Clears out request storage