Suggestion: driver.execute_script

Hi,

First of all, thanks for making this, I was getting a headacke trying to use scrapy + splash. This solution is much better, the only thing I’m missing is an option to execute custom javascript code. Right now I solved this using class inheritance, but it would be great if you can integrate it into the main program.

middlewares.py

class CustomSeleniumMiddleware(SeleniumMiddleware):

    def __init__(self, driver_name, driver_executable_path, driver_arguments):
        super().__init__(driver_name, driver_executable_path, driver_arguments)
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

settings.py

DOWNLOADER_MIDDLEWARES = {
    'xxx.middlewares.CustomSeleniumMiddleware': 800
}

[eyusupov]

I was also looking for a way to pass the script with the request (because I was coming from scrapy_splash), but then figured out that the same can be done in the Spider code with

response.meta['driver'].execute_script("window.scrollTo(0, document.body.scrollHeight);")

[j4n7]

Well,

I think that is probably enough for most cases, but sometimes (as I do) yo will need a different proxy setting for each spider. For example, there are some websites you can only access from a certain country and you will need different proxies for that. That is whay I have come up with the idea of making a proxy parameter in SeleniumRequest.

The problem I see is that the Middleware class initializes the driver before executing the _from_crawler method. So if you create a proxy parameter in SeleniumRequest you would have to initialize the driver again, because as far as I know the Selenium Driver class only accepts the proxy parameter the moment it is initialized.

Excuse me if I’m a wrong, I’m just and amateur programmer and very new to Selenium.

Cheers.