Suggestion: driver.execute_script
See original GitHub issueHi,
First of all, thanks for making this, I was getting a headacke trying to use scrapy + splash. This solution is much better, the only thing I’m missing is an option to execute custom javascript code. Right now I solved this using class inheritance, but it would be great if you can integrate it into the main program.
middlewares.py
class CustomSeleniumMiddleware(SeleniumMiddleware):
def __init__(self, driver_name, driver_executable_path, driver_arguments):
super().__init__(driver_name, driver_executable_path, driver_arguments)
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
settings.py
DOWNLOADER_MIDDLEWARES = {
'xxx.middlewares.CustomSeleniumMiddleware': 800
}
Issue Analytics
- State:
- Created 5 years ago
- Comments:5 (4 by maintainers)
Top Results From Across the Web
JavaScriptExecutor in Selenium WebDriver with Example
JavaScriptExecutor is an Interface that helps to execute JavaScript through Selenium Webdriver. JavaScriptExecutor provides two methods “ ...
Read more >JavascriptExecutor in Selenium Complete Guide
JavascriptExecutor in Selenium examples explained with all details. You will learn all executeScript and executeAsyncScript methods ...
Read more >How to execute executeScript method through Java in ...
executeScript (). Executes JavaScript in the context of the currently selected frame or window. The script fragment provided will be executed ...
Read more >How to use JavaScriptExecutor in Selenium : A Complete Guide
The ExecuteScript is used only to execute JS in the subject of the presently accessed webpage in Selenium. This script runs as an...
Read more >How To Send Texts And Get Texts Using JavascriptExecutor?
Here is the sample code of command one to send texts using JavascriptExecutor. JavascriptExecutor js = (JavascriptExecutor)driver; js.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I was also looking for a way to pass the script with the request (because I was coming from
scrapy_splash
), but then figured out that the same can be done in the Spider code withWell,
I think that is probably enough for most cases, but sometimes (as I do) yo will need a different proxy setting for each spider. For example, there are some websites you can only access from a certain country and you will need different proxies for that. That is whay I have come up with the idea of making a proxy parameter in SeleniumRequest.
The problem I see is that the Middleware class initializes the driver before executing the _from_crawler method. So if you create a proxy parameter in SeleniumRequest you would have to initialize the driver again, because as far as I know the Selenium Driver class only accepts the proxy parameter the moment it is initialized.
Excuse me if I’m a wrong, I’m just and amateur programmer and very new to Selenium.
Cheers.