Caching css_to_xpath()'s recently used patterns to improve efficiency
See original GitHub issueI profiled the scrapy-bench spider which uses response.css()
for extracting information.
The profiling results are here. The function css_to_xpath() takes 5% of the total time.
When response.xpath()
(profiling result) was used, the items extracted per second (benchmark result) was higher.
Hence, I’m proposing caching for the recently used patterns, so that the function takes lesser time. I’m working on a prototype for the same and will add the results for it too.
Issue Analytics
- State:
- Created 6 years ago
- Reactions:1
- Comments:9 (9 by maintainers)
Top Results From Across the Web
Five Data-Loading Patterns To Boost Web Performance
Caching partially stores your data and is not used as permanent storage. Using the cache as permanent storage is an anti-pattern.
Read more >Caching patterns - Database Caching Strategies Using Redis
Implementing this approach is straightforward and produces immediate performance gains, whether you use an application framework that encapsulates lazy caching ...
Read more >A Hitchhiker's Guide to Caching Patterns - Hazelcast
Deciding to use caching is just the first step in a long journey. The next step is to think about how your application...
Read more >Cache-Aside pattern - Azure Architecture Center
Learn how to load data on demand into a cache, which can improve performance and maintain consistency between cached data and the underlying...
Read more >Common Caching Patterns | Momento Serverless Cache
The first caching choice you need to make is on where to cache your data. In thinking about caching, we often jump to...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
What @lopuhin said! 👍
Though I believe this is the stable backport for functools: https://github.com/MiCHiLU/python-functools32 It would need to conditionally add
functools32
toinstall_requires
insetup.py
This is fixed by https://github.com/scrapy/parsel/pull/109.