Derived spider with overwritten custom_settings ignores custom_settings of the base spider
See original GitHub issueIf the base spider has custom_settings
field and the inherited spider from the base spider also has custom_settings
field then all settings from base spider are ignored by derived spider.
For example:
import scrapy
class BaseSpider(scrapy.Spider):
custom_settings = {'ITEM_PIPELINES': {
'test_proj.pipelines.FirstPipeline': 300,
'test_proj.pipelines.SecondPipeline': 301,
}}
class DerivedSpider(BaseSpider):
name = 'derived'
custom_settings = {'CONCURRENT_REQUESTS': 2}
In this example DerivedSpider
will have only {'CONCURRENT_REQUESTS': 2}
among overwritten settings whereas item pipelines from the base spider will not be presented in the DerivedSpider
settings.
So I suggest to merge custom_settings
of the all base spider classes of a spider when spider settings are initialized. What do you think about it?
Issue Analytics
- State:
- Created 6 years ago
- Reactions:2
- Comments:5 (4 by maintainers)
Top Results From Across the Web
RGD-functionalized bioengineered spider dragline silk ...
Abstract. Spider silk fibers have remarkable mechanical properties that suggest the component proteins could be useful biopolymers for fabricating biomaterial ...
Read more >Spider Silk-Augmented Scaffolds and Adipose-Derived ... - MDPI
The experimental approach of the present study focused on the use of the biomaterial spider silk as a biocompatible and very stable carrier...
Read more >CumulusCI Documentation
CumulusCI derives the library of tasks and flows available for any project ... However, a separate task is supplied to deploy Custom Settings...
Read more >CurseForge
Some of your custom settings may get overwritten if you switch back and forth.. Changing these options may interfere with other nameplate addons...
Read more >Deltek Maconomy® 2.4.2 - Deltek Software Manager
The purpose was to establish a common base for estimating network load in a new ... Additional files with custom settings can be...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
This feels a bit too magical to merge custom_settings, and it’d be a backwards incompatible change. But I can admit the current way is also clumsy. In Python 3.5 you can write it like this:
The approach in https://github.com/scrapy/scrapy/issues/3001#issuecomment-344226446 looks clean enough to me, I’m not sure this is worth any change.