Derived spider with overwritten custom_settings ignores custom_settings of the base spider

If the base spider has custom_settings field and the inherited spider from the base spider also has custom_settings field then all settings from base spider are ignored by derived spider.

For example:

import scrapy 

class BaseSpider(scrapy.Spider):
    custom_settings = {'ITEM_PIPELINES': {
        'test_proj.pipelines.FirstPipeline': 300,
        'test_proj.pipelines.SecondPipeline': 301,
    }}

class DerivedSpider(BaseSpider):
    name = 'derived'
    custom_settings = {'CONCURRENT_REQUESTS': 2}

In this example DerivedSpider will have only {'CONCURRENT_REQUESTS': 2} among overwritten settings whereas item pipelines from the base spider will not be presented in the DerivedSpider settings.

So I suggest to merge custom_settings of the all base spider classes of a spider when spider settings are initialized. What do you think about it?

[kmike]

This feels a bit too magical to merge custom_settings, and it’d be a backwards incompatible change. But I can admit the current way is also clumsy. In Python 3.5 you can write it like this:

class DerivedSpider(BaseSpider):
    # ...
    custom_settings = {'CONCURRENT_REQUESTS': 2}
    custom_settings = {**BaseSpider.custom_settings, **custom_settings}

    # or just:
    custom_settings = {
        **BaseSpider.custom_settings,
        'CONCURRENT_REQUESTS': 2,
    }

[Gallaecio]

The approach in https://github.com/scrapy/scrapy/issues/3001#issuecomment-344226446 looks clean enough to me, I’m not sure this is worth any change.