Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Derived spider with overwritten custom_settings ignores custom_settings of the base spider

See original GitHub issue

If the base spider has custom_settings field and the inherited spider from the base spider also has custom_settings field then all settings from base spider are ignored by derived spider.

For example:

import scrapy 

class BaseSpider(scrapy.Spider):
    custom_settings = {'ITEM_PIPELINES': {
        'test_proj.pipelines.FirstPipeline': 300,
        'test_proj.pipelines.SecondPipeline': 301,

class DerivedSpider(BaseSpider):
    name = 'derived'
    custom_settings = {'CONCURRENT_REQUESTS': 2}

In this example DerivedSpider will have only {'CONCURRENT_REQUESTS': 2} among overwritten settings whereas item pipelines from the base spider will not be presented in the DerivedSpider settings.

So I suggest to merge custom_settings of the all base spider classes of a spider when spider settings are initialized. What do you think about it?

Issue Analytics

  • State:open
  • Created 6 years ago
  • Reactions:2
  • Comments:5 (4 by maintainers)

github_iconTop GitHub Comments

kmikecommented, Nov 24, 2017

This feels a bit too magical to merge custom_settings, and it’d be a backwards incompatible change. But I can admit the current way is also clumsy. In Python 3.5 you can write it like this:

class DerivedSpider(BaseSpider):
    # ...
    custom_settings = {'CONCURRENT_REQUESTS': 2}
    custom_settings = {**BaseSpider.custom_settings, **custom_settings}

    # or just:
    custom_settings = {
Gallaeciocommented, Apr 26, 2021

The approach in looks clean enough to me, I’m not sure this is worth any change.

Read more comments on GitHub >

github_iconTop Results From Across the Web

RGD-functionalized bioengineered spider dragline silk ...
Abstract. Spider silk fibers have remarkable mechanical properties that suggest the component proteins could be useful biopolymers for fabricating biomaterial ...
Read more >
Spider Silk-Augmented Scaffolds and Adipose-Derived ... - MDPI
The experimental approach of the present study focused on the use of the biomaterial spider silk as a biocompatible and very stable carrier...
Read more >
CumulusCI Documentation
CumulusCI derives the library of tasks and flows available for any project ... However, a separate task is supplied to deploy Custom Settings...
Read more >
Some of your custom settings may get overwritten if you switch back and forth.. Changing these options may interfere with other nameplate addons...
Read more >
Deltek Maconomy® 2.4.2 - Deltek Software Manager
The purpose was to establish a common base for estimating network load in a new ... Additional files with custom settings can be...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found