Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

String value for order of Scrapy component

See original GitHub issue

If Scrapy component order is defined as a string, it leads to undefined behaviour on Python 2 and to the following errors on Python 3:

File "/usr/local/lib/python3.5/site-packages/scrapy/middleware.py", line 58, in from_crawler
 return cls.from_settings(crawler.settings, crawler)
File "/usr/local/lib/python3.5/site-packages/scrapy/middleware.py", line 29, in from_settings
 mwlist = cls._get_mwlist_from_settings(settings)
File "/usr/local/lib/python3.5/site-packages/scrapy/core/spidermw.py", line 21, in _get_mwlist_from_settings
 return build_component_list(settings.getwithbase('SPIDER_MIDDLEWARES'))
File "/usr/local/lib/python3.5/site-packages/scrapy/utils/conf.py", line 47, in build_component_list
 return [k for k, v in sorted(six.iteritems(compdict), key=itemgetter(1))]
builtins.TypeError: unorderable types: str() < int()

My guess that 1) order of a Scrapy component should be stated as of integer type (or None) and there should be a check somewhere, 2) or the sorting logic should be fixed.

Issue Analytics

State:
Created 7 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

1reaction

elacuestacommented, Dec 23, 2016

PR created ☝️ Thanks for your comment Viktor 😃

0reactions

vshlapakovcommented, Dec 21, 2016

@elacuesta Makes sense to me, that would be nice, thanks. In my case I encountered with a string with an integer inside as order, so it should be covered by the similar changes. Btw there’s no need to strip value before converting to int, but that’s nitpicking 😃

Top Results From Across the Web

Requests and Responses — Scrapy 2.7.1 documentation

The dict values can be strings (for single valued headers) or lists (for multi-valued headers). If None is passed as value, the HTTP...

Order a json by field using scrapy - python - Stack Overflow

seem not to be so simple. I've studied every single component (middlewares, pipelines, exporters, signals, etc...) but no one seems useful for ...

Scrapy Beginners Series Part 2 - Cleaning & Processing Data

Python Scrapy Playbook - Scrapy Beginner Series Part 2: Cleaning ... The integer values you assign to classes in this setting determine the...

How to use Scrapy Items? - GeeksforGeeks

It consists of key-value pair, of Field objects used, ... Scrapy provides us, with Selectors, to “select” parts of the webpage, desired.

An Introduction | Python Scrapy Tutorial - Great Learning

get_value(value,*processors,**kw) – given value is processed by the processors. get_xpath(xpath,*processors,**kw) – xpath will extract list of Unicode strings ...