Adding Type Hints to Scrapy and its Modules

Summary

We should add Variable Annotations/ Type hints as supported in PEP 526 , Python 3.6 to Scrapy to help out existing and new contributors and developers in understanding scrapy code.

Motivation

Intellisense enabled IDES like PyCharm need Type hints to provide better experience. For new contributors to understand Scrapy comprehensively, type hints are vital.

Consider someone not that familiar with scrapy, stumbling upon scheduler’s constructor.

    def __init__(self, dupefilter, jobdir=None, dqclass=None, mqclass=None,
                 logunser=False, stats=None, pqclass=None, crawler=None):
        self.df = dupefilter
        self.dqdir = self._dqdir(jobdir)
        self.pqclass = pqclass
        self.dqclass = dqclass
        self.mqclass = mqclass
        self.logunser = logunser
        self.stats = stats
        self.crawler = crawler

[synodriver]

Well I think pydantic would be useful if type hints could be added to scrapy in future versions. It could help validate types, also make it easier to read configuration files and environment variables. After adding these settings, the code will be easier to understand. For example, Item can be defined as subclasses of pydantic.BaseModel , making it more convenient to use, and settings can be defined as subclasses of pydantic.BaseSettings

[grammy-jiang]

Considering there are already hundreds and thousands of lines code in Scrapy, Monkeytype may be a good choice to automatically add typing hint for existing code:

MonkeyType collects runtime types of function arguments and return values, and can automatically generate stub files or even add draft type annotations directly to your Python code based on the types collected at runtime.

• Let your code type-hint itself: introducing open source MonkeyType

Before each data type is added manually, I think this can give some help for typing hint.

I have tested MonkeyType, and get the following conclusions:

• MonkeyType ONLY add typing hint to the input (arguments) and output (return) of the methods which are tested in the test cases - no variables inside methods can be typing hint
• Because the input (arguments) may be mocked in the test cases, the typing hint may include this mocked objects’ types, which is not accurate and the imports is also polluted, e.g.:

...
from unittest.mock import MagicMock
...
class RobotsTxtMiddleware(object):
    DOWNLOAD_PRIORITY = 1000

    def __init__(self, crawler: Union[Crawler, MagicMock]) -> None:
	...

• The from_crawler classmethod will be added a typing hint for returning an instance of the class, Monkeytype can’t add from __future__ import annotations automatically
• Some imports will be added for typing hint, but this may cause some duplications - Monkeytype will only import from the very detail module path and ignore the imports already imported, e.g.:

...
from scrapy.http import Request  # original imports
...
from scrapy.http.request import Request  # added by MonkeyType
...

It seems Monkeytype can help for typing hint in some way, but manually checking the result is still necessary.