`schematics` required but not installed by default

I installed with pip install spidermon, got 1.9.0.

So I rechecked the installation docs just in case, but trying to use jsonschema validation I get:

2019-04-05 19:51:42 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/crawler.py", line 80, in crawl
    self.engine = self._create_engine()
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/crawler.py", line 105, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/core/engine.py", line 70, in __init__
    self.scraper = Scraper(crawler)
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/core/scraper.py", line 71, in __init__
    self.itemproc = itemproc_cls.from_crawler(crawler)
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/middleware.py", line 53, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/spidermon/contrib/scrapy/pipelines.py", line 13, in <module>
    from spidermon.contrib.validation import SchematicsValidator, JSONSchemaValidator
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/spidermon/contrib/validation/__init__.py", line 1, in <module>
    from .schematics.validator import SchematicsValidator
  File "/home/andrew/scraping/temp/30/env/lib/python3.6/site-packages/spidermon/contrib/validation/schematics/validator.py", line 4, in <module>
    import schematics
ModuleNotFoundError: No module named 'schematics'

Installing schematics fixed the issue.

It’s possible I have my project misconfigured (#139). My settings look like this:

SPIDERMON_ENABLED = True
EXTENSIONS = {
    'spidermon.contrib.scrapy.extensions.Spidermon': 500,
}
ITEM_PIPELINES = {
    'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline': 800,
}
SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True
SPIDERMON_VALIDATION_SCHEMAS = [
    'schema.json',
]

[andrewbaxter]

I agree with all the points here - I think what I’d like to highlight is:

the documentation clearly mentions to install schematics if you plan to use it

In my case I do not plan to use it, and it’s not documented that schematics is required if you want to use jsonschema.

If you need item validation, then pip install spidermon[validation] is the way to go

This sounds like the proper solution, but https://spidermon.readthedocs.io/en/latest/item-validation.html doesn’t mention spidermon[validation] - I would assert that people who decide to add validation later are unlikely to go and re-read the installation docs. And in my case I’m not sure I’ve ever read that page, I actually read these installation instructions which don’t mention validation.

–

Since #88 and #89 came up again, I would add my vote for batteries included - this is the only package I’ve ever used that uses the package[opt] notation. Perhaps it would be less confusing though if the validation stuff were broken off, along with dependencies, into a separate spidermon-validation package? Then it would be clear that a separate package is required because you’d get an error doing import spidermon_validation.whatever rather than a less instructional error about schematics later.

[rennerocha]

There is a plan to refactor ItemValidationPipeline to remove this requirement. After that you will only be asked to install schematics if your are using it (the same for JSON Schema). For now, you really need to install it to be able to use spidermon.