Are optional dependencies really optional?

This is probably related to #88.

It seems to me that if you want to use spidermon with Scrapy you need to use spidermon.contrib.scrapy.extensions.Spidermon. That requires at least jsonschema (imported via spidermon.python.factory) from the optional validation feature. On the other hand, the contents of the optional monitoring feature look important for any spidermon usage. Are there real use cases where one or both optional features can be skipped?

Disclaimer: I’m not familiar with optional installation features that much, as most modules I’ve used don’t use them.

[raphapassini]

Or we can fix the problem by optionally import the jsonschema. IMHO I can see Spidermon being used without the validation, but doesn’t make sense if an error raises if you don’t install it

[andrewbaxter]

FWIW I’m getting an exception since it seems like jsonschema gets imported when using the extension:

Traceback (most recent call last):
  File "/myproject/env/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/myproject/env/lib/python3.6/site-packages/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/crawler.py", line 170, in crawl
    crawler = self.create_crawler(crawler_or_spidercls)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/crawler.py", line 198, in create_crawler
    return self._create_crawler(crawler_or_spidercls)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/crawler.py", line 203, in _create_crawler
    return Crawler(spidercls, self.settings)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/crawler.py", line 55, in __init__
    self.extensions = ExtensionManager.from_crawler(self)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/myproject/env/lib/python3.6/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/myproject/env/lib/python3.6/site-packages/spidermon/contrib/scrapy/extensions.py", line 11, in <module>
    from spidermon.python import factory
  File "/myproject/env/lib/python3.6/site-packages/spidermon/python/__init__.py", line 3, in <module>
    from . import factory
  File "/myproject/env/lib/python3.6/site-packages/spidermon/python/factory.py", line 3, in <module>
    from jsonschema import validate
ModuleNotFoundError: No module named 'jsonschema'

I’m not using the item validation tools in this project, just item count/error count monitors.