Cannot deploy spiders when importing `urlparse`

Environment:

1. macOS Sierra 10.12.3 (16D32)
2. Python 3.6 [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin] installed via brew
3. Scrapy 1.3.2
4. shub 2.5.1

Steps:

mkdir shubissue
cd shubissue
python3 -m venv .pyenv
source .pyenv/bin/activate
pip install scrapy shub

scrapy startproject myscrapy

cd myscrapy
scrapy genspider example example.com

shub deploy
# provide project ID
# set as default

Message

{"status": "ok", "spiders": 1, "project": XXXXXX, "version": "1.0"}

Change the contents of myscrapy/myscrapy/spiders/example.py to:

import scrapy
from urllib.parse import urlparse


class ExampleSpider(scrapy.Spider):
     name = "example"
     allowed_domains = ["example.com"]
     start_urls = ['http://example.com/']

     def parse(self, response):
         pass

Rerun:

shub deploy

Message:

{"status": "ok", "spiders": 0, "project": XXXXXX, "version": "1.0"}

If you create new spiders they will be ignored also.

I will try to reproduce on a linux environment.

[jdemaeyer]

Hey @rtodea, thanks for the issue!

For compatibility reasons Scrapy Cloud uses Python 2 by default, where the urllib module still had a different structure and your import fails with an ImportError. You can switch to Python 3 by specifying a corresponding stack in your scrapinghub.yml, e.g. like this:

projects:
  default:
    id: XXX_YOUR_PROJECT_ID
    stack: scrapy:1.3-py3

What’s curious is that your deploy didn’t fail with a build error but apparently the build went through just fine but dropped the spiders. This is what following your steps produced on my machine:

jakob@MosEisley ~/playground/shubissue/myscrapy % shub deploy
Packing version 1.0
Deploying to Scrapy Cloud project "43100"
Deploy log last 30 lines:
    sys.exit(list_spiders())
  File "/usr/local/lib/python2.7/dist-packages/sh_scrapy/crawl.py", line 170, in list_spiders
    _run_usercode(None, ['scrapy', 'list'], _get_apisettings)
  File "/usr/local/lib/python2.7/dist-packages/sh_scrapy/crawl.py", line 127, in _run_usercode
    _run(args, settings)
  File "/usr/local/lib/python2.7/dist-packages/sh_scrapy/crawl.py", line 87, in _run
    _run_scrapy(args, settings)
  File "/usr/local/lib/python2.7/dist-packages/sh_scrapy/crawl.py", line 95, in _run_scrapy
    execute(settings=settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 142, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 209, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 115, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 296, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 30, in from_settings
    return cls(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 21, in __init__
    for module in walk_modules(name):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/app/__main__.egg/myscrapy/spiders/example.py", line 3, in <module>
ImportError: No module named parse
{"message": "List exit code: 193", "details": null, "error": "build_error"}

{"message": "Internal build error", "status": "error"}
Deploy log location: /tmp/shub_deploy_dwkz8229.log
Error: Deploy failed: b'{"message": "Internal build error", "status": "error"}'

[redapple]

Thanks for the heads up @rubhanazeem