missing service_identity module

When I run scrapy, I got warning:

You do not have a working installation of the service_identity module: 'No module named pyasn1_modules.rfc2459.......

However when I tried to install the module, it tell me that it has been installed:

While when scrapy job start, I got too many errors like:

What’s going on? How to fix that?

[redapple]

I think the only thing left to check is the missing pyasn1-modules, which should be installed when service-identity gets installed. pip install pyasn1-modules

Let me share a scrapy install log inside a virtualenv with Python 2 (and on Ubuntu 16.04) It gets pyasn1-modules==0.0.8

(scrapy1.1-deps) paul@plaptop:~$ mkvirtualenv scrapy1.1-deps
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/paul/.virtualenvs/scrapy1.1-deps/bin/python2
Also creating executable in /home/paul/.virtualenvs/scrapy1.1-deps/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy1.1-deps/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy1.1-deps/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy1.1-deps/bin/preactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy1.1-deps/bin/postactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy1.1-deps/bin/get_env_details
(scrapy1.1-deps) paul@plaptop:~$ pip install scrapy --no-cache-dir
Collecting scrapy
  Downloading Scrapy-1.1.2-py2.py3-none-any.whl (295kB)
    100% |████████████████████████████████| 296kB 1.6MB/s 
Collecting queuelib (from scrapy)
  Downloading queuelib-1.4.2-py2.py3-none-any.whl
Collecting pyOpenSSL (from scrapy)
  Downloading pyOpenSSL-16.1.0-py2.py3-none-any.whl (43kB)
    100% |████████████████████████████████| 51kB 2.9MB/s 
Collecting Twisted>=10.0.0 (from scrapy)
  Downloading Twisted-16.3.2.tar.bz2 (2.9MB)
    100% |████████████████████████████████| 2.9MB 1.8MB/s 
Collecting w3lib>=1.14.2 (from scrapy)
  Downloading w3lib-1.15.0-py2.py3-none-any.whl
Collecting six>=1.5.2 (from scrapy)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from scrapy)
  Downloading cssselect-0.9.2-py2.py3-none-any.whl
Collecting lxml (from scrapy)
  Downloading lxml-3.6.4-cp27-cp27mu-manylinux1_x86_64.whl (4.2MB)
    100% |████████████████████████████████| 4.2MB 2.0MB/s 
Collecting parsel>=0.9.3 (from scrapy)
  Downloading parsel-1.0.3-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from scrapy)
  Downloading PyDispatcher-2.0.5.tar.gz
Collecting cryptography>=1.3.4 (from pyOpenSSL->scrapy)
  Downloading cryptography-1.4.tar.gz (399kB)
    100% |████████████████████████████████| 409kB 2.0MB/s 
Collecting zope.interface>=3.6.0 (from Twisted>=10.0.0->scrapy)
  Downloading zope.interface-4.2.0.tar.gz (146kB)
    100% |████████████████████████████████| 153kB 2.2MB/s 
Collecting pyasn1 (from service-identity->scrapy)
  Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting pyasn1-modules (from service-identity->scrapy)
  Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting attrs (from service-identity->scrapy)
  Downloading attrs-16.0.0-py2.py3-none-any.whl
Collecting idna>=2.0 (from cryptography>=1.3.4->pyOpenSSL->scrapy)
  Downloading idna-2.1-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 2.9MB/s 
Requirement already satisfied (use --upgrade to upgrade): setuptools>=11.3 in ./.virtualenvs/scrapy1.1-deps/lib/python2.7/site-packages (from cryptography>=1.3.4->pyOpenSSL->scrapy)
Collecting enum34 (from cryptography>=1.3.4->pyOpenSSL->scrapy)
  Downloading enum34-1.1.6-py2-none-any.whl
Collecting ipaddress (from cryptography>=1.3.4->pyOpenSSL->scrapy)
  Downloading ipaddress-1.0.16-py27-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.3.4->pyOpenSSL->scrapy)
  Downloading cffi-1.7.0-cp27-cp27mu-manylinux1_x86_64.whl (383kB)
    100% |████████████████████████████████| 389kB 2.0MB/s 
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3.4->pyOpenSSL->scrapy)
  Downloading pycparser-2.14.tar.gz (223kB)
    100% |████████████████████████████████| 225kB 1.9MB/s 
Installing collected packages: queuelib, idna, pyasn1, six, enum34, ipaddress, pycparser, cffi, cryptography, pyOpenSSL, zope.interface, Twisted, w3lib, pyasn1-modules, attrs, service-identity, cssselect, lxml, parsel, PyDispatcher, scrapy
  Running setup.py install for pycparser ... done
  Running setup.py install for cryptography ... done
  Running setup.py install for zope.interface ... done
  Running setup.py install for Twisted ... done
  Running setup.py install for PyDispatcher ... done
Successfully installed PyDispatcher-2.0.5 Twisted-16.3.2 attrs-16.0.0 cffi-1.7.0 cryptography-1.4 cssselect-0.9.2 enum34-1.1.6 idna-2.1 ipaddress-1.0.16 lxml-3.6.4 parsel-1.0.3 pyOpenSSL-16.1.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.14 queuelib-1.4.2 scrapy-1.1.2 service-identity-16.0.0 six-1.10.0 w3lib-1.15.0 zope.interface-4.2.0
(scrapy1.1-deps) paul@plaptop:~$ pip freeze
attrs==16.0.0
cffi==1.7.0
cryptography==1.4
cssselect==0.9.2
enum34==1.1.6
idna==2.1
ipaddress==1.0.16
lxml==3.6.4
parsel==1.0.3
pkg-resources==0.0.0
pyasn1==0.1.9
pyasn1-modules==0.0.8
pycparser==2.14
PyDispatcher==2.0.5
pyOpenSSL==16.1.0
queuelib==1.4.2
Scrapy==1.1.2
service-identity==16.0.0
six==1.10.0
Twisted==16.3.2
w3lib==1.15.0
zope.interface==4.2.0
(scrapy1.1-deps) paul@plaptop:~$ scrapy version -v
Scrapy    : 1.1.2
lxml      : 3.6.4.0
libxml2   : 2.9.4
Twisted   : 16.3.2
Python    : 2.7.12 (default, Jul  1 2016, 15:12:24) - [GCC 5.4.0 20160609]
pyOpenSSL : 16.1.0 (OpenSSL 1.0.2g-fips  1 Mar 2016)
Platform  : Linux-4.4.0-34-generic-x86_64-with-Ubuntu-16.04-xenial
(scrapy1.1-deps) paul@plaptop:~$ 

[redapple]

You’re welcome. I’m still not sure why you didn’t have pyasn1-modules in the first place. In the general case, I would recommend that you use virtualenvs for any python project, including scrapy.