Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot use scrapy-splash with scrapy<1.1

See original GitHub issue

$ pip install "scrapy<1.1" scrapy-splash
Collecting scrapy<1.1
  Downloading Scrapy-1.0.6-py2-none-any.whl (291kB)
    100% |████████████████████████████████| 296kB 396kB/s
Collecting scrapy-splash
  Downloading scrapy_splash-0.7-py2.py3-none-any.whl (45kB)
    100% |████████████████████████████████| 51kB 1.7MB/s
Collecting queuelib (from scrapy<1.1)
  Downloading queuelib-1.4.2-py2.py3-none-any.whl
Collecting cssselect>=0.9 (from scrapy<1.1)
  Downloading cssselect-0.9.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): lxml in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from scrapy<1.1)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from scrapy<1.1)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from scrapy<1.1)
Collecting w3lib>=1.8.0 (from scrapy<1.1)
  Downloading w3lib-1.14.2-py2.py3-none-any.whl
Collecting Twisted>=10.0.0 (from scrapy<1.1)
  Downloading Twisted-16.2.0.tar.bz2 (2.9MB)
    100% |████████████████████████████████| 2.9MB 187kB/s
Requirement already satisfied (use --upgrade to upgrade): service-identity in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from scrapy<1.1)
Collecting zope.interface>=3.6.0 (from Twisted>=10.0.0->scrapy<1.1)
  Downloading zope.interface-4.1.3.tar.gz (141kB)
    100% |████████████████████████████████| 143kB 47kB/s
Requirement already satisfied (use --upgrade to upgrade): attrs in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from service-identity->scrapy<1.1)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from service-identity->scrapy<1.1)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages (from service-identity->scrapy<1.1)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages/setuptools-22.0.5-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy<1.1)
Building wheels for collected packages: cssselect, Twisted, zope.interface
  Running setup.py bdist_wheel for cssselect ... done
  Stored in directory: /Users/rolando/Library/Caches/pip/wheels/1b/41/70/480fa9516ccc4853a474faf7a9fb3638338fc99a9255456dd0
  Running setup.py bdist_wheel for Twisted ... done
  Stored in directory: /Users/rolando/Library/Caches/pip/wheels/fe/9d/3f/9f7b1c768889796c01929abb7cdfa2a9cdd32bae64eb7aa239
  Running setup.py bdist_wheel for zope.interface ... done
  Stored in directory: /Users/rolando/Library/Caches/pip/wheels/52/04/ad/12c971c57ca6ee5e6d77019c7a1b93105b1460d8c2db6e4ef1
Successfully built cssselect Twisted zope.interface
Installing collected packages: queuelib, cssselect, w3lib, zope.interface, Twisted, scrapy, scrapy-splash
Successfully installed Twisted-16.2.0 cssselect-0.9.1 queuelib-1.4.2 scrapy-1.0.6 scrapy-splash-0.7 w3lib-1.14.2 zope.interface-4.1.3

$ ipython

In [1]: import scrapy_splash
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-64c0fb72e7d6> in <module>()
----> 1 import scrapy_splash

/Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages/scrapy_splash/__init__.py in <module>()
      2 from __future__ import absolute_import
      3
----> 4 from .middleware import (
      5     SplashMiddleware,
      6     SplashCookiesMiddleware,

/Users/rolando/miniconda3/envs/tmp-splash2/lib/python2.7/site-packages/scrapy_splash/middleware.py in <module>()
     15 from scrapy.http.headers import Headers
     16 from scrapy import signals
---> 17 from scrapy.utils.python import to_native_str
     18
     19 from scrapy_splash.responsetypes import responsetypes

ImportError: cannot import name to_native_str

Issue Analytics

State:
Created 7 years ago
Comments:7 (3 by maintainers)

Top GitHub Comments

1reaction

phpmongodbcommented, Aug 10, 2016

Same problem in my mac i simply run bellow command and it is working fine

pip install -U Scrapy

1reaction

yoyfongcommented, Jun 15, 2016

Try to reinstall scrapy. I in ubuntu worked. pip install scrapy --upgrade

Top Results From Across the Web

scrapy-splash - PyPI

This library provides Scrapy and JavaScript integration using Splash. The license is BSD 3-clause. Installation. Install scrapy-splash using pip:

Scrapy selector not working on Splash response

The selectors work just fine using the Scrapy shell, so I'm very confused about what is not working. The only difference I can...

Scrapy Splash: How to scrape JS rendered websites (2022)

In this project, we use Scrapy Splash to scrape javascript ... 1.1 K views · 3 months ago ... How I Scrape Amazon...

FAQ — Splash 3.5 documentation - Read the Docs

Scrapy is a popular web crawling and scraping framework. For Scrapy + Splash integration use scrapy-splash library. R language¶. There is a third-party...

Installation guide — Scrapy 2.7.1 documentation

If you're using Anaconda or Miniconda, you can install the package from the conda-forge channel, which has up-to-date packages for Linux, ...