Error Running Scrapy through script(Py/Shell)

1)Running the Scrapy command from the Python script with the subprocess

from subprocess import call import sys

sys.path.append(“SCRAPYPROJECTPATH”) call ([“scrapy”,“crawl”,“example”])

2)Running the Shell Script from the Python Script (The call value from the above script changes to sh shellscriptname.sh)

Output

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 109, in execute
    settings = get_project_settings()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/project.py", line 60, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "/usr/local/lib/python2.7/dist-packages/scrapy/settings/__init__.py", line 108, in setmodule
    module = import_module(module)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named project.settings

[kmike]

@kgrvamsi if you’re going to built it with Flask you may find answers at http://stackoverflow.com/questions/36384286/how-to-integrate-flask-scrapy and http://stackoverflow.com/questions/32724537/building-a-restful-flask-api-for-scrapy/32784312 useful.

[kmike]

@kgrvamsi it should work in Scrapy 1.0+, including Scrapy 1.0.5. Note that Scrapy 1.0.x is no longer supported by us; upgrading to 1.1.x is a good idea 😃