Possible import name clash between project directory and installed *.py executables

Python’s import system has one weird aspect that make it vulnerable to import name clashes between modules in the working directory and in the directory of the main entry point. This affects Scrapy’s import mechanism.

Minimal example:

touch $(dirname `which scrapy`)/myproject.py   # Could be some file from another program

scrapy startproject myproject
cd myproject
scrapy genspider example example.com
scrapy crawl example

… fails with: ModuleNotFoundError: No module named 'myproject.settings'; 'myproject' is not a package

Possible solution

Just remove the problematic folder from sys.path, i.e. add sys.path.remove(os.path.dirname(__file__)) to the Scrapy executable (usually /usr/bin/scrapy or /usr/local/bin/scrapy).

[Gallaecio]

I had forgotten about it, but I think #3669 may be tackling this issue (on the documentation level).

[wRAR]

I definitively see it as a bug in Scrapy when a specific name of the project folder can result in a crash.

Can you please explain how is it a bug in Scrapy specifically and not just a common Python problem with having modules with the same name in different sys.path components?

I don’t see an argument against modifying sys.path in general ­- it is the common method to adjust imports paths.

Sure, it’s a common part of solutions to path-specific problems, but there is no single thing you can or should do to prevent conflicts. What you are proposing can only solve some specific problems you have and not the general one.