FileNotFoundError: [Errno 2] No such file or directory: 'webcam/webcam/settings.py.tmpl'

Description

when I issue the following command

scrapy startproject webcam

I get the error message:

vagrant@vagrant:/vagrant$ scrapy startproject webcam
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/local/lib/python3.6/dist-packages/scrapy/cmdline.py", line 146, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/cmdline.py", line 100, in _run_print_help
    func(*a, **kw)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/cmdline.py", line 154, in _run_command
    cmd.run(args, opts)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/commands/startproject.py", line 109, in run
    ProjectName=string_camelcase(project_name))
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/template.py", line 9, in render_templatefile
    with open(path, 'rb') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'webcam/webcam/settings.py.tmpl'

Steps to Reproduce

1. Create a fresh vagrant machine with ubuntu 18.04 on a mac 10.14.6
2. Install scrapy per these instruction with all the dependancies: https://docs.scrapy.org/en/latest/intro/install.html
3. issue the command scrapy startproject webcam

Expected behavior: I expect it to create the project structure without errors.

Actual behavior: It creates the folder structure as expected BUT produces the error message in the description.

Reproduces how often: This happens on a fresh install. on a new Ubuntu 18.04 box. I tried to uninstall and reinstall scrapy. It produces the same error.

Versions

vagrant@vagrant:/vagrant$ pip show scrapy
Name: Scrapy
Version: 1.8.0
Summary: A high-level Web Crawling and Web Scraping framework
Home-page: https://scrapy.org
Author: Scrapy developers
Author-email: None
License: BSD
Location: /home/vagrant/.local/lib/python2.7/site-packages
Requires: pyOpenSSL, protego, service-identity, cryptography, Twisted, lxml, six, parsel, cssselect, w3lib, zope.interface, PyDispatcher, queuelib

Additional context

This issue was identified here: https://github.com/scrapy/scrapy/issues/4019. I provided the same information above but that issue is closed. I’m opening a new ticket here.

[kumaxim]

I catch the same issue recently in my fresh Vagrant enviroment.

I have Ubuntu 20.04.3 LTS on host and the same OS in guest. Here is very basic Vagrantfile that allow to reproduce the error:

Vagrant.configure("2") do |config|
  config.vm.define "ubuntu-desktop" do |c|
    c.vm.box = "peru/ubuntu-20.04-desktop-amd64"
    c.vm.box_version = "20210901.01"
    c.vm.box_check_update = false
  end

  config.vm.synced_folder "app/", "/home/vagrant/app",
    create: true,
    owner: "vagrant",
    group: "vagrant"

  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
  end

  config.vm.provision "shell", inline: <<-SHELL
    apt-get install -y curl python3-dev python3-pip python3-venv python3-wheel
  SHELL
end

As @elacuesta supposed this is Vagrant-specific issue. He was not far from the truth. The problem is the filesystem that Vagrant use for VirtualBox machines by default. This is vboxsf and it’s very slow in operation.

If you’ll create new Virtualbox machine based on Vagrantfile below, login into(vagrant ssh) and perform the following commands:

python3 -m venv /home/vagrant/app/venv
source /home/vagrant/app/venv/bin/activate
pip3 install wheel
pip3 install selenium scrapy

scrapy startproject webcam

…you’ll get the error that @ngufyi listed in the first message.

The solution would be change filesystem for shared folders. The simplies way is using nfs instead of vboxfs.

First you should prepare both host and guest OS.

Install NFS packages on host: # apt-get install nfs-kernel-server nfs-common

…and this one into guest OS: # apt-get install nfs-common

After that you must change Vagrantfile in the following way:

Vagrant.configure("2") do |config|
   ...
  # Create a private network, which allows host-only access to the machine
  config.vm.network "private_network", type: "dhcp"

  # Change sync_folders options
  config.vm.synced_folder "app/", "/home/vagrant/app",
    create: true,
    type: "nfs"
end

Reload machine vagrant reload and confirm to export setting into /etc/exports when Vagrant ask you to confirm privileges. Login into machine again vagrant ssh and run scrapy startproject webcam into you Python’s virtual enviroment.

Now everything will be fine.

P.S.: It was not easy to catch this error. I use PyCharm. Initially I set breakpoint in 9 line of render_templatefile function within /home/vagrant/app/venv/lib/python3.8/dist-packages/scrapy/utils/template.py file and run scrapy startproject webcam with pauses on this liine. I surprised when command will be completed successfully in debug mode. However, this gave me idea to put my attention into right field.

I not sure whether Scapy’s developers should do anything. I leave it to their discretion.

[0x3639]

Understood and thank you for looking at this. I moved to anaconda and that environment is working well. I’ll keep trouble shooting this on my end and post a solution if I ever figure it out.