Running scrapy shell against a local file

Before Scrapy 1.0, I could execute:

 scrapy shell index.html

In >=1.0, it started to throw ValueError: Missing scheme in request url: index.html:

$ scrapy shell index.html
2015-10-12 15:32:59 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2015-10-12 15:32:59 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-10-12 15:32:59 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0}
Traceback (most recent call last):
  File "/Users/user/.virtualenvs/so/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/cmdline.py", line 143, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/cmdline.py", line 150, in _run_command
    cmd.run(args, opts)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/commands/shell.py", line 50, in run
    spidercls = spidercls_for_request(spider_loader, Request(url),
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 24, in __init__
    self._set_url(url)
  File "/Users/user/.virtualenvs/so/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 59, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: index.html 

As a workaround, I’ve used the “file” protocol providing the full path to a file:

$ scrapy shell file:////absolute/path/to/index.html

From a comment to the relevant SO topic http://stackoverflow.com/questions/33088877/scrapy-shell-against-a-local-file, we can see that the relevant change was introduced here.

Would it be possible and would it make sense to bring the previous behavior back so that we can execute the shell against a local file as easy as scrapy shell filename?

Thanks!

[tarunteckedge]

Good to see it works in scrapy shell “./path/to/file/hello.html”

But same url doesn’t work in spider. Anyone can help on that or can confirm this is not supposed to work there?

[alecxe]

Thanks everyone for the time! Glad to see it being a part of 1.1.