Add note that "allowed_domains" should be a list of domains, not URLs

(just logging the issue before I forget)

It may seem obvious by the name of the attribute that allowed_domains is about domain names, but it’s not uncommon for scrapy users to make the mistake of doing allowed_domains = ['http://www.example.com']

I believe it is worth adding a note in http://doc.scrapy.org/en/latest/topics/spiders.html?#scrapy.spiders.Spider.allowed_domains

[redapple]

+1 to issue a warning.

I’m less sure about inferring domain, for example for http://www.example.com, should it infer example.com or www.example.com?

[eliasdorneles]

What if instead of simply documenting, Scrapy detect this case and issues a warning?

Even better, it could extract the domain from the URL and use that, while issuing a warning like:

logging.warn("allowed_domains accepts only domains, not URLs. Using allowed_domains = %r" % effective_allowed_domains)