Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Redirecting doesn't work in Scrapy version 1.1.1

See original GitHub issue

I run a shell:

scrapy shell

Then open the google:

fetch('https://www.google.com')

In [1]: fetch('https://www.google.com')
2016-08-12 15:55:14 [scrapy] INFO: Spider opened
2016-08-12 15:55:14 [scrapy] DEBUG: Crawled (302) <GET https://www.google.com> (referer: None)

In [2]:

Redirecting just doesn’t work, what’s the matter? appreciate if someone help.

C:\Users\...>scrapy version -v
Scrapy    : 1.1.1
lxml      : 3.6.0.0
libxml2   : 2.9.3
Twisted   : 16.1.1
Python    : 2.7.11 |Continuum Analytics, Inc.| (default, Mar  4 2016, 15:18:41) [MSC v.1500 32 bit (Intel)]
pyOpenSSL : 0.15.1 (OpenSSL 1.0.2h  3 May 2016)
Platform  : Windows-10-10.0.10586

Issue Analytics

State:
Created 7 years ago
Comments:8 (5 by maintainers)

Top GitHub Comments

1reaction

kmikecommented, Aug 12, 2016

I think it makes sense to handle redirections in scrapy shell by default.

0reactions

eliasdornelescommented, Sep 28, 2016

Bien sûr ! Created issue: https://github.com/scrapy/scrapy/issues/2290

Top Results From Across the Web

Source code for scrapy.downloadermiddlewares.redirect

Source code for scrapy.downloadermiddlewares.redirect. import logging from urllib.parse import urljoin, urlparse from w3lib.url import ...

how to handle 302 redirect in scrapy - python - Stack Overflow

3- go to another process , using callback, and then be within this process to complete all scraping work by recursive loop calling...

Scrapy Documentation - Read the Docs

Scrapy (/skrepa/) is an application framework for crawling web sites and extracting structured data which can be used.

Bug listing with status RESOLVED with resolution OBSOLETE ...

Bug:1523 - "[IDEA] Offload work by distributing trivial ebuild ... with custom FTP-port doesn't work" status:RESOLVED resolution:OBSOLETE severity:major ...

CVE-2021-41125 - Scrapy is a high-level web crawling and ...

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to...