Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

should raise `AttributeError: Response.meta not available` when access response.meta in `process_resonse`?

See original GitHub issue

should raise AttributeError: Response.meta not available when access response.meta in process_resonse?

2018-12-03 23:12:47 [scrapy.core.scraper] ERROR: Error downloading <GET https://httpbin.org/status/200>
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "C:\Program Files\Python37\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "C:\Program Files\Python37\lib\site-packages\twisted\internet\defer.py", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 https://httpbin.org/status/200>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\scrapy\http\response\__init__.py", line 30, in meta
    return self.request.meta
AttributeError: 'NoneType' object has no attribute 'meta'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "C:\Program Files\Python37\lib\site-packages\scrapy\core\downloader\middleware.py", line 53, in process_response
    spider=spider)
  File "C:\Users\user\Desktop\try_scrapy_Snippet.py", line 11, in process_response
    print('in response:', response.url, response.meta)
  File "C:\Program Files\Python37\lib\site-packages\scrapy\http\response\__init__.py", line 33, in meta
    "Response.meta not available, this response "
AttributeError: Response.meta not available, this response is not tied to any request

Is this right behavior? process_response can’t access response.meta???

Issue Analytics

State:
Created 5 years ago
Comments:6 (6 by maintainers)

Top GitHub Comments

3reactions

elacuestacommented, Dec 10, 2018

It is not, it’s an implementation detail. Downloader middlewares sit between the downloader and the engine (see https://doc.scrapy.org/en/latest/topics/architecture.html). As the second link I posted shows, the request gets bound to the response in the engine, and by the time your process_response method is executed the response hasn’t reached the engine yet.

2reactions

NewUserHacommented, Dec 10, 2018

you’re right! Thanks!

Top Results From Across the Web

Scrapy crawlera bug - Stack Overflow

I think scrapy_crawlera should access meta differently ... AttributeError: Response.meta not available, this response is not tied to any ...

Requests and Responses — Scrapy 2.7.1 documentation

Scrapy uses Request and Response objects for crawling web sites. ... and can also be accessed, in your spider, from the response.meta ......

Python process response - ProgramCreek.com

This page shows Python code examples for process response. ... is None: return response; # No CAPTCHA is present elif request.meta.get(RETRY_KEY, self.

Release Scrapy developers

At this point Python 2.7 and pip package manager must be working, ... This method is responsible for parsing the response data and ......

Scrapy 2.6.1 documentation

As mentioned above, the received Response object will contain the text of the link that produced the Request in its meta dictionary (under...