question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to get request/response meta info when errback is called?

See original GitHub issue

I wonder if it’s possible to get the meta and response.url when errback is called. Here is my code:

class CdDvdSpider(scrapy.Spider):
    ...
    ...

    def make_requests_from_url(self, url):
        return Request(url, dont_filter=True, meta={'foo': 'foo'},
                       callback=self.parse, errback=self.error_handler)

    def error_handler(self, response):    # FIXME: argument is not response
        # print response.url
        # print response.meta['foo']
        pass

What I got is:

exceptions.AttributeError: Failure instance has no attribute 'meta'

I found a related post here on SO. It seems the latest Scrapy (0.24.4 as of this post) still doesn’t provide relevant API. If this is true, is it possible to add this feature?

Issue Analytics

  • State:closed
  • Created 9 years ago
  • Comments:8 (2 by maintainers)

github_iconTop GitHub Comments

21reactions
baby5commented, Nov 12, 2018

in 1.5.1 you can get meta in this way

def errback(self, failure):
    print(failure.request.meta)

this is not in doc

7reactions
vionemccommented, Oct 20, 2015

The solution is use response.request.meta

Read more comments on GitHub >

github_iconTop Results From Across the Web

Requests and Responses — Scrapy 2.7.1 documentation
Note that if exceptions are raised during processing, errback is called instead. method (str) – the HTTP method of this request.
Read more >
scrapy - Trigger errback when process_exception() is called in ...
I didn't find it in the docs, but looking at the source I find DownloaderMiddleware.process_exception() can return twisted.python.failure.
Read more >
Requests and Responses - Scrapy documentation
Note that if exceptions are raised during processing, errback is called instead. method (string) – the HTTP method of this request. Defaults to...
Read more >
Scrapy - Requests and Responses - Tutorialspoint
Request(url[, callback, method = 'GET', headers, body, cookies, meta, ... The errback is a callable function to be called when an exception while...
Read more >
Scrapy学习-10-Request&Response对象- 前路~ - 博客园
__name__) assert callback or not errback, "Cannot use errback ... Defaults to 'GET'. meta (dict) – the initial values for the Request.meta ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found