callback Issues

Firstly, thanks for a wonderful project. It is really helpful in scaling web crawling without much efforts. The issue I am facing is that I am trying to add a callback function other than the parse method in Request object but on doing so, callback function is not getting called while parse() gets called everytime. Snippet.

req = Request(url, callback=self.parse_contents)
for key in response.meta.keys():
    req.meta[key] = response.meta[key]
yield req

If I move the parse_contents code within parse function, everything works fine but to do so I have to a couple of checks that I wish to avoid. So, is the callback function supposed to be only parse function or am I missing something.

[madisonb]

@rksaxena Please check the dev branch and associated docker containers for the update I just made. I created an simple spider that generated different requests with different callback methods and everything seems to work correctly.

If this solves your problem please close this!

[rksaxena]

@nguyenanhnhat : Patch the code from this commit in the crawler/crawling/distributed_scheduler.py and it should work