Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

KeyError on listjobs request after scrapyd restart

See original GitHub issue

Seems self.root.poller.queues is not updated on scrapyd daemon start. It leads to error on listjobs and schedule requests for all deployed projects before new scrapyd-deploy. For fixing listjobs request I’ve added self.root.poller.update_projects() before line 124 in webservice.py https://github.com/scrapy/scrapyd/blob/master/scrapyd/webservice.py#L124. I’m not sure that’s the best deal but it works.

This is the error that I got:

2017-05-25T16:21:15+0300 [_GenericHTTPChannelProtocol,0,127.0.0.1] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib64/python3.4/site-packages/twisted/web/http.py", line 1906, in allContentReceived
            req.requestReceived(command, path, version)
          File "/usr/lib64/python3.4/site-packages/twisted/web/http.py", line 771, in requestReceived
            self.process()
          File "/usr/lib64/python3.4/site-packages/twisted/web/server.py", line 190, in process
            self.render(resrc)
          File "/usr/lib64/python3.4/site-packages/twisted/web/server.py", line 241, in render
            body = resrc.render(self)
        --- <exception caught here> ---
          File "/usr/lib/python3.4/site-packages/scrapyd/webservice.py", line 21, in render
            return JsonResource.render(self, txrequest).encode('utf-8')
          File "/usr/lib/python3.4/site-packages/scrapyd/utils.py", line 20, in render
            r = resource.Resource.render(self, txrequest)
          File "/usr/lib64/python3.4/site-packages/twisted/web/resource.py", line 250, in render
            return m(request)
          File "/usr/lib/python3.4/site-packages/scrapyd/webservice.py", line 124, in render_GET
            queue = self.root.poller.queues[project]
        builtins.KeyError: 'medse'

Issue Analytics

State:
Created 6 years ago
Comments:7 (3 by maintainers)

Top GitHub Comments

1reaction

Digeniscommented, May 26, 2017

Could it be #70? What init/systemd script do you use? In which directory do you run scrapyd?

0reactions

padaliyajaycommented, Jun 3, 2020

Yes, that was the problem. Thank you. I didn’t use --rundir. Instead I set up default directory when created a user. I thought that the process start working in this directory and it worked before restarting the daemon. I’ve added WorkingDirectory to service definition now and it works as expected. Have a good day:).

I have also the same problem, can you help me to fix it. how to use --rundir