exceptions.AttributeError: '_SIGCHLDWaker' object has no attribute 'doWrite'

Hello,

Thank your for your fantastic project. We are facing a really hard to solve bug while running scapy inside celery task. Sometimes we get this error:

Unhandled Error
Traceback (most recent call last):
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/python/log.py", line 101, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/python/log.py", line 84, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/home/wirebot/.virtualenvs/cayzn_tracking.scraping/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 602, in _doReadOrWrite
    why = selectable.doWrite()
exceptions.AttributeError: '_SIGCHLDWaker' object has no attribute 'doWrite'

I think it’s related to a threading problem (I am not sure twisted is very thread safe) but maybe you already have an explanation ? Anyway, it seems bloated to run scrapy inside celery tasks because we need to recreate the crawler for each task, do you have any good advice on how to built a system to distribute crawls ? We are open to use scrapinghub but we have some very specific pipelines & download middlewares which prevent us to do so 😦

[chainstair]

I am still having issues at the moment. First I thought that it had to do with Python 3.8. I tried now with Python 3.7 and I still experience the same issues with Multiprocessing and ThreadPool.

Please get this fixed. 🙏

[thomaszdxsn]

happen a similar problem with my project too. And I used the multiprocessing and apscheduler to manage various spiders.

I find a answer from stackoverflow, it say can replace reactor to asyncioreactor, and I heard reactor cannot restart in one process, maybe this is solution?

after do that, no more exception