Scrapy mail.send error

Hi, I’m new to scrapy and I want to send some emails after the spider closed. But I got some errors, anyone know ? I’m using python2.7 and scrapy 1.5.1. Here are my codes:

class AlertSpider(scrapy.Spider):
    name = "alert"
    start_urls = ['http://www.test.com']
    mails = []

    def parse(self, response):
        # Do something work

    @classmethod
    def from_crawler(cls, crawler):
        spider = cls()
        crawler.signals.connect(spider.spider_closed, signals.spider_closed)
        return spider

    def spider_closed(self, spider):
        settings = get_project_settings()
        mailer = MailSender.from_settings(settings)
       # first e-mail
        mailer.send(to=["[e-mail]"], subject='subject1', body='body1')
       # second e-mail
        return mailer.send(to=["[e-mail]"], subject='subject2', body='body2')

I want to send two e-mails after the spider close, but I get below errors: (By the way, there is no problem if I just send one e-mail)

File "C:\Software\Python27\lib\site-packages\twisted\internet\selectreactor.py", line 149, in _doReadOrWrite why = getattr(selectable, method)() File "C:\Software\Python27\lib\site-packages\twisted\internet\tcp.py", line 243, in doRead return self._dataReceived(data) File "C:\Software\Python27\lib\site-packages\twisted\internet\tcp.py", line 249, in _dataReceived rval = self.protocol.dataReceived(data) File "C:\Software\Python27\lib\site-packages\twisted\protocols\tls.py", line 330, in dataReceived self._flushReceiveBIO() File "C:\Software\Python27\lib\site-packages\twisted\protocols\tls.py", line 300, in _flushReceiveBIO self._flushSendBIO() File "C:\Software\Python27\lib\site-packages\twisted\protocols\tls.py", line 252, in _flushSendBIO bytes = self._tlsConnection.bio_read(2 ** 15) exceptions.AttributeError: 'NoneType' object has no attribute 'bio_read'

It seems to the twisted doesn’t close the io, but I don’t find any close method in MailSender class, so anyone have met this error?

[111qqz]

Same problem here. I try to send emails in the “close_spider” method of the pipeline class , because I have serveral spiders and I don’t want to add the “sending email” code serveral times. After I repalce “mailer.send(…)” with “return mailer.send(…)”,the problem disappered

[iveney]

I have an email pipeline that sends email during process_item and have the error 'NoneType' object has no attribute 'bio_read', e.g., something like:

def process_item(self, item, spider):
  if meets_criteria:
    mailer.send(...)
  return item

Changing the function to async and use await seems to solve it for me, as mailer.send returns the deferred object.

async def process_item(self, item, spider):
  if meets_criteria:
    await mailer.send(...)
  return item

Not sure if this is the right way to solve but it seems working for me.