Trinity logs lots of "socket.send() raised exception" warnings

on current master @ 32cc41d5315003289b1c86b31680f2501894da06

What is wrong?

When running trinity a lot of these warnings happen.

 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.
 WARNING  07-14 03:48:12  selector_events  socket.send() raised exception.

How can it be fixed

It looks like it may related to this issue found here on stack overflow:

https://stackoverflow.com/questions/32175217/how-to-detect-write-failure-in-asyncio

However, brief experimentation with adding a await asyncio.sleep(0) to the only place in our code that calls writer.drain() didn’t fix anything…

[pipermerriam]

@carver

The transaction pool could use the new pool subscriber API from #1009 then: https://github.com/ethereum/py-evm/pull/1009/files#diff-0c5a90ea9349bdb0b8d152d6c4b24583R694 (after it’s merged)

I think this is part of the solution, however, I believe what we’re seeing is a race condition where the peer is connected at the start of the tx pool send loop, but by the time that it comes time to send to that peer they have disconnected.

[gsalgado]

However, I’m curious if your tests show that the disconnected peer is stuck in the pool, or if it is just in the process of disconnecting?

I couldn’t reproduce the socket.send() errors yesterday; those logs are from last week but they show a Peer instance is kept in the pool’s connected_nodes forever after the remote disconnects. I’ve opened #1023 for it but can’t seem to reproduce it. Can you grep your logs for running=False to see if that’s the cause of your socket.send() errors as well?