Tornado adapter: passive exchange.declare scenario where close hangs

Updated bug description:

When a channel is closed due to a passive declaration of an exchange that does not exist, processing of the resulting channel closure hangs.

---

Original description (kept so the context of earlier comments is not lost):

Not sure exactly when it started happening, but lately, we’ve been getting this message, a lot, when an AMQP client tries to passively declare a number of exchanges that are not present.

Correct behaviour is that the AMQP broker closes the channel with a “not found” exception, and our code retries opening the channel. It seems pika trips up on the latter bit, confusing the AMQP broker which then responds by closing the entire connection.

We’re using the Tornado client adapter on Python 2.7.

2018-02-05 01:02:34,929 pika.connection[connection.py:1382] WARNING Disconnected from RabbitMQ at rabbitmq:5672 (504): CHANNEL_ERROR - second 'channel.open' seen
2018-02-05 01:02:35,042 pika.adapters.base_connection[base_connection.py: 212] INFO Connecting to 172.18.0.4:5672
2018-02-05 01:02:35,047 pika.credentials[credentials.py:  71] INFO Erasing stored credential values
2018-02-05 01:02:35,763 pika.adapters.base_connection[base_connection.py: 180] WARNING Socket closed when connection was open
2018-02-05 01:02:35,764 pika.adapters.base_connection[base_connection.py: 308] WARNING Connection is closed but not stopping IOLoop
2018-02-05 01:02:35,764 pika.connection[connection.py:1382] WARNING Disconnected from RabbitMQ at rabbitmq:5672 (504): CHANNEL_ERROR - second 'channel.open' seen
2018-02-05 01:02:38,288 pika.adapters.base_connection[base_connection.py: 212] INFO Connecting to 172.18.0.4:5672
2018-02-05 01:02:38,295 pika.credentials[credentials.py:  71] INFO Erasing stored credential values
2018-02-05 01:02:38,995 pika.adapters.base_connection[base_connection.py: 180] WARNING Socket closed when connection was open
2018-02-05 01:02:38,996 pika.adapters.base_connection[base_connection.py: 308] WARNING Connection is closed but not stopping IOLoop
2018-02-05 01:02:38,997 pika.connection[connection.py:1382] WARNING Disconnected from RabbitMQ at rabbitmq:5672 (504): CHANNEL_ERROR - second 'channel.open' seen

I’ll also point out that the “Connection is closed but not stopping IOLoop” is annoying; the IOLoop does not belong to pika, so it isn’t pika’s place to say whether the IOLoop should be running or stopped.

[sjlongland]

Yeah, my other code base actually opens one channel per queue / exchange, an idea I got from QAMQP. It’s a device driver which creates an exchange per device… and in the situation where we had the problem, there’s nearly 200 devices.

[sjlongland]

Okay, I have a simplified test case here:

https://gist.github.com/sjlongland/a8acc05e399ec1700d61fe11aba2f0f4

Basically, pypi releases 0.9.14 works on Python 2.7 only (not sure if it’s Python 3 compatible, this could be a known issue); 0.10.0 works on both, 0.11.2 fails on both. I’m not seeing the connection getting closed due to double-ups on channel.open like with my other code (which is proprietary sadly, otherwise I’d happily share it), but it shouldn’t hang like this either.

In our code base, it could also be a race condition, which is a hard thing to reproduce. This isn’t the exact problem I saw in the original report, but on the other hand, when someone says close, they mean it.