Deadlock publishing a message after start/stop the rabbitmq service

The issue

I’m having a problem with the latest version of masstransit while publishing to the queue and starting/stopping the RabbitMQ service.

So, if I start the bus and, before I send any message to the queue, stop the rabbitmq service and try to publish the first message it will fail (of course, the service is down). Then, I will start the service again and try to publish again and I will get a deadlock.

I created a unit testing to reproduce this behavior, but I think you will need to run visual studio as administrator to be able to start and stop the service.

Unit test

[Test]
public async Task Should_not_block_when_rabbitmqservices_goes_down()
{
    var rabbitMqService = new ServiceController("RabbitMQ");
    if(rabbitMqService.Status != ServiceControllerStatus.Running)
    {
        rabbitMqService.Start();
    }

    var busControl = Bus.Factory.CreateUsingRabbitMq(x =>
    {
        x.Host(new Uri("rabbitmq://localhost/"), h =>
        {
            h.Username("guest");
            h.Password("guest");
        });
    });

    busControl.Start();
    rabbitMqService.Stop();

    int index = 0;
    while (index < 3)
    {
        Debug.WriteLine("Go for the {0} round...", index + 1);
        try
        {
            //await busControl.Publish<object>(new { Id = 1, Name = "Abc" });
            Debug.WriteLine("Publishing");
            TaskUtil.Await(() => busControl.Publish<object>(new { Id = 1, Name = "Abc" }));
        }
        catch (Exception)
        {
            Debug.WriteLine("Error, starting the rabbitmq again...");
            rabbitMqService.Start();
        }

        index++;
    }

    Assert.IsTrue(true);
}

await busControl.Publish<object>(new { Id = 1, Name = "Abc" });

This code is commented but I did try with it also with the same result.

Log

Go for the 1 round...
2015-11-19 17:33:13,560 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - Consumer Model Shutdown (amq.ctag-Q4Lafwjtvmyv8hKmYAsc1g), Concurrent Peak: 0, 320-CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-11-19 17:33:13,561 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqConsumerFilter - Consumer amq.ctag-Q4Lafwjtvmyv8hKmYAsc1g: 0 received, 0 concurrent
2015-11-19 17:33:13,564 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connecting: guest@localhost:5672/
2015-11-19 17:33:15,570 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - The connection usage threw an exception
RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:5672
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
   at RabbitMQ.Client.Impl.SocketFrameHandler.Connect(TcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.Impl.SocketFrameHandler.Connect(TcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout)
   at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, Int32 timeout)
   at RabbitMQ.Client.Framing.Impl.ProtocolBase.CreateFrameHandler(AmqpTcpEndpoint endpoint, Func`2 socketFactory, Int32 timeout)
   at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler()
   at RabbitMQ.Client.ConnectionFactory.CreateConnection()
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnection()
   at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.<SendUsingNewConnection>d__13.MoveNext() in C:\Users\bruno.costa\Documents\GitHub\MassTransit\src\MassTransit.RabbitMqTransport\Integration\RabbitMqConnectionCache.cs:line 94
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.ConnectionScope.<Attach>d__5.MoveNext() in C:\Users\bruno.costa\Documents\GitHub\MassTransit\src\MassTransit.RabbitMqTransport\Integration\RabbitMqConnectionCache.cs:line 213
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache.<SendUsingExistingConnection>d__14.MoveNext() in C:\Users\bruno.costa\Documents\GitHub\MassTransit\src\MassTransit.RabbitMqTransport\Integration\RabbitMqConnectionCache.cs:line 159
Error, starting the rabbitmq again...
Go for the 2 round...
2015-11-19 17:33:15,582 [AMQP Connection amqp://localhost:5672] ERROR MassTransit.RabbitMqTransport.RabbitMqReceiveTransport - RabbitMQ connection failed: Connect failed: guest@localhost:5672/
2015-11-19 17:33:15,583 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connecting: guest@localhost:5672/
2015-11-19 17:33:17,583 [AMQP Connection amqp://localhost:5672] ERROR MassTransit.RabbitMqTransport.RabbitMqReceiveTransport - RabbitMQ connection failed: Connect failed: guest@localhost:5672/
2015-11-19 17:33:17,583 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connecting: guest@localhost:5672/
2015-11-19 17:33:18,086 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Connected: guest@localhost:5672/ (address: [::1]:5672, local: [::1]:58007
2015-11-19 17:33:18,086 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Integration.RabbitMqConnectionCache - Using new connection: guest@localhost:5672/
2015-11-19 17:33:18,086 [AMQP Connection amqp://localhost:5672] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - Consumer Model Shutdown (amq.ctag-Q4Lafwjtvmyv8hKmYAsc1g), Concurrent Peak: 0, 320-CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2015-11-19 17:33:18,088 [9] DEBUG MassTransit.RabbitMqTransport.Pipeline.PrepareReceiveQueueFilter - Queue: bus-FFPTN006-TE.ProcessHost.Managed-bmeyyygqwgsx4jn6bdjxnbhync (auto-delete)
2015-11-19 17:33:18,090 [10] DEBUG MassTransit.RabbitMqTransport.Pipeline.RabbitMqBasicConsumer - ConsumerOk: rabbitmq://localhost:5672/bus-FFPTN006-TE.ProcessHost.Managed-bmeyyygqwgsx4jn6bdjxnbhync?durable=false&autodelete=true&prefetch=16 - amq.ctag-CeyDSRWhPfFWwxMaQv8fiQ

[phatboyg]

This was subtle and nasty, but found and fixed! Thanks!

[phatboyg]

Okay now I’m seeing it by changing my test to match yours.