ActiveMQ: connection fails with 'channel was inactive for too long' and stays failed

Is this a bug report?

Yes.

Can you also reproduce the problem with the latest version?

Probably; the issue happened for the first time today with 5.0.2, but I don’t see any changes in 5.1 that seemed relevant to this. The issue happens after a long time so it’s difficult to test.

Environment

1. Operating system: IIS 10.0 running on 64bit Windows Server 2016, Amazon Beanstalk AMI 1.2.0
2. Visual Studio version: Visual Studio 2017 Community 15.6.6 (on dev PC)
3. Dotnet version: .NET Framework 4.7

Steps to Reproduce

This happened for the first time today, so not sure how reproducible it is.

1. Launch one or more servers with a connection to Amazon MQ:

   IBusControl bus = Bus.Factory.CreateUsingActiveMq(c =>
   {
       var host = c.Host("b-***-1.mq.us-east-1.amazonaws.com", 61617, h =>
       {
           h.Username("service-bus");
           h.Password("***");
           h.UseSsl();
       });
       c.ReceiveEndpoint(host, "gd-publishers", endpoint => ...);
   });
   

2. Leave them running for a few days until the connection fails.

Expected Behavior

The connection is either kept alive as long as the servers are using it, or MassTransit detects when it fails and opens a new connection.

Actual Behavior

Using MassTransit 5.0.2, the connection to Amazon MQ failed on all servers after a few days with the error below. The connection stayed in an invalid state (reporting that error continuously) until the servers were manually restarted.

[Apache.NMS.ActiveMQ.IOException: Channel was inactive for too long: ssl://b-***-1.mq.us-east-1.amazonaws.com:61617/?wireFormat.tightEncodingEnabled=true&nms.AsyncSend=true]
   at Apache.NMS.ActiveMQ.Connection.Oneway(Command command) in c:\dev\NMS.ActiveMQ\src\main\csharp\Connection.cs:line 907
   at Apache.NMS.ActiveMQ.Session..ctor(Connection connection, SessionId sessionId, AcknowledgementMode acknowledgementMode) in c:\dev\NMS.ActiveMQ\src\main\csharp\Session.cs:line 72
   at Apache.NMS.ActiveMQ.Connection.CreateActiveMQSession(AcknowledgementMode ackMode) in c:\dev\NMS.ActiveMQ\src\main\csharp\Connection.cs:line 610
   at Apache.NMS.ActiveMQ.Connection.CreateSession() in c:\dev\NMS.ActiveMQ\src\main\csharp\Connection.cs:line 597
   at MassTransit.ActiveMqTransport.Contexts.ActiveMqConnectionContext.<CreateSession>b__15_0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MassTransit.ActiveMqTransport.Transport.SessionContextFactory.<>c__DisplayClass7_0.<<CreateSession>b__0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MassTransit.ActiveMqTransport.Transport.SessionContextFactory.<CreateSharedSession>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at GreenPipes.Agents.PipeContextSupervisor`1.<GreenPipes-IPipeContextSource<TContext>-Send>d__8.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at GreenPipes.Agents.PipeContextSupervisor`1.<GreenPipes-IPipeContextSource<TContext>-Send>d__8.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at GreenPipes.Agents.PipeContextSupervisor`1.<GreenPipes-IPipeContextSource<TContext>-Send>d__8.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MassTransit.ActiveMqTransport.Transport.ActiveMqSendTransport.<MassTransit-Transports-ISendTransport-Send>d__6`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MassTransit.Transports.PublishEndpoint.<Publish>d__16`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at MassTransit.Transports.PublishEndpoint.<Publish>d__16`1.MoveNext()

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at GuestDriven.Scheduler.SchedulerRunner.RecurringMessageJob`1.<Execute>d__0.MoveNext() in C:\source\_GuestDriven\Platform\GuestDriven.Scheduler\SchedulerRunner.cs:line 186

Reproducible Demo

This is difficult to test since it happened after several days. Based on my understanding of the issue, it should happen with the same code used in the MassTransit unit tests if the connection is kept open for several days.

[phatboyg]

I’m going to call this fixed, since I can reproduce it and now it works properly.

[phatboyg]

You can add TransportOptions in the host configurator, it is a string/string collection. So add any of those properties and they’ll be specified when connecting to the broker. Like enable keep alive, or disable connection monitoring.