Infinite loop after NATS restart

We experience high CPU load in our services after NATS gets restarted or crashes for some reason. It seems like there is an infinite loop somewhere in the STAN.CLIENT.

We have profiled it and it seems like the issue is related to this:

instance bool [STAN.CLIENT] STAN.Client.BlockingDictionary`2[System.__Canon,System.__Canon]::Remove(!0,!1&,int32)
instance class STAN.Client.PublishAck [STAN.CLIENT] STAN.Client.Connection::removeAck(string)
instance void [STAN.CLIENT] STAN.Client.PublishAck::ackTimerCb(object)

After a quick view of the code this snippet might be the issue when 0 is passed to BlockingDictionary.Remove() as timeout from removeAck().

while (d.Count == 0)
{
    if (timeout < 0)
    {
        Monitor.Wait(dLock);
    }
    else
    {
        if (timeout > 0)
        {
            if (Monitor.Wait(dLock, timeout) == false)
            {
                throw new Exception("timeout");
            }
        }
    }
}

To reproduce it I do the following: (It doesn’t happen every time)

• Crash NATS
  • I do this by sending a message which is too large for the MySQL data field. This was the reason we encountered the problem in the first place.
• Restart NATS
• Send a message to NATS through the STAN.CLIENT
• Inspect the docker container for high CPU usage

Let me know if I need to provide a repository for reproducing this issue. I hope it’s not nessesary 😃

[ColinSullivan1]

FYI - I’ve found the issue, and have a fix but it’s not ideal imo. I want to clean up some of the code around publish ack processing to keep this clean. I’m hoping to have something tomorrow.

[ColinSullivan1]

Good to hear! Thank you for testing, much appreciated!

If you do get more information about your performance issue or can isolate a test case please open an issue…