Multiple connections in multithreaded environment fail

Tried to use multiple NATS clients in multithreaded environment, but connections are failing with Invalid connection or Connection Request Time out when number of items scheduled in thread pool is above 6 (the number varies between 6 and 13).

This is my test code:

class Program
{
  static void Main(string[] args)
  {
    for (int i = 1; i < 16; i++)
    {
      QueueAndExecute(i);
    }

    Console.ReadLine();
  }

  private static void QueueAndExecute(int instances)
  {
    var finished = new CountdownEvent(1);

    for (var i = 0; i < instances; i++)
    {
      var instance = i + 1;

      finished.AddCount();
      ThreadPool.QueueUserWorkItem(state =>
      {
        var clientId = Guid.NewGuid().ToString("N");

        try
        {
          using (var c = new StanConnectionFactory().CreateConnection("test-cluster", clientId))
          {
            var bytes = Encoding.UTF8.GetBytes("this is test");

            c.Publish("test-channel", bytes);
            Console.WriteLine($"{instance}/{instances}: published");
          }
        }
        catch (Exception e)
        {
          Console.WriteLine($"{instance}/{instances}: {e.Message}");
        }

        finished.Signal();
      });
    }

    Console.WriteLine($"queued {instances}");
    finished.Signal();
    finished.Wait();
  }
}

Which produces following output:

queued 1
1/1: published
queued 2
2/2: published
1/2: published
queued 3
3/3: published
2/3: published
1/3: published
queued 4
1/4: published
3/4: published
2/4: published
4/4: published
queued 5
2/5: published
3/5: published
1/5: published
4/5: published
5/5: published
queued 6
3/6: published
1/6: published
2/6: published
6/6: published
5/6: published
4/6: published
queued 7
7/7: Invalid connection.
4/7: Invalid connection.
5/7: Invalid connection.
6/7: Connection Request Timed out.
2/7: Connection Request Timed out.
1/7: Connection Request Timed out.
3/7: published
queued 8
1/8: Invalid connection.
2/8: Invalid connection.
7/8: Invalid connection.
4/8: Invalid connection.
5/8: Invalid connection.
3/8: Invalid connection.
6/8: Invalid connection.
8/8: Invalid connection.
queued 9
1/9: Invalid connection.
2/9: Invalid connection.
3/9: Invalid connection.
4/9: Invalid connection.
5/9: Invalid connection.
6/9: Invalid connection.
7/9: Invalid connection.
8/9: Invalid connection.
9/9: Invalid connection.
queued 10
1/10: Invalid connection.
2/10: Invalid connection.
3/10: Invalid connection.
4/10: Invalid connection.
5/10: Invalid connection.
6/10: Invalid connection.
7/10: Invalid connection.
8/10: Invalid connection.
9/10: Invalid connection.
10/10: Invalid connection.
queued 11
1/11: Invalid connection.
2/11: Invalid connection.
3/11: Invalid connection.
4/11: Invalid connection.
5/11: Invalid connection.
6/11: Invalid connection.
7/11: Invalid connection.
8/11: Invalid connection.
9/11: Invalid connection.
10/11: Invalid connection.
11/11: Invalid connection.
queued 12
1/12: Invalid connection.
2/12: Invalid connection.
3/12: Invalid connection.
4/12: Invalid connection.
5/12: Invalid connection.
6/12: Invalid connection.
7/12: Invalid connection.
8/12: Invalid connection.
9/12: Invalid connection.
10/12: Invalid connection.
11/12: Invalid connection.
12/12: published
queued 13
1/13: Invalid connection.
2/13: Invalid connection.
3/13: Invalid connection.
4/13: Invalid connection.
5/13: Invalid connection.
6/13: Invalid connection.
7/13: Invalid connection.
8/13: Invalid connection.
9/13: Invalid connection.
10/13: Invalid connection.
11/13: Invalid connection.
12/13: Invalid connection.
13/13: Invalid connection.
queued 14
1/14: Invalid connection.
2/14: Invalid connection.
3/14: Invalid connection.
4/14: Invalid connection.
5/14: Invalid connection.
6/14: Invalid connection.
7/14: Invalid connection.
8/14: Invalid connection.
9/14: Invalid connection.
10/14: Invalid connection.
11/14: Invalid connection.
12/14: Invalid connection.
13/14: Invalid connection.
14/14: Invalid connection.
queued 15
1/15: Invalid connection.
2/15: Invalid connection.
3/15: Invalid connection.
4/15: Invalid connection.
5/15: Invalid connection.
6/15: Invalid connection.
7/15: Invalid connection.
8/15: Invalid connection.
9/15: Invalid connection.
10/15: Invalid connection.
11/15: Invalid connection.
12/15: Invalid connection.
13/15: Invalid connection.
14/15: Invalid connection.
15/15: published

I have nats-streaming-server version 0.10.2 running on my local machine (windows 10, 64-bit). Thread safety of creating connections seems to not be mentioned in documentation anywhere so I assume this should be allowed. When I lock around CreateConnection everything works fine.

Originally submitted by @mabzd as https://github.com/nats-io/csharp-nats/issues/224

[senzacionale]

Any info about this issue?

[ColinSullivan1]

@danielwertheim, thanks for testing! I’d consider this to be resolved… @senzacionale or others - if this continues to be an issue please re-open.