Large number of ports in use

Hey team,

First of all, I want to say that this is a pretty cool project and it has been really easy to get up and running. I just got a test up and running and on about 40 computers at our office all connected to a single Directory Service. We opened up TCPView to see what the network effect of every computer running a bus and at first glance it appears all the peers have a direct connection to every other peer? Is this the correct behavior? Could I get a little more detail on what each peer is connected to?

Just for reference. This is what the setup looks like.

Directory Service Initialization

public void Start() {

    _busFactory = new BusFactory();
    InjectDirectoryServiceSpecificConfiguration(_busFactory);

    _busFactory
        .WithConfiguration(new AppSettingsBusConfiguration(), ConfigurationManager.AppSettings["Environment"])
        .WithScan()
        .WithEndpoint(ConfigurationManager.AppSettings["Endpoint"])
        .WithPeerId(ConfigurationManager.AppSettings["PeerId"]);

    _busFactory.CreateAndStartBus();
    _log.Info("In memory directory started");

    _log.Info("Starting dead peer detector");
    _deadPeerDetector = _busFactory.Container.GetInstance<IDeadPeerDetector>();
    _deadPeerDetector?.Start();

}

public void Stop() {
    _log.Info("Stopping dead peer detector");
    _deadPeerDetector?.Stop();
}

private void InjectDirectoryServiceSpecificConfiguration(BusFactory busFactory) {
    busFactory.ConfigureContainer(c => {
        c.AddRegistry<DirectoryRegistry>();
        c.ForSingletonOf<IDirectoryConfiguration>().Use<AppSettingsDirectoryConfiguration>();

        c.For<IDeadPeerDetector>().Use<DeadPeerDetector>();
        c.ForSingletonOf<IPeerRepository>().Use<MemoryPeerRepository>();
        c.ForSingletonOf<PeerDirectoryServer>().Use<PeerDirectoryServer>();
        c.ForSingletonOf<IPeerDirectory>().Use(ctx => ctx.GetInstance<PeerDirectoryServer>());

        c.ForSingletonOf<IMessageDispatcher>().Use(typeof(Func<IContext, MessageDispatcher>).Name, ctx => {
            var dispatcher = ctx.GetInstance<MessageDispatcher>();
            dispatcher.ConfigureHandlerFilter(x => x != typeof(PeerDirectoryClient));

            return dispatcher;
        });
    });
}

Then to connect to it, we use something like this:

_container = new Container();
_container.Configure(x => x.ForSingletonOf<BridgeFourLegacy>().Use(this));
var factory = new Abc.Zebus.Core.BusFactory(_container);

factory
    .WithConfiguration("tcp://Bridge4:129", "Demo")
    .WithPeerId($"BridgeFourLegacy.*")
    .WithHandlers(typeof(LegacyMessageReceivedHandler));

_bus = factory.CreateAndStartBus();

Thanks so much.

[ocoanet]

Ok, I just checked the socket usage. It is clearly quite high! However, you might notice that most of the sockets are local-only: they are internal ZMQ sockets.

Internal ZMQ sockets

Zebus uses ZMQ sockets to send and receive messages. Those sockets are high level components that can create multiple TCP sockets to communicate with multiple nodes. But ZMQ does not only create sockets for inter-node messaging: it also uses TCP sockets for its internal inter-thread communications.

Starting a ZMQ context (with no ZMQ sockets) creates 2 “local” TCP sockets on my machine:

using (var context = new ZmqContext())
{
    Console.WriteLine("Context created");
    Console.WriteLine("Press enter to exit...");
    Console.ReadLine();
}

Creating and binding a ZMQ PULL socket adds 6 “local” TCP sockets and 1 “network” TCP socket:

using (var socket = new ZmqSocket(context, ZmqSocketType.PULL))
{
    socket.Bind("tcp://*:123");

    Console.WriteLine("Socket bound");
    Console.WriteLine("Press enter to exit...");
    Console.ReadLine();
}

If port usage is not an issue on your environment, you can ignore the “local” sockets. If this is an issue for you, there is not much we can do because those sockets are a ZMQ implementation detail.

ZMQ inter-node sockets

Zebus uses PULL and PUSH ZMQ sockets. Every Zebus peer has one PULL socket for inbound messages and multiple PUSH sockets for outbound messages, one for each target peer.

The PULL/PUSH ZMQ sockets should generate one inter-node TCP socket every time a peer needs to send messages to an other peer. However, because those sockets are unidirectional, another pair of PULL/PUSH ZMQ sockets is used for communication in the other direction, which creates another inter-node TCP socket.

So, the inter-node TCP socket usage is slightly sub-optimal: 2 sockets are created for bidirectional communication when just one could be used. This behavior is not an issue for us despite the large number of peers that we run in our production environment. If you think it is an issue for you, it might be possible to add an option to use different, bidirectional, ZMQ socket types.

Hope this helps.

[veenroid]

750! Awesome. Thank you for your efforts.