Epoll consuming lot more CPU than Nio

I am building a websocket based broker. During load testing, we found that Epoll transport uses around 55% CPU compared to around 20% for Nio, just for maintaining the connections without doing any business specific IO on those connections. Is this expected? What could I be doing wrong? Happy to share any more info required around this.

Total concurrent connections: Around 27K Boss Threads: 1 Worker Threads: 32

Cores in the VM: 8

Relevant code that sets up netty.

  EventExecutorChooserFactory chooserFactory = DefaultEventExecutorChooserFactory.INSTANCE;

  ThreadFactory bossThreadFactory = ThreadFactoryUtil.createInstrumented("boss", metricRegistry);

  Executor workerExecutor = new ThreadPerTaskExecutor(
          ThreadFactoryUtil.createAffinityThreadFactory("worker.thread", metricRegistry));

private void setupNioEventLoopGroups(EventExecutorChooserFactory chooserFactory, ThreadFactory bossThreadFactory,
            Executor workerExecutor) {
        bossPool = new NioEventLoopGroup(acceptorThreads, bossThreadFactory);
        workerPool = new NioEventLoopGroup(workerThreads, workerExecutor, chooserFactory, SelectorProvider.provider(),
                DefaultSelectStrategyFactory.INSTANCE);
        channelType = NioServerSocketChannel.class;
        log.info("Initialiazing Java NIO Event System");
    }

    private void setupEpollEventLoopGroups(EventExecutorChooserFactory chooserFactory, ThreadFactory bossThreadFactory,
            Executor workerExecutor) {
        bossPool = new EpollEventLoopGroup(acceptorThreads, bossThreadFactory);
        workerPool = new EpollEventLoopGroup(workerThreads, workerExecutor, chooserFactory,
                DefaultSelectStrategyFactory.INSTANCE);
        channelType = EpollServerSocketChannel.class;
        log.info("Initialiazing Epoll IO Event System");
    }


        ServerBootstrap serverBootstrap = new ServerBootstrap().group(bossPool, workerPool);

        // Choose socket options.
        Map<ChannelOption<?>, Object> channelOptions = new HashMap<>();
        channelOptions.put(ChannelOption.SO_BACKLOG, 256);
        channelOptions.put(ChannelOption.ALLOCATOR, new PooledByteBufAllocator(true));
        channelOptions.put(ChannelOption.SO_TIMEOUT, 3000);

        channelOptions.forEach(
                (key, value) -> serverBootstrap.option(ChannelOption.valueOf(String.valueOf(key)), value));

        // Set transport options
        serverBootstrap.childOption(ChannelOption.TCP_NODELAY, true);
        serverBootstrap.childOption(ChannelOption.SO_KEEPALIVE, true);
        serverBootstrap.childOption(ChannelOption.SO_LINGER, -1);
        serverBootstrap.childOption(ChannelOption.SO_REUSEADDR, true);
        serverBootstrap.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
        serverBootstrap.childOption(ChannelOption.ALLOW_HALF_CLOSURE, false);
        serverBootstrap.childOption(ChannelOption.SO_SNDBUF, 10 * 1024);
        serverBootstrap.channel(serverGroup.getChannelType());
        serverBootstrap.childHandler(channelInitializer);

Netty version

4.1.68

JVM version (e.g. java -version)

openjdk version “11.0.12” 2021-07-20 LTS OpenJDK Runtime Environment Zulu11.50+19-CA (build 11.0.12+7-LTS) OpenJDK 64-Bit Server VM Zulu11.50+19-CA (build 11.0.12+7-LTS, mixed mode)

OS version (e.g. uname -a)

Linux bolt-004 5.8.0-1041-azure #44~20.04.1-Ubuntu SMP Fri Aug 20 20:41:09 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Epoll

Nio

[chrisvest]

Only using epoll_wait when the timeout is greater than a millisecond will cause rounding in timeouts, though. For 200 millisecond timeouts that probably won’t matter, but it might for a 1,5 millisecond timeout. So we’d need to pick a cut-off point. On kernel 5.11 and newer, we can use epoll_pwait2, which takes a timespec parameter.

[normanmaurer]

I wonder if the time used by timerfd_settime is any indicator that we might be better of to not use it when possible (when the timeout is milliseconds) and just use epoll_wait(...) with the right timeout. WDYT ?