CPU thrash and slow response time on low number of Websocket connections

Expected behavior

any operation like channel.write(new TextWebSocketFrame(data)); should ideally take only a couple of microsecs at best.

Actual behavior

response times exceeding 10 seconds. Suddenly all available CPUs (32 cores) utilized 100%, even though there is only a single websocket connection. LOGS of the sample app - see the “SLOW SEND” line:

>  java -jar netty-example.jar
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Apr 12, 2019 10:54:32 AM io.netty.handler.logging.LoggingHandler channelRegistered
INFO: [id: 0x5e1c5fad] REGISTERED
Apr 12, 2019 10:54:32 AM io.netty.handler.logging.LoggingHandler bind
INFO: [id: 0x5e1c5fad] BIND(/127.0.0.1:7847)
Apr 12, 2019 10:54:32 AM io.netty.handler.logging.LoggingHandler channelActive
INFO: [id: 0x5e1c5fad, L:/127.0.0.1:7847] ACTIVE
Apr 12, 2019 10:54:33 AM io.netty.handler.logging.LoggingHandler logMessage
INFO: [id: 0x5e1c5fad, L:/127.0.0.1:7847] READ: [id: 0x8c54f3e3, L:/127.0.0.1:7847 - R:/127.0.0.1:41756]
Apr 12, 2019 10:54:33 AM io.netty.handler.logging.LoggingHandler channelReadComplete
INFO: [id: 0x5e1c5fad, L:/127.0.0.1:7847] READ COMPLETE
SLOW SEND. Over 5 secs
Apr 12, 2019 10:54:53 AM io.netty.handler.logging.LoggingHandler close
INFO: [id: 0x5e1c5fad, L:/127.0.0.1:7847] CLOSE()
Apr 12, 2019 10:54:53 AM io.netty.handler.logging.LoggingHandler channelUnregistered
INFO: [id: 0x5e1c5fad, L:/127.0.0.1:7847] UNREGISTERED

Steps to reproduce

compile and start the code below on a LINUX with reasonable number of cores (tested on 32 cores) Intel Sky-Lake add a websocket connection and watch the logs of the server:

curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: localhost" -H "Origin: http://localhost:7847" http://localhost:7847/wsSubscribe > /dev/null

Minimal yet complete reproducer code (or URL to code)

using https://github.com/buremba/netty-rest/blob/master/src/main/java/org/rakam/server/http/RakamHttpRequest.java as a dependency to make it easier to set up the websocket server.

The important method in question is:

public void sendDataToAllChannels(String data) throws InterruptedException {
            for (Channel c : sessions) {
                if (c.isOpen()) {
                    if (!c.isWritable()) {
                        c.flush();
                    }
                    c.write(new TextWebSocketFrame(data));
                }
            }
        }

Full sources:

/*
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */
package org.example.netty.ws;

import io.netty.channel.Channel;
import io.netty.channel.ChannelHandlerContext;
import io.netty.handler.codec.http.websocketx.TextWebSocketFrame;
import io.swagger.annotations.ApiOperation;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import org.rakam.server.http.HttpServer;
import org.rakam.server.http.HttpServerBuilder;
import org.rakam.server.http.HttpService;
import org.rakam.server.http.WebSocketService;
import org.rakam.server.http.annotations.ApiParam;
import org.rakam.server.http.annotations.JsonRequest;

public class Example implements AutoCloseable {

    private static final String MSG = "FIXED MESSAGE THAT IS SENT TO ALL WEBCHANNELS.";

    NettyWebSocketService wsService;

    HttpServer build;

    public static void main(String[] args) throws Exception {
        try (Example example = new Example()) {
            example.startServer();

            while (true) {
                long start = System.currentTimeMillis();
                example.wsService.sendDataToAllChannels(MSG);
                long delta = System.currentTimeMillis() - start;
                if (delta > 5000) {
                    System.out.println("SLOW SEND. Over 5 secs");
                    break;
                }
            }
        }
    }

    public void startServer() throws Exception {
        wsService = new NettyWebSocketService();
        build = new HttpServerBuilder()
                .setUseEpollIfPossible(false)
                .enableDebugMode(false)
                .setHttpServices(new HashSet<>(Arrays.asList(new CustomHttpServer())))
                .setWebsocketServices(new HashSet<>(Arrays.asList(wsService)))
                .build();

        build.bind("127.0.0.1", 7847);
    }

    @Override
    public void close() {
        build.stop();
    }

    @Path("/wsSubscribe")
    public static class NettyWebSocketService extends WebSocketService {

        public List<Channel> sessions = new ArrayList<>();

        @Override
        public void onOpen(WebSocketService.WebSocketRequest request) {
            sessions.add(request.context().channel());
        }

        @Override
        public void onMessage(ChannelHandlerContext ctx, String message) {
            if (message.equalsIgnoreCase("Hello")) {
                ctx.writeAndFlush(new TextWebSocketFrame("world."));
            } else {
                // ignore message
            }
        }

        @Override
        public void onClose(ChannelHandlerContext ctx) {
            sessions.remove(ctx.channel());
        }

        public void sendDataToAllChannels(String data) throws InterruptedException {
            for (Channel c : sessions) {
                if (c.isOpen()) {
                    if (!c.isWritable()) {
                        c.flush();
                    }
                    c.write(new TextWebSocketFrame(data));
                }
            }
        }
    }

    @Path("/v1")
    public static class CustomHttpServer extends HttpService {

        @JsonRequest
        @ApiOperation(value = "Parameter demo endpoint")
        @Path("/parameter")
        public String testJsonParameter(@ApiParam("param1") String param1, @ApiParam("param2") int param2) {
            return param1 + param2;
        }

//        @JsonRequest
        @Path("/hello")
        @GET
        public String sendHelloWorld() {
            return "Hello world";
        }
    }

}

Netty version

4.0.56

JVM version (e.g. java -version)

openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

OS version (e.g. uname -a)

Linux CentOS Linux release 7.6.1810 (Core) 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[normanmaurer]

@ratcashdev no its not a blocking operation…… If you would like to block until it is flushed you could use something like this:

for (Channel c : sessions) {
    if (c.isOpen()) {
        if (!c.isWritable()) {
            c.writeAndFlush(new TextWebSocketFrame(data)).sync();
        } else {
            c.write(new TextWebSocketFrame(data));
        }
    }
}

Just note that you must never do this within an EventLoop thread.

[fredericBregier]

To add to @normanmaurer comment, you may also, but after his recommendation since he is fully right, to limit the memory pressure. If I understand correctly, you want to send the very same TextWebSocketFrame(data). Maybe using retainedDuplicate() by creating only one such object could help…