Event Source receives heartbeat event but still gives error when timeout: Error: No activity within N milliseconds. 96 characters received. Reconnecting.

I am working on a web application using react and spring boot. To add live notification feature, I choosed server-sent-events with your library on the client. I set heartbeatTimeout to 120s and periodically send a heartbeat event every 40s to keep the connection open.

It works OK when I test it locally, but when I deploy the app it doesn’t anymore. The connection is still open normally, the client still receives the heartbeat events in full and at the right time, but usually every 3 heartbeat events, the client gives an error: Error: No activity within N milliseconds. 96 characters received. Reconnecting.

I think the biggest difference between local environment and my deployment environment is that in local environment, client connects directly to backend, and in deployment environment, I use Nginx between them. But I still can’t figure out which part of the deployment pattern is the reason of error.

Here is the code I use: React

React.useEffect(() => {
    let eventSource;
    let reconnectFrequencySeconds = 1;

    // Putting these functions in extra variables is just for the sake of readability
    const wait = function () {
      return reconnectFrequencySeconds * 1000;
    };

    const tryToSetup = function () {
      setupEventSource();
      reconnectFrequencySeconds *= 2;

      if (reconnectFrequencySeconds >= 64) {
        reconnectFrequencySeconds = 64;
      }
    };

    // Reconnect on every error
    const reconnect = function () {
      setTimeout(tryToSetup, wait());
    };

    function setupEventSource() {
      fetchNotification();

      eventSource = new EventSourcePolyfill(
        `${API_URL}/notification/subscription`,
        {
          headers: {
            "X-Auth-Token": store.getState().auth.token,
          },
          heartbeatTimeout: 120000,
        }
      );

      eventSource.onopen = (event) => {
        console.info("SSE opened");
        reconnectFrequencySeconds = 1;
      };

      // This event only to keep sse connection alive
      eventSource.addEventListener(SSE_EVENTS.HEARTBEAT, (e) => {
        console.log(new Date(), e);
      });

      eventSource.addEventListener(SSE_EVENTS.NEW_NOTIFICATION, (e) =>
        handleNewNotification(e)
      );

      eventSource.onerror = (event) => {
        // When server SseEmitters timeout, it cause error
        console.error(
          `EventSource connection state: ${
            eventSource.readyState
          }, error occurred: ${JSON.stringify(event)}`
        );

        if (event.target.readyState === EventSource.CLOSED) {
          console.log(
            `SSE closed (event readyState = ${event.target.readyState})`
          );
        } else if (event.target.readyState === EventSource.CONNECTING) {
          console.log(
            `SSE reconnecting (event readyState = ${event.target.readyState})`
          );
        }

        eventSource.close();
        reconnect();
      };
    }

    setupEventSource();

    return () => {
      eventSource.close();
      console.info("SSE closed");
    };
  }, []);

Spring

    /**
     * @param toUser
     * @return
     */
    @GetMapping("/subscription")
    public ResponseEntity<SseEmitter> events(
        @CurrentSecurityContext(expression = "authentication.name") String toUser
    ) {
        log.info(toUser + " subscribes at " + getCurrentDateTime());

        SseEmitter subscription;

        if (subscriptions.containsKey(toUser)) {
            subscription = subscriptions.get(toUser);
        } else {
            subscription = new SseEmitter(Long.MAX_VALUE);
            Runnable callback = () -> subscriptions.remove(toUser);

            subscription.onTimeout(callback); // OK
            subscription.onCompletion(callback); // OK
            subscription.onError((exception) -> { // Must consider carefully, but currently OK
                subscriptions.remove(toUser);
                log.info("onError fired with exception: " + exception);
            });

            subscriptions.put(toUser, subscription);
        }

        HttpHeaders responseHeaders = new HttpHeaders();
        responseHeaders.set("X-Accel-Buffering", "no");
        responseHeaders.set("Cache-Control", "no-cache");

        return ResponseEntity.ok().headers(responseHeaders).body(subscription);
    }

    /**
     * To keep connection alive
     */
    @Async
    @Scheduled(fixedRate = 40000)
    public void sendHeartbeatSignal() {
        subscriptions.forEach((toUser, subscription) -> {
            try {
                subscription.send(SseEmitter
                                      .event()
                                      .name(SSE_EVENT_HEARTBEAT)
                                      .comment(":\n\nkeep alive"));
//                log.info("SENT HEARBEAT SIGNAL AT: " + getCurrentDateTime());
            } catch (Exception e) {
                // Currently, nothing need be done here
            }
        });
    }

Nginx

events{
}
http {

server {
    client_max_body_size 200M;
    proxy_send_timeout 12000s;
    proxy_read_timeout 12000s;
    fastcgi_send_timeout 12000s;
    fastcgi_read_timeout 12000s;
    location = /api/notification/subscription {
        proxy_pass http://baseweb:8080;
        proxy_set_header Connection '';
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
        proxy_buffering off;
        proxy_buffer_size 0;
        proxy_cache off;
        proxy_connect_timeout 600s;
        fastcgi_param NO_BUFFERING "";
        fastcgi_buffering off;
    }
}
}

I really need support now

[AnhTuan-AiT]

After trying to find the cause of the error, I obtained the following symptoms.

Before talking about the phenomenon, I will describe how my application works. When the user accesses the application, I will check the token in localStorage, if exists (regardless of whether the token is valid or not) then the application’s state will be set as logged in and the component containing the Event Source object will be rendered and initiate a connection to the server with an expired token placed in the headers. And of course it can’t connect, then the onerror callback function will be called and the reconnect function will be called. And here is the next phenomenon:

1. When I press the login button, the application goes to the login screen and of course the component containing the Event Source object is unmounted, but when I look at the console, what I see is the reconnect function keeps executing.

2. I have a theory and to test the theory, I added a flag value as follows:

headers: {
          "X-Auth-Token": store.getState().auth.token,
          Count: count++,
        },

Here, count will be incremented by 1 every time the reconnect function is called. And after I have successfully logged in, got a valid new token, successfully connected to the server, usually every 3 heartbeat events, I get the error: No activity within N milliseconds. 96 characters received. Reconnecting. Thanks to the count variable, I know that the error was thrown from the Event Source object that failed in the previous connection, when I wasn’t logged in.

1. When I remove the reconnect. At the time I access the application, of course the Event Source will not be able to connect and the onerror callback function is called, but this time there is no reconnect function so only an Event Source object is created. After logging in, I see the Event Source successfully connected to the server and received the heartbeat events fully and in the correct cycle, no more errors.

I don’t know why the reconnect function can continue to run when the component is unmounted and why the previous Event Source objects continue to exist

[Yaffle]

@AnhTuan-AiT , no, usage of onmessage in place of addEventListener should not make any difference