Excessive memory usage after many reconnection attempts

I found some extra ordinary logs.

2017/04/20 01:24:45.722 DEBUG (Slf4JLogger.java:81) - SSL_read failed: OpenSSL error: error:10000414:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_REVOKED
2017/04/20 01:24:45.722  INFO (ApnsClient.java:523) - Failed to connect.
javax.net.ssl.SSLHandshakeException: error:10000414:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_REVOKED
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:696)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:899)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:853)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:931)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:974)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1097)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:968)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:902)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)
2017/04/20 01:24:45.722  WARN (Slf4JLogger.java:146) - [id: 0x93247371, L:/10.0.13.69:41672 - R:api.push.apple.com/17.188.163.207:443] TLS handshake failed:
javax.net.ssl.SSLHandshakeException: error:10000414:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_REVOKED
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:696)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:899)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:853)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:931)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:974)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1097)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:968)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:902)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)
2017/04/20 01:24:45.723 DEBUG (ApnsClient.java:471) - Disconnected. Next automatic reconnection attempt in 1 seconds.

---

found the expired certificate. so that apnsClient tried to reconnect. In this case, when I connected at first time, the certificate was not expired yet. but after few days, it’s been expired. how can I avoid this problem? I want to ignore to reconnect in this case.

[jchambers]

Attempting to reconnect when the connection is lost is the expected behavior. We’ve been through this territory a number of times, and the fundamental issue is that we don’t have a reliable way of detecting all of the “certificate has stopped working due to expiration or revocation” cases; instead, we have to rely on callers to be performing that maintenance on their own.

We could consider adding a special-case listener system for SSL-related connection exceptions, but we’ve tried that in the past and found it to be less than reliable. It seems like the best course of action may be to use a metrics listener to watch for a high rate of connection failures.

how can I avoid this problem?

Try using token-based authentication; token signing keys can still be revoked, but won’t expire. Because tokens come into play after a connection has been established, signing key revocation won’t cause connection cycling. Here, you’d still need to be monitoring notification rejection rates for signs of key revocation (i.e. all notifications are failing with InvalidProviderToken).

Also the retries went in infinite loop and crashed the system due to memory consumption.

The memory consumption sounds like a bug, and we’ll look into it.

[firesofmay]

I have found similar issue as well today in our production use. Is there a way to fix this? Also the retries went in infinite loop and crashed the system due to memory consumption.