DNS Resolver leaking direct memory

Expected behavior

DNS resolver does not leak direct memory when buffers are freed correctly.

Actual behavior

Memory is leaked, JVM runs out of direct memory.

[error] Exception in thread "main" io.netty.resolver.dns.DnsNameResolverException: [/10.0.2.3:53] failed to send a query via UDP (no stack trace available)
[error] Caused by: io.netty.handler.codec.EncoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 234881024, max: 239075328)

full stacktrace: https://gist.github.com/dziemba/ffeb8f2c3b131a74c042746151ac05c0

Steps to reproduce

• Use DnsNameResolver.resolveAll to query SRV records for a name that contains a lot of answers (e.g. hugedns.test.dziemba.net)
• Run query repeatedly and .release() returned DNS records.
• Watch java process memory usage increase until it finally throws with io.netty.util.internal.OutOfDirectMemoryError

It might be important that the DNS response is large so it gets truncated and then retried via TCP.

Minimal yet complete reproducer code (or URL to code)

https://gist.github.com/dziemba/c904d227d105b6fc7cf00495257fbb40

• run with -Xmx128m -XX:MaxDirectMemorySize=4m to make it fail quickly
• will fail as described after around 100-200 iterations (with above settings)
• sometimes it fails with actual network errors, re-run a few times if that happens

Netty version

4.1.42.Final

JVM version (e.g. java -version)

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.222-b10, mixed mode)

I was not able to reproduce this on Java 11 or 12.

OS version (e.g. uname -a)

Darwin wopro3 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64

I also tried to reproduce this on Linux. It shows the same error behavior (fails after 100 runs of the test script) but does not output the OutOfDirectMemoryError for some reason…

[njhill]

@normanmaurer @dziemba please see #9479

FYI the main leak in the current code is here - the result of the future needs to be released, since the finish method that it’s passed to does not take ref-count ownership: https://github.com/netty/netty/blob/4dc1eccf60252f8a690610c20495ed095e274d3a/resolver-dns/src/main/java/io/netty/resolver/dns/DnsNameResolver.java#L1302

[normanmaurer]

@njhill @dziemba imho the retain() is correct here. I will need to look into this in more detail.