Socket issues on Springboot 1.5.4

After migration for Spring Boot 1.5.3, after a timeframe of 30 min to 4h, Spring application starts to fail due:

2017-07-13 10:45:43.946 DEBUG 7 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Cannot acquire connection from data source
--
  | com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
  | The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
  | at sun.reflect.GeneratedConstructorAccessor93.newInstance(Unknown Source)
  | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  | at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  | at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
  | at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
  | at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
  | at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2192)
  | at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
  | at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
  | at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
  | at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
  | at sun.reflect.GeneratedConstructorAccessor86.newInstance(Unknown Source)
  | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  | at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  | at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
  | at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
  | at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
  | at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
  | at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
  | at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
  | at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193)
  | at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:430)
  | at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:64)
  | at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:570)
  | at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:563)
  | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  | at java.lang.Thread.run(Thread.java:748)
  | Caused by: java.net.UnknownHostException: <OUR_DB_HOSTNAME>: Temporary failure in name resolution
  | at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
  | at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
  | at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
  | at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
  | at java.net.InetAddress.getAllByName(InetAddress.java:1192)
  | at java.net.InetAddress.getAllByName(InetAddress.java:1126)
  | at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:188)
  | at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
  | ... 23 common frames omitted
  | 2017-07-13 10:45:43.949 ERROR 7 --- [8081-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint : Socket accept failed
  | java.io.IOException: Too many open files
  | at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  | at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
  | at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
  | at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:453)
  | at java.lang.Thread.run(Thread.java:748)

Then, I migrated to Spring boot 1.5.4 and it presented the same issue, but on 1.5.1 it works fine. Then, I’ve created two different environments, one working with 1.5.1 and another working with 1.5.4.

Walking through lsof results on failing container, we can see many entries like: java 7 root DEL REG 253,13 272650 /tmp/jar_cache5430334089419640871.tmp , which tell us that the too many open files is because we have many non closed socket files.

Java version:

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Docker version:

Docker version 17.05.0-ce, build 89658be

Complete log of app failing https://gist.github.com/sergiofigueras/72223e6c9e01790725927ecc83e67581

Results of lsof in FAILING container https://gist.github.com/sergiofigueras/a138545619ffeeac60becd70cac196a2

Telnet on mysql from FAILING container

root@963c12df9cfc:/home/project/neo4jport# telnet <OUR_DB_HOSTNAME> 3306
Trying 10.69.11.109...
Connected to <OUR_DB_HOSTNAME>.
Escape character is '^]'.

Telnet on mysql from WORKING container

root@e13d22d8099b:/home/project/neo4jport# telnet <OUR_DB_HOSTNAME> 3306
Trying 10.69.11.109...
Connected to <OUR_DB_HOSTNAME>.
Escape character is '^]'. 

Socket Summary on FAILING container

root@963c12df9cfc:/home/project/neo4jport#  ss -s
Total: 27024 (kernel 398412)
TCP:   659 (estab 2, closed 656, orphaned 4, synrecv 0, timewait 71/0), ports 0

Connections to 3306 on FAILING container

root@963c12df9cfc:/home/project/neo4jport# ss -nt dst :3306
State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port

Socket summary on WORKING container

[ec2-user@ip-10-224-82-219 ~]$ docker exec -it e13d22d8099b bash
root@e13d22d8099b:/home/project/neo4jport# ss -s
Total: 27029 (kernel 398412)
TCP:   631 (estab 12, closed 618, orphaned 0, synrecv 0, timewait 42/0), ports 0

Connections to 3306 on RUNNING container

root@e13d22d8099b:/home/project/neo4jport# ss -nt dst :3306
State      Recv-Q Send-Q                                           Local Address:Port                                                          Peer Address:Port
ESTAB      0      0                                           ::ffff:172.17.0.23:45536                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46558                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46358                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46736                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:45854                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46902                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46900                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:47064                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46360                                                  ::ffff:10.69.11.109:3306
ESTAB      0      0                                           ::ffff:172.17.0.23:46898                                                  ::ffff:10.69.11.109:3306

Container sockets configuration

root@e13d22d8099b:/home/project/neo4jport# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3934790
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3934790
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
  | data seg size           (kbytes, -d) unlimited
  | scheduling priority             (-e) 0
  | file size               (blocks, -f) unlimited
  | pending signals                 (-i) 3934790
  | max locked memory       (kbytes, -l) 64
  | max memory size         (kbytes, -m) unlimited
  | open files                      (-n) 1024
  | pipe size            (512 bytes, -p) 8
  | POSIX message queues     (bytes, -q) 819200
  | real-time priority              (-r) 0
  | stack size              (kbytes, -s) 8192
  | cpu time               (seconds, -t) unlimited
  | max user processes              (-u) 3934790
  | virtual memory          (kbytes, -v) unlimited
  | file locks                      (-x) unlimited

For me it seems that smth in Spring Boot is not closing these connections until no more connections can be opened. Does anybody have faced this before? What can be done in that case? What more information do you need?

[wilkinsona]

I’d connect with a debugger and add a breakpoint to the method that I linked to above. That should show what’s opening the file handles and may help to track down why they’re not being closed. If it were me, in the interests of simplifying things as much as possible, I’d do that without involving Docker. Alternatively, you can map ports in the container to the host and debug as you normally would.

[wilkinsona]

Duplicate of #9866

While this issue was opened first, #9866 has homed right in on the problem so I’m going to close this one.