Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

All connections from pool are suddenly closed or never released

See original GitHub issue

I have an Express server which do some queries to our Neo4j database. At the very beginning of the server lifetime, I create the neo4j driver singleton as recommended in the docs. After some days of normal use, the driver suddenly closes all the connections of the pool (there is no driver.close(), no reassignation to the driver, nor nothing… only getting and closing sessions and writeTransactions in the code).

Neo4j Version: 3.4.12 Community
Neo4j Mode: Single instance
Driver version: JS driver 1.7.1
Operating System of DB: CentOS 7 on GCP Operating System of Server using neo4j-driver: Debian GNU/Linux 9 (stretch) on GCP

Pre-requisites

Having a Neo4j DB instance running and the server correctly configured(which I might be doing wrong maybe?) to communicate to the neo4j instance.

Steps to reproduce

  1. Start the server
  2. Some API calls are being correctly served by the server for a couple of days. The server queries using the driver and writeTransactions.
  3. After a couple of days of working correctly, out of a sudden all the connections of the pool are closed
  4. When serving the next API calls, the server throws the error Neo4jError: Connection acquisition timed out in 60000 ms.

Expected behavior

All the connections are open during the whole driver’s lifecycle (which should be the whole application lifetime according to the docs)

Actual behavior

The connections are all closed and I cannot get more connections from the driver.

This is the driver configuration:

const driver = neo4j.driver(
    process.env.NEO4J_URI,
    neo4j.auth.basic(process.env.NEO4J_USER, process.env.NEO4J_PASSWORD),
    {
      maxConnectionLifetime: 60 * 60 * 1000, // 1 hour
      maxConnectionPoolSize: 300,
      encrypted: "ENCRYPTION_ON",
      trust: "TRUST_CUSTOM_CA_SIGNED_CERTIFICATES",
      trustedCertificates: [process.env.NEO4J_TRUSTED_CERTS],
      logging: {
        level: 'debug',
        logger: (level, message) => console.log('+++' + level + ' ' + message)
      }
    }
);

I also implemented the logger as one of the suggestions in the answers you guys gave in one issue. Since it was running for a while, it generated a log of about 210 MB. I split it in 25 log files, the first 24 useless and the last one is where the error can be seen. I’m just quoting the important parts of the log, but I’ll also attach the last log file if that helps. The connections closing starts at line 2,700 of the file.

Lastly, I just wanted to say thanks guys for the nice efforts done in this module! I really appreciate the work done here. I just want some help with my issue.

xay.txt

+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":["p.port_id","p.name","p.longitude","p.latitude"]}]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":1847,"high":0},"PORT HEDLAND",118.5769444,-20.31750002]]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"r"}]}
+++debug Connection [4837][] C: RUN COMMIT {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":1,"high":0},"fields":["p.port_id","p.name","p.longitude","p.latitude"]}]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":2829,"high":0},"PARADIP",86.68,20.26]]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"r"}]}
+++debug Connection [5065][] C: RUN COMMIT {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [4837][] C: RESET
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [5065][] C: RESET
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [4837][] acquired from the pool
+++debug Connection [4837][] C: RUN BEGIN {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [4837][] C: RUN WITH point({longitude:118.5769444, latitude:-20.31750002}) AS p, 5000 as distance_nm
        CALL spatial.withinDistance('all_connected_ports', p, distance_nm*1.852)
          YIELD node, distance
          RETURN node.port_id AS port_id, node.name, node.type AS name, distance/1.852 AS distance, node.longitude, node.latitude
          LIMIT 1 {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [5065][] acquired from the pool
+++debug Connection [5065][] C: RUN BEGIN {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [5065][] C: RUN WITH point({longitude:86.68, latitude:20.26}) AS p, 5000 as distance_nm
        CALL spatial.withinDistance('all_connected_ports', p, distance_nm*1.852)
          YIELD node, distance
          RETURN node.port_id AS port_id, node.name, node.type AS name, distance/1.852 AS distance, node.longitude, node.latitude
          LIMIT 1 {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":2,"high":0},"fields":["port_id","node.name","name","distance","node.longitude","node.latitude"]}]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":1803,"high":0},"DAMPIER","P",107.67802056701662,116.695,-20.64666669]]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"rw"}]}
+++debug Connection [4837][] C: RUN COMMIT {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":3,"high":0},"fields":["port_id","node.name","name","distance","node.longitude","node.latitude"]}]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":2829,"high":0},"PARADIP","P",0.00005126096002483986,86.68,20.26]]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"rw"}]}
+++debug Connection [5065][] C: RUN COMMIT {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [4837][] C: RESET
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [5065][] C: RESET
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [5065][] acquired from the pool
+++debug Connection [5065][] C: RUN BEGIN {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [5065][] C: RUN MATCH (start:Port{port_id: 1803}), (end:Port{port_id: 2829})
          CALL algo.shortestPath.astar.stream(start, end, 'NM', 'latitude', 'longitude',{relationshipQuery:'DISTANCE',direction:'BOTH'})
          YIELD nodeId, cost
          MATCH (route:Port) WHERE ID(route)=nodeId
          RETURN route.port_id AS port_id, route.name AS name, route.longitude AS longitude, route.latitude AS latitude, cost AS nm {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":["port_id","name","longitude","latitude","nm"]}]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":1803,"high":0},"DAMPIER",116.695,-20.64666669,0]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":4032,"high":0},"SUMBAWA STRAIT",116.6867,-8.5945,720]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":4004,"high":0},"LOMBOK STRAIT",115.8872,-8.3256,802]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":4000,"high":0},"KARIMATA STRAIT",108.6225,-2.0428,1386]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":4019,"high":0},"SINGAPORE STRAIT",103.4689,1.1892,1775]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":8206,"high":0},"WAYPOINT 56",98.12277778,8.07972222,2306]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":8210,"high":0},"WAYPOINT 60",93.59083333,14.22888889,2760]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":8209,"high":0},"WAYPOINT 59",89.81472222,19.31777778,3133]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":2814,"high":0},"HALDIA",88.092874,22.036944,3327]]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":2829,"high":0},"PARADIP",86.68,20.26,3466]]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":8,"high":0},"type":"r"}]}
+++debug Connection [5065][] C: RUN COMMIT {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [5065][] C: RESET
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [5065][] acquired from the pool
+++debug Connection [4837][] acquired from the pool
+++debug Connection [5065][] C: RUN BEGIN {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [5065][] C: RUN MATCH (p:Port)
        WHERE p.port_id = 1847
        RETURN p.port_id, p.name, p.longitude, p.latitude LIMIT 1 {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] C: RUN BEGIN {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [4837][] C: RUN MATCH (p:Port)
        WHERE p.port_id = 300
        RETURN p.port_id, p.name, p.longitude, p.latitude LIMIT 1 {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":1,"high":0},"fields":["p.port_id","p.name","p.longitude","p.latitude"]}]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":1847,"high":0},"PORT HEDLAND",118.5769444,-20.31750002]]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"r"}]}
+++debug Connection [5065][] C: RUN COMMIT {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":["p.port_id","p.name","p.longitude","p.latitude"]}]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":300,"high":0},"KASHIMA",140.6833,35.909345]]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"r"}]}
+++debug Connection [4837][] C: RUN COMMIT {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [5065][] C: RESET
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [4837][] C: RESET
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [5065][] acquired from the pool
+++debug Connection [5065][] C: RUN BEGIN {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [5065][] C: RUN WITH point({longitude:118.5769444, latitude:-20.31750002}) AS p, 5000 as distance_nm
        CALL spatial.withinDistance('all_connected_ports', p, distance_nm*1.852)
          YIELD node, distance
          RETURN node.port_id AS port_id, node.name, node.type AS name, distance/1.852 AS distance, node.longitude, node.latitude
          LIMIT 1 {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [4837][] acquired from the pool
+++debug Connection [4837][] C: RUN BEGIN {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [4837][] C: RUN WITH point({longitude:140.6833, latitude:35.909345}) AS p, 5000 as distance_nm
        CALL spatial.withinDistance('all_connected_ports', p, distance_nm*1.852)
          YIELD node, distance
          RETURN node.port_id AS port_id, node.name, node.type AS name, distance/1.852 AS distance, node.longitude, node.latitude
          LIMIT 1 {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":2,"high":0},"fields":["port_id","node.name","name","distance","node.longitude","node.latitude"]}]}
+++debug Connection [5065][] S: RECORD {"signature":113,"fields":[[{"low":1803,"high":0},"DAMPIER","P",107.67802056701662,116.695,-20.64666669]]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"rw"}]}
+++debug Connection [5065][] C: RUN COMMIT {}
+++debug Connection [5065][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":3,"high":0},"fields":["port_id","node.name","name","distance","node.longitude","node.latitude"]}]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":2526,"high":0},"CHIBA","P",36.87185405565536,140.059187,35.562222]]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":0,"high":0},"type":"rw"}]}
+++debug Connection [4837][] C: RUN COMMIT {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [5065][] C: RESET
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [4837][] C: RESET
+++debug Connection [5065][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [5065][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [4837][] acquired from the pool
+++debug Connection [4837][] C: RUN BEGIN {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [4837][] C: RUN MATCH (start:Port{port_id: 1803}), (end:Port{port_id: 2526})
          CALL algo.shortestPath.astar.stream(start, end, 'NM', 'latitude', 'longitude',{relationshipQuery:'DISTANCE',direction:'BOTH'})
          YIELD nodeId, cost
          MATCH (route:Port) WHERE ID(route)=nodeId
          RETURN route.port_id AS port_id, route.name AS name, route.longitude AS longitude, route.latitude AS latitude, cost AS nm {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":0,"high":0},"fields":["port_id","name","longitude","latitude","nm"]}]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":1803,"high":0},"DAMPIER",116.695,-20.64666669,0]]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":4001,"high":0},"KOLANA AREA",125.2411,-8.2211,912]]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":3980,"high":0},"EAST MANGOLE",126.5572,-1.8558,1303]]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":8238,"high":0},"WAYPOINT 88",128.3052778,3.96861111,1667]]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":8239,"high":0},"WAYPOINT 89",131.2980556,12.76305556,2221]]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":8240,"high":0},"WAYPOINT 90",134.7,22.32027778,2824]]}
+++debug Connection [4837][] S: RECORD {"signature":113,"fields":[[{"low":2526,"high":0},"CHIBA",140.059187,35.562222,3671]]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_consumed_after":{"low":7,"high":0},"type":"r"}]}
+++debug Connection [4837][] C: RUN COMMIT {}
+++debug Connection [4837][] C: PULL_ALL
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"result_available_after":{"low":1,"high":0},"fields":[]}]}
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{"bookmark":"neo4j:bookmark:v1:tx1637"}]}
+++debug Connection [4837][] C: RESET
+++debug Connection [4837][] S: SUCCESS {"signature":112,"fields":[{}]}
+++debug Connection [4837][] released to the pool neo4j-prod.theofe.com:7687
+++debug Connection [4837][] closing
+++debug Connection [5065][] closing
+++debug Connection [5107][] closing
+++debug Connection [4847][] closing
+++debug Connection [4875][] closing
+++debug Connection [5098][] closing
+++debug Connection [5163][] closing
+++debug Connection [5054][] closing
+++debug Connection [4991][] closing
+++debug Connection [5041][] closing
+++debug Connection [4989][] closing
+++debug Connection [5181][] closing
+++debug Connection [4882][] closing
+++debug Connection [5205][] closing
+++debug Connection [4933][] closing
+++debug Connection [5293][] closing
+++debug Connection [5131][] closing
+++debug Connection [4853][] closing
+++debug Connection [5270][] closing
+++debug Connection [5123][] closing
+++debug Connection [5105][] closing
+++debug Connection [5149][] closing
+++debug Connection [5028][] closing
+++debug Connection [4769][] closing
+++debug Connection [5238][] closing
+++debug Connection [4943][] closing
+++debug Connection [5225][] closing
+++debug Connection [5191][] closing
+++debug Connection [5069][] closing
+++debug Connection [4755][] closing
+++debug Connection [5231][] closing
+++debug Connection [5111][] closing
+++debug Connection [4798][] closing
+++debug Connection [5119][] closing
+++debug Connection [4783][] closing
+++debug Connection [4987][] closing
+++debug Connection [5023][] closing
+++debug Connection [5172][] closing
+++debug Connection [4947][] closing
+++debug Connection [5243][] closing
+++debug Connection [4885][] closing
+++debug Connection [4838][] closing
+++debug Connection [5236][] closing
+++debug Connection [5063][] closing
+++debug Connection [4969][] closing
+++debug Connection [5085][] closing
+++debug Connection [5277][] closing
+++debug Connection [5242][] closing
+++debug Connection [4797][] closing
+++debug Connection [4893][] closing
+++debug Connection [5173][] closing
+++debug Connection [5144][] closing
+++debug Connection [5228][] closing
+++debug Connection [4945][] closing
+++debug Connection [5200][] closing
+++debug Connection [5095][] closing
+++debug Connection [5250][] closing
+++debug Connection [5083][] closing
+++debug Connection [4753][] closing
+++debug Connection [4821][] closing
+++debug Connection [5271][] closing
+++debug Connection [5142][] closing
+++debug Connection [5229][] closing
+++debug Connection [5057][] closing
+++debug Connection [5206][] closing
+++debug Connection [5284][] closing
+++debug Connection [4889][] closing
+++debug Connection [4972][] closing
+++debug Connection [5143][] closing
+++debug Connection [5298][] closing
+++debug Connection [4951][] closing
+++debug Connection [4863][] closing
+++debug Connection [5244][] closing
+++debug Connection [4918][] closing
+++debug Connection [5079][] closing
+++debug Connection [5087][] closing
+++debug Connection [5179][] closing
+++debug Connection [5138][] closing
+++debug Connection [4934][] closing
+++debug Connection [5040][] closing
+++debug Connection [5021][] closing
+++debug Connection [4845][] closing
+++debug Connection [5137][] closing
+++debug Connection [4983][] closing
+++debug Connection [5022][] closing
+++debug Connection [5253][] closing
+++debug Connection [5162][] closing
+++debug Connection [5257][] closing
+++debug Connection [5100][] closing
+++debug Connection [4940][] closing
+++debug Connection [4976][] closing
+++debug Connection [5068][] closing
+++debug Connection [5121][] closing
+++debug Connection [5067][] closing
+++debug Connection [5248][] closing
+++debug Connection [4785][] closing
+++debug Connection [5189][] closing
+++debug Connection [5050][] closing
+++debug Connection [5188][] closing
+++debug Connection [5064][] closing
+++debug Connection [4879][] closing
+++debug Connection [4811][] closing
+++debug Connection [5033][] closing
+++debug Connection [4862][] closing
+++debug Connection [5212][] closing
+++debug Connection [5286][] closing
+++debug Connection [5136][] closing
+++debug Connection [5296][] closing
+++debug Connection [5195][] closing
+++debug Connection [5093][] closing
+++debug Connection [5249][] closing
+++debug Connection [4971][] closing
+++debug Connection [5237][] closing
+++debug Connection [4924][] closing
+++debug Connection [4985][] closing
+++debug Connection [5091][] closing
+++debug Connection [5184][] closing
+++debug Connection [4999][] closing
+++debug Connection [5132][] closing
+++debug Connection [4923][] closing
+++debug Connection [5241][] closing
+++debug Connection [5226][] closing
+++debug Connection [5036][] closing
+++debug Connection [5013][] closing
+++debug Connection [4939][] closing
+++debug Connection [4761][] closing
+++debug Connection [5099][] closing
+++debug Connection [5166][] closing
+++debug Connection [4791][] closing
+++debug Connection [5287][] closing
+++debug Connection [4799][] closing
+++debug Connection [5077][] closing
+++debug Connection [5224][] closing
+++debug Connection [4888][] closing
+++debug Connection [5274][] closing
+++debug Connection [5045][] closing
+++debug Connection [4895][] closing
+++debug Connection [5160][] closing
+++debug Connection [5300][] closing
+++debug Connection [5019][] closing
+++debug Connection [5139][] closing
+++debug Connection [5075][] closing
+++debug Connection [5256][] closing
+++debug Connection [5275][] closing
+++debug Connection [5127][] closing
+++debug Connection [4817][] closing
+++debug Connection [5027][] closing
+++debug Connection [4829][] closing
+++debug Connection [5246][] closing
+++debug Connection [4995][] closing
+++debug Connection [4927][] closing
+++debug Connection [5213][] closing
+++debug Connection [5217][] closing
+++debug Connection [4855][] closing
+++debug Connection [5015][] closing
+++debug Connection [4911][] closing
+++debug Connection [5199][] closing
+++debug Connection [5010][] closing
+++debug Connection [5235][] closing
+++debug Connection [5183][] closing
+++debug Connection [5140][] closing
+++debug Connection [5285][] closing
+++debug Connection [4955][] closing
+++debug Connection [5268][] closing
+++debug Connection [4865][] closing
+++debug Connection [5084][] closing
+++debug Connection [5158][] closing
+++debug Connection [5081][] closing
+++debug Connection [5261][] closing
+++debug Connection [5090][] closing
+++debug Connection [5204][] closing
+++debug Connection [5202][] closing
+++debug Connection [4763][] closing
+++debug Connection [5197][] closing
+++debug Connection [5214][] closing
+++debug Connection [5251][] closing
+++debug Connection [5103][] closing
+++debug Connection [5034][] closing
+++debug Connection [5109][] closing
+++debug Connection [5146][] closing
+++debug Connection [5110][] closing
+++debug Connection [4773][] closing
+++debug Connection [5283][] closing
+++debug Connection [4937][] closing
+++debug Connection [4905][] closing
+++debug Connection [5047][] closing
+++debug Connection [5278][] closing
+++debug Connection [5207][] closing
+++debug Connection [4953][] closing
+++debug Connection [5164][] closing
+++debug Connection [5182][] closing
+++debug Connection [5153][] closing
+++debug Connection [5210][] closing
+++debug Connection [5301][] closing
+++debug Connection [4877][] closing
+++debug Connection [4870][] closing
+++debug Connection [5305][] closing
+++debug Connection [4914][] closing
+++debug Connection [4997][] closing
+++debug Connection [5252][] closing
+++debug Connection [5170][] closing
+++debug Connection [4903][] closing
+++debug Connection [5096][] closing
+++debug Connection [4809][] closing
+++debug Connection [5177][] closing
+++debug Connection [4977][] closing
+++debug Connection [5009][] closing
+++debug Connection [4965][] closing
+++debug Connection [5082][] closing
+++debug Connection [5156][] closing
+++debug Connection [4781][] closing
+++debug Connection [5280][] closing
+++debug Connection [5263][] closing
+++debug Connection [4803][] closing
+++debug Connection [4757][] closing
+++debug Connection [5117][] closing
+++debug Connection [4979][] closing
+++debug Connection [5029][] closing
+++debug Connection [5151][] closing
+++debug Connection [5066][] closing
+++debug Connection [4946][] closing
+++debug Connection [4941][] closing
+++debug Connection [5154][] closing
+++debug Connection [5279][] closing
+++debug Connection [5192][] closing
+++debug Connection [5048][] closing
+++debug Connection [5295][] closing
+++debug Connection [4871][] closing
+++debug Connection [5043][] closing
+++debug Connection [5032][] closing
+++debug Connection [5187][] closing
+++debug Connection [5178][] closing
+++debug Connection [4892][] closing
+++debug Connection [4819][] closing
+++debug Connection [5220][] closing
+++debug Connection [5070][] closing
+++debug Connection [5059][] closing
+++debug Connection [5294][] closing
+++debug Connection [5303][] closing
+++debug Connection [4967][] closing
+++debug Connection [5258][] closing
+++debug Connection [5092][] closing
+++debug Connection [4827][] closing
+++debug Connection [4857][] closing
+++debug Connection [4975][] closing
+++debug Connection [5001][] closing
+++debug Connection [5147][] closing
+++debug Connection [5233][] closing
+++debug Connection [4916][] closing
+++debug Connection [5171][] closing
+++debug Connection [4887][] closing
+++debug Connection [5168][] closing
+++debug Connection [5194][] closing
+++debug Connection [4830][] closing
+++debug Connection [4897][] closing
+++debug Connection [5097][] closing
+++debug Connection [4861][] closing
+++debug Connection [4928][] closing
+++debug Connection [5130][] closing
+++debug Connection [5190][] closing
+++debug Connection [4765][] closing
+++debug Connection [4986][] closing
+++debug Connection [4922][] closing
+++debug Connection [4823][] closing
+++debug Connection [4751][] closing
+++debug Connection [4831][] closing
+++debug Connection [4793][] closing
+++debug Connection [5254][] closing
+++debug Connection [5122][] closing
+++debug Connection [5165][] closing
+++debug Connection [5055][] closing
+++debug Connection [5129][] closing
+++debug Connection [5014][] closing
+++debug Connection [4899][] closing
+++debug Connection [4919][] closing
+++debug Connection [5289][] closing
+++debug Connection [5234][] closing
+++debug Connection [4787][] closing
+++debug Connection [5211][] closing
+++debug Connection [4835][] closing
+++debug Connection [4907][] closing
+++debug Connection [5037][] closing
+++debug Connection [5152][] closing
+++debug Connection [5255][] closing
+++debug Connection [4801][] closing
+++debug Connection [5150][] closing
+++debug Connection [4833][] closing
+++debug Connection [4901][] closing
+++debug Connection [5176][] closing
+++debug Connection [4949][] closing
+++debug Connection [5025][] closing
+++debug Connection [4805][] closing
+++debug Connection [5201][] closing
+++debug Connection [5011][] closing
+++debug Connection [5222][] closing
+++debug Connection [5219][] closing
+++debug Connection [5049][] closing
+++debug Connection [5265][] closing
+++debug Connection [5053][] closing
+++debug Connection [4964][] closing
+++debug Connection [5272][] closing
+++debug Connection [4930][] closing
+++debug Connection [5240][] closing
+++debug Connection [5020][] closing
+++debug Connection [4891][] closing
+++debug Connection [4921][] closing
+++debug Connection [4968][] closing
+++debug Connection [5203][] closing
+++debug Connection [5196][] closing
+++debug Connection [5145][] closing
+++debug Connection [5157][] closing
+++debug Connection [5031][] closing
+++debug Connection [5161][] closing
+++debug Connection [5056][] closing
+++debug Connection [5113][] closing
+++debug Connection [5273][] closing
+++debug Connection [5223][] closing
+++debug Connection [5267][] closing
+++debug Connection [4917][] closing
+++debug Connection [5218][] closing
+++debug Connection [4825][] closing
+++debug Connection [4815][] closing
+++debug Connection [4929][] closing
+++debug Connection [5175][] closing
+++debug Connection [5276][] closing
+++debug Connection [4779][] closing
+++debug Connection [5304][] closing
+++debug Connection [5262][] closing
+++debug Connection [5088][] closing
+++debug Connection [5247][] closing
+++debug Connection [4849][] closing
+++debug Connection [5185][] closing
+++debug Connection [5089][] closing
+++debug Connection [5058][] closing
+++debug Connection [5292][] closing
+++debug Connection [4883][] closing
+++debug Connection [5159][] closing
+++debug Connection [4942][] closing
+++debug Connection [5141][] closing
+++debug Connection [4795][] closing
+++debug Connection [4851][] closing
+++debug Connection [5101][] closing
+++debug Connection [4978][] closing
+++debug Connection [5282][] closing
+++debug Connection [5155][] closing
+++debug Connection [5281][] closing
+++debug Connection [5073][] closing
+++debug Connection [4931][] closing
+++debug Connection [5297][] closing
+++debug Connection [5216][] closing
+++debug Connection [4915][] closing
+++debug Connection [4938][] closing
+++debug Connection [4813][] closing
+++debug Connection [4908][] closing
+++debug Connection [4852][] closing
+++debug Connection [5291][] closing
+++debug Connection [5126][] closing
+++debug Connection [5128][] closing
+++debug Connection [4909][] closing
+++debug Connection [4959][] closing
+++debug Connection [5114][] closing
+++debug Connection [5245][] closing
+++debug Connection [5174][] closing
+++debug Connection [5232][] closing
+++debug Connection [5118][] closing
+++debug Connection [5030][] closing
+++debug Connection [5239][] closing
+++debug Connection [4789][] closing
+++debug Connection [5039][] closing
+++debug Connection [4993][] closing
+++debug Connection [5167][] closing
+++debug Connection [4961][] closing
+++debug Connection [4981][] closing
+++debug Connection [5186][] closing
+++debug Connection [5094][] closing
+++debug Connection [4767][] closing
+++debug Connection [5061][] closing
+++debug Connection [4963][] closing
+++debug Connection [4807][] closing
+++debug Connection [4873][] closing
+++debug Connection [4957][] closing
+++debug Connection [5259][] closing
+++debug Connection [4864][] closing
+++debug Connection [5290][] closing
+++debug Connection [4858][] closing
+++debug Connection [4843][] closing
+++debug Connection [5051][] closing
+++debug Connection [5133][] closing
+++debug Connection [5071][] closing
+++debug Connection [4759][] closing
+++debug Connection [4771][] closing
+++debug Connection [5135][] closing
+++debug Connection [5017][] closing
+++debug Connection [5269][] closing
+++debug Connection [5115][] closing
+++debug Connection [4777][] closing
+++debug Connection [5104][] closing
+++debug Connection [5046][] closing
+++debug Connection [4839][] closing
+++debug Connection [4913][] closing
+++debug Connection [5065][] closed
+++debug Connection [4847][] closed
+++debug Connection [4989][] closed
+++debug Connection [5163][] closed
+++debug Connection [5181][] closed
+++debug Connection [5107][] closed
+++debug Connection [5054][] closed
+++debug Connection [4837][] closed
+++debug Connection [5098][] closed
+++debug Connection [4991][] closed
+++debug Connection [4875][] closed
+++debug Connection [5041][] closed
+++debug Connection [4882][] closed
+++debug Connection [5205][] closed
+++debug Connection [4933][] closed
+++debug Connection [5293][] closed
+++debug Connection [5131][] closed
+++debug Connection [4853][] closed
+++debug Connection [5270][] closed
+++debug Connection [5123][] closed
+++debug Connection [5105][] closed
+++debug Connection [5149][] closed
+++debug Connection [5028][] closed
+++debug Connection [4769][] closed
+++debug Connection [5238][] closed
+++debug Connection [4943][] closed
+++debug Connection [5225][] closed
+++debug Connection [5191][] closed
+++debug Connection [5069][] closed
+++debug Connection [4755][] closed
+++debug Connection [5231][] closed
+++debug Connection [5111][] closed
+++debug Connection [4798][] closed
+++debug Connection [4783][] closed
+++debug Connection [5119][] closed
+++debug Connection [4987][] closed
+++debug Connection [5023][] closed
+++debug Connection [5172][] closed
+++debug Connection [4947][] closed
+++debug Connection [5243][] closed
+++debug Connection [4885][] closed
+++debug Connection [4838][] closed
+++debug Connection [5236][] closed
+++debug Connection [5063][] closed
+++debug Connection [4969][] closed
+++debug Connection [5085][] closed
+++debug Connection [5277][] closed
+++debug Connection [5242][] closed
+++debug Connection [4797][] closed
+++debug Connection [4893][] closed
+++debug Connection [5173][] closed
+++debug Connection [5144][] closed
+++debug Connection [5228][] closed
+++debug Connection [4945][] closed
+++debug Connection [5200][] closed
+++debug Connection [5095][] closed
+++debug Connection [5250][] closed
+++debug Connection [5083][] closed
+++debug Connection [4753][] closed
+++debug Connection [4821][] closed
+++debug Connection [5271][] closed
+++debug Connection [5142][] closed
+++debug Connection [5229][] closed
+++debug Connection [5057][] closed
+++debug Connection [5206][] closed
+++debug Connection [4889][] closed
+++debug Connection [5284][] closed
+++debug Connection [4972][] closed
+++debug Connection [5143][] closed
+++debug Connection [5298][] closed
+++debug Connection [4951][] closed
+++debug Connection [4863][] closed
+++debug Connection [5244][] closed
+++debug Connection [4918][] closed
+++debug Connection [5179][] closed
+++debug Connection [5087][] closed
+++debug Connection [5079][] closed
+++debug Connection [5138][] closed
+++debug Connection [4934][] closed
+++debug Connection [5040][] closed
+++debug Connection [5021][] closed
+++debug Connection [4845][] closed
+++debug Connection [5137][] closed
+++debug Connection [4983][] closed
+++debug Connection [5022][] closed
+++debug Connection [5253][] closed
+++debug Connection [5162][] closed
+++debug Connection [5257][] closed
+++debug Connection [5100][] closed
+++debug Connection [4940][] closed
+++debug Connection [4976][] closed
+++debug Connection [5068][] closed
+++debug Connection [5067][] closed
+++debug Connection [5121][] closed
+++debug Connection [5248][] closed
+++debug Connection [4785][] closed
+++debug Connection [5189][] closed
+++debug Connection [5050][] closed
+++debug Connection [5188][] closed
+++debug Connection [5064][] closed
+++debug Connection [4879][] closed
+++debug Connection [4811][] closed
+++debug Connection [5033][] closed
+++debug Connection [4862][] closed
+++debug Connection [5212][] closed
+++debug Connection [5286][] closed
+++debug Connection [5136][] closed
+++debug Connection [5296][] closed
+++debug Connection [5195][] closed
+++debug Connection [5093][] closed
+++debug Connection [5237][] closed
+++debug Connection [5091][] closed
+++debug Connection [5184][] closed
+++debug Connection [4924][] closed
+++debug Connection [4999][] closed
+++debug Connection [4971][] closed
+++debug Connection [5249][] closed
+++debug Connection [4985][] closed
+++debug Connection [5132][] closed
+++debug Connection [4923][] closed
+++debug Connection [5241][] closed
+++debug Connection [5226][] closed
+++debug Connection [5036][] closed
+++debug Connection [5013][] closed
+++debug Connection [4939][] closed
+++debug Connection [4761][] closed
+++debug Connection [5099][] closed
+++debug Connection [5166][] closed
+++debug Connection [4791][] closed
+++debug Connection [5287][] closed
+++debug Connection [4799][] closed
+++debug Connection [5077][] closed
+++debug Connection [5224][] closed
+++debug Connection [4888][] closed
+++debug Connection [5274][] closed
+++debug Connection [5045][] closed
+++debug Connection [4895][] closed
+++debug Connection [5160][] closed
+++debug Connection [5300][] closed
+++debug Connection [5019][] closed
+++debug Connection [5139][] closed
+++debug Connection [5075][] closed
+++debug Connection [5256][] closed
+++debug Connection [5275][] closed
+++debug Connection [5127][] closed
+++debug Connection [4817][] closed
+++debug Connection [5027][] closed
+++debug Connection [4829][] closed
+++debug Connection [5246][] closed
+++debug Connection [4927][] closed
+++debug Connection [4995][] closed
+++debug Connection [5213][] closed
+++debug Connection [5217][] closed
+++debug Connection [4855][] closed
+++debug Connection [5015][] closed
+++debug Connection [4911][] closed
+++debug Connection [5199][] closed
+++debug Connection [5010][] closed
+++debug Connection [5235][] closed
+++debug Connection [5183][] closed
+++debug Connection [5140][] closed
+++debug Connection [5285][] closed
+++debug Connection [4955][] closed
+++debug Connection [5268][] closed
+++debug Connection [4865][] closed
+++debug Connection [5084][] closed
+++debug Connection [5158][] closed
+++debug Connection [5081][] closed
+++debug Connection [5261][] closed
+++debug Connection [5090][] closed
+++debug Connection [5204][] closed
+++debug Connection [5202][] closed
+++debug Connection [4763][] closed
+++debug Connection [5197][] closed
+++debug Connection [5214][] closed
+++debug Connection [5251][] closed
+++debug Connection [5103][] closed
+++debug Connection [5034][] closed
+++debug Connection [5109][] closed
+++debug Connection [5146][] closed
+++debug Connection [5110][] closed
+++debug Connection [4773][] closed
+++debug Connection [5283][] closed
+++debug Connection [4937][] closed
+++debug Connection [4905][] closed
+++debug Connection [5047][] closed
+++debug Connection [5278][] closed
+++debug Connection [5207][] closed
+++debug Connection [4953][] closed
+++debug Connection [5164][] closed
+++debug Connection [5182][] closed
+++debug Connection [5153][] closed
+++debug Connection [5210][] closed
+++debug Connection [5301][] closed
+++debug Connection [4877][] closed
+++debug Connection [4870][] closed
+++debug Connection [5305][] closed
+++debug Connection [4914][] closed
+++debug Connection [4997][] closed
+++debug Connection [5252][] closed
+++debug Connection [5170][] closed
+++debug Connection [4903][] closed
+++debug Connection [5096][] closed
+++debug Connection [4809][] closed
+++debug Connection [5177][] closed
+++debug Connection [4977][] closed
+++debug Connection [5009][] closed
+++debug Connection [4965][] closed
+++debug Connection [5082][] closed
+++debug Connection [5156][] closed
+++debug Connection [4781][] closed
+++debug Connection [5280][] closed
+++debug Connection [5263][] closed
+++debug Connection [4803][] closed
+++debug Connection [4757][] closed
+++debug Connection [5117][] closed
+++debug Connection [4979][] closed
+++debug Connection [5029][] closed
+++debug Connection [5151][] closed
+++debug Connection [5066][] closed
+++debug Connection [4946][] closed
+++debug Connection [4941][] closed
+++debug Connection [5154][] closed
+++debug Connection [5279][] closed
+++debug Connection [5192][] closed
+++debug Connection [5048][] closed
+++debug Connection [5295][] closed
+++debug Connection [4871][] closed
+++debug Connection [5043][] closed
+++debug Connection [5032][] closed
+++debug Connection [5187][] closed
+++debug Connection [5178][] closed
+++debug Connection [4892][] closed
+++debug Connection [4819][] closed
+++debug Connection [5220][] closed
+++debug Connection [5070][] closed
+++debug Connection [5059][] closed
+++debug Connection [5294][] closed
+++debug Connection [5303][] closed
+++debug Connection [4967][] closed
+++debug Connection [5258][] closed
+++debug Connection [5092][] closed
+++debug Connection [4827][] closed
+++debug Connection [4857][] closed
+++debug Connection [5001][] closed
+++debug Connection [4975][] closed
+++debug Connection [5147][] closed
+++debug Connection [5233][] closed
+++debug Connection [4916][] closed
+++debug Connection [5171][] closed
+++debug Connection [4887][] closed
+++debug Connection [5194][] closed
+++debug Connection [5168][] closed
+++debug Connection [4830][] closed
+++debug Connection [4897][] closed
+++debug Connection [5097][] closed
+++debug Connection [4861][] closed
+++debug Connection [5130][] closed
+++debug Connection [4928][] closed
+++debug Connection [5190][] closed
+++debug Connection [4765][] closed
+++debug Connection [4986][] closed
+++debug Connection [4922][] closed
+++debug Connection [4823][] closed
+++debug Connection [4751][] closed
+++debug Connection [4831][] closed
+++debug Connection [4793][] closed
+++debug Connection [5254][] closed
+++debug Connection [5122][] closed
+++debug Connection [5165][] closed
+++debug Connection [5129][] closed
+++debug Connection [5055][] closed
+++debug Connection [5014][] closed
+++debug Connection [4899][] closed
+++debug Connection [4919][] closed
+++debug Connection [5289][] closed
+++debug Connection [5234][] closed
+++debug Connection [4787][] closed
+++debug Connection [5211][] closed
+++debug Connection [4835][] closed
+++debug Connection [4907][] closed
+++debug Connection [5255][] closed
+++debug Connection [5152][] closed
+++debug Connection [5037][] closed
+++debug Connection [4801][] closed
+++debug Connection [5150][] closed
+++debug Connection [4833][] closed
+++debug Connection [4901][] closed
+++debug Connection [5176][] closed
+++debug Connection [4949][] closed
+++debug Connection [5025][] closed
+++debug Connection [4805][] closed
+++debug Connection [5201][] closed
+++debug Connection [5011][] closed
+++debug Connection [5222][] closed
+++debug Connection [5219][] closed
+++debug Connection [5265][] closed
+++debug Connection [5049][] closed
+++debug Connection [5053][] closed
+++debug Connection [4964][] closed
+++debug Connection [5272][] closed
+++debug Connection [4930][] closed
+++debug Connection [5240][] closed
+++debug Connection [5020][] closed
+++debug Connection [4891][] closed
+++debug Connection [4921][] closed
+++debug Connection [4968][] closed
+++debug Connection [5203][] closed
+++debug Connection [5196][] closed
+++debug Connection [5145][] closed
+++debug Connection [5157][] closed
+++debug Connection [5223][] closed
+++debug Connection [4825][] closed
+++debug Connection [4917][] closed
+++debug Connection [5267][] closed
+++debug Connection [4929][] closed
+++debug Connection [5175][] closed
+++debug Connection [5304][] closed
+++debug Connection [4815][] closed
+++debug Connection [5262][] closed
+++debug Connection [5218][] closed
+++debug Connection [5031][] closed
+++debug Connection [5113][] closed
+++debug Connection [5276][] closed
+++debug Connection [5056][] closed
+++debug Connection [5273][] closed
+++debug Connection [4779][] closed
+++debug Connection [5161][] closed
+++debug Connection [5088][] closed
+++debug Connection [5247][] closed
+++debug Connection [4849][] closed
+++debug Connection [5185][] closed
+++debug Connection [5089][] closed
+++debug Connection [5058][] closed
+++debug Connection [5292][] closed
+++debug Connection [4883][] closed
+++debug Connection [5159][] closed
+++debug Connection [4942][] closed
+++debug Connection [5141][] closed
+++debug Connection [4795][] closed
+++debug Connection [4851][] closed
+++debug Connection [5101][] closed
+++debug Connection [4978][] closed
+++debug Connection [5282][] closed
+++debug Connection [5155][] closed
+++debug Connection [5281][] closed
+++debug Connection [5073][] closed
+++debug Connection [4931][] closed
+++debug Connection [5297][] closed
+++debug Connection [5216][] closed
+++debug Connection [4915][] closed
+++debug Connection [4908][] closed
+++debug Connection [4813][] closed
+++debug Connection [4852][] closed
+++debug Connection [5291][] closed
+++debug Connection [4938][] closed
+++debug Connection [5126][] closed
+++debug Connection [5128][] closed
+++debug Connection [4909][] closed
+++debug Connection [4959][] closed
+++debug Connection [5114][] closed
+++debug Connection [5245][] closed
+++debug Connection [5174][] closed
+++debug Connection [5232][] closed
+++debug Connection [5118][] closed
+++debug Connection [5030][] closed
+++debug Connection [5239][] closed
+++debug Connection [4789][] closed
+++debug Connection [5039][] closed
+++debug Connection [4993][] closed
+++debug Connection [5167][] closed
+++debug Connection [4961][] closed
+++debug Connection [4981][] closed
+++debug Connection [5186][] closed
+++debug Connection [5094][] closed
+++debug Connection [4767][] closed
+++debug Connection [5061][] closed
+++debug Connection [4963][] closed
+++debug Connection [4807][] closed
+++debug Connection [4873][] closed
+++debug Connection [4957][] closed
+++debug Connection [5259][] closed
+++debug Connection [4864][] closed
+++debug Connection [5290][] closed
+++debug Connection [4858][] closed
+++debug Connection [4843][] closed
+++debug Connection [5051][] closed
+++debug Connection [5133][] closed
+++debug Connection [5071][] closed
+++debug Connection [4759][] closed
+++debug Connection [4771][] closed
+++debug Connection [5135][] closed
+++debug Connection [5017][] closed
+++debug Connection [5269][] closed
+++debug Connection [5115][] closed
+++debug Connection [4777][] closed
+++debug Connection [5104][] closed
+++debug Connection [5046][] closed
+++debug Connection [4839][] closed
+++debug Connection [4913][] closed
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 252): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 253): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 254): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 255): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 256): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 257): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 258): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 259): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 260): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 261): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 262): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 263): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 264): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 265): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 266): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 267): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 268): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 301): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 302): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 303): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 304): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 305): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 306): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 307): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 308): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 309): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 310): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 311): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 312): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 313): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 314): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 315): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 316): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 329): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 330): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 331): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 332): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 333): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 334): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 349): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 350): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 351): Neo4jError: Connection acquisition timed out in 60000 ms.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 352): Neo4jError: Connection acquisition timed out in 60000 ms.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:29 (8 by maintainers)

github_iconTop GitHub Comments

10reactions
zhenlineocommented, Nov 21, 2019

Hi @elielr01 , Thanks for the detailed logging. I think this is actually caused by this bug on the server side:

Failure message is not flushed when no free work thread is available When the worker thread pool is full, bolt server will fail new requests from clients. However this error is not flushed correctly to clients. This fix makes bolt server to first flush the failure message to clients and then close the connection to enforce the clients failing fast without waiting for more responses.

The bug was found in 3.5 and then fixed in 3.5.6.

From your logs, it looks like connection 0 sends messages to server but none of the messages was answered by the server. Your later print shows the connection 0 get rejected by server with a NoThreadAvailable error. This error give one reply to one of the sent messages of the driver, but the driver are waiting for more. However server think the conversation is already done, so the server does not pick up any more messages from this connection. Thus we end up with this driver connection is hold up in the middle.

I just verified 3.4 servers all suffers from this bug. I would suggest you to upgrade to latest 3.5 servers. If you cannot upgrade to latest 3.5, and you have a support contract with Neo4j, you shall consider raising a support case there.

Thanks again for the detailed logging to help us understand this issue! Cheers, Zhen

3reactions
zhenlineocommented, May 3, 2019

Hi @elielr01,

We identified one js bug that could cause your problem. The js bug is fixed in this PR https://github.com/neo4j/neo4j-javascript-driver/pull/453. We might not release the connection when the connection is failed to establish in this line.

We will do a patch release soon. It would be great if you could verify if the fix addresses your issue in the coming release.

Cheers, Zhen

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to deal with closed connections in database pool
This article explains how to overcome the "connection is closed" error that sometimes is seen on the mule logs when connecting to a...
Read more >
Why do is my sql pool connection being closed suddenly
Your sudden Pool Closed may be the result of some kind of server fault. (You didn't give us many details, so that's only...
Read more >
Connection Pool Problem - Oracle Communities
Right. At that time, if they all go inactive for 1800 seconds, it will start tearing down connections until there is a min...
Read more >
Troubleshooting connection pooling (J2C) problems in ... - IBM
If the application does not call close(), the connection is leaked and never returns to the free pool. Eventually, the pool might become...
Read more >
JIRA become unresponsive when DB connection pool i...
It is worrying that you are using up your pool, it suggests something is going wrong and not releasing connections back into the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

MaxConnectionPoolSize is not respected under heavy load

Next page

Bolt V2 should always support byte arrays