Cannot acquire connection from data source while using HiKari Connection Pooling

Here is the stack trace The version of clickhouse-jdbc dependency is 0.1.46

Cannot acquire connection from data source
java.lang.IllegalStateException: cannot initialize http client
        at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:73)
        at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58)
        at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50)
        at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:316)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:173)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:431)
        at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:66)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:574)
        at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I experienced this issue every now and then, after a few hours the pool runs out of connections caused the application to crash. From the stack trace, you can see that the exception is thrown by ClickHouseConnectionImpl. It works fine when connecting directly to clickhouse DB using the Clickhouse DataSource.

This is how we used to make a connection to clickhouse db( The data source was rebuilt every time for a new connection)

ClickHouseDataSource cds=ClickHouseDataSource(jdbcUrl , 
getClickHouseProperties(user,password,socketTimeout));
Connection con= cds.getConnection();

Now we use Hikari pooling,

HikariConfig config = new HikariConfig();
config.setJdbcUrl(jdbcUrl);
config.setDataSourceProperties(getClickHouseProperties(user, password, socketTimeout));
config.setPoolName("clickhouse-pool-1");
config.setDriverClassName("ru.yandex.clickhouse.ClickHouseDriver");
config.setMaximumPoolSize(connectionCount);
HikariDataSource dataSource = new HikariDataSource(config);

Here we reuse the data source every time.

[zhicwu]

Have you configured defaultMaxPerRoute(default max connection per route) and maxTotal(default max connection)? Defaults can be found in ClickHouseConnectionSettings:

    DEFAULT_MAX_PER_ROUTE("defaultMaxPerRoute", 500, ""),
    MAX_TOTAL("maxTotal", 10000, ""),

Perhaps we should disable http client pooling by default, so that we don’t have to configure two pools(Hikari and HttpClient in this case) for one simple thing.

[reimai]

Hi, got a question about ClickhouseDatasource. You’ve said:

ClickHouse internally uses PoolingHttpClientConnectionManager for setting up the HttpClient which is the base of the ClickHouse Datasource.

and should be used without external connection pools like hikari. I’m testing that on a simple app that opens two connections from the same DataSource and pings the db in each of them. And It looks like the HttpClient and it’s pool are created per ClickhouseConnectionImpl, not per ClickHouseDatasource.

A code like this (scala):

  def ping(): Boolean = {
    val res =
      Using(ds.getConnection) { connection1 =>
        Using(ds.getConnection) { connection2 =>
          val res1 = Try {
            connection1.isValid(0)
          }.getOrElse(false)
          val res2 = Try {
            connection2.isValid(0)
          }.getOrElse(false)
          res1 || res2
        }.getOrElse(false)
      }.getOrElse(false)
    reportResult(res)
    res
  }

Leaves this trace for org.apache.http.impl.conn.PoolingHttpClientConnectionManager:

  2022-02-09 21:53:33.117 Connection request: [route: {s}->myHost:8443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 10000]   
  2022-02-09 21:53:33.117 Connection leased: [id: 775][route: {s}->myHost:8443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.138 Connection released: [id: 775][route: {s}->myHost:8443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.138 Connection [id: 775][route: {s}->myHost:8443] can be kept alive for 3.0 seconds   

  2022-02-09 21:53:33.139 Connection request: [route: {s}->myHost:8443][total available: 0; route allocated: 0 of 5; total allocated: 0 of 10000]   
  2022-02-09 21:53:33.140 Connection leased: [id: 778][route: {s}->myHost:8443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.159 Connection released: [id: 778][route: {s}->myHost:8443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.159 Connection [id: 778][route: {s}->myHost:8443] can be kept alive for 3.0 seconds   

  2022-02-09 21:53:33.160 Connection request: [route: {s}->myHost:8443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.160 Connection leased: [id: 775][route: {s}->myHost:8443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.165 Connection released: [id: 775][route: {s}->myHost:8443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.165 Connection [id: 775][route: {s}->myHost:8443] can be kept alive for 3.0 seconds   

  2022-02-09 21:53:33.165 Connection request: [route: {s}->myHost:8443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.165 Connection leased: [id: 778][route: {s}->myHost:8443][total available: 0; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.170 Connection released: [id: 778][route: {s}->myHost:8443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 10000]   
  2022-02-09 21:53:33.170 Connection [id: 778][route: {s}->myHost:8443] can be kept alive for 3.0 seconds   

  2022-02-09 21:53:33.170 Connection manager is shutting down
  2022-02-09 21:53:33.170 Connection manager is shutting down
  2022-02-09 21:53:33.171 Connection manager shut down
  2022-02-09 21:53:33.170 Connection manager shut down

Here I have two connection pools not sharing their http connections with each other. Each has two requests (the first one being select timezone() on connection initialization, as I’m using default settings), which do use connections’ pools. But that means keeping the same ClickHouseConnection for the application, not ClickHouseDataSource, and sharing ClickHouseConnection does not look nice, because is has a state (e.g. could setCatalog on it).

So it looks like you do need a hikari pool to work with clickhouse-jdbc? I’m using version 0.2.6 of the driver, this issue probably references 0.2.5, but there should not be much of a difference.