[RFC] Support multiple endpoints for load balancing and failoverSee original GitHub issue
ClickHouse supports clustering and often runs in a cluster. Typical approaches for load balancing and failover are:
- distributed table
- proxy/gateway(e.g. chproxy, traefik etc.) sitting between client and server
- or modern DNS like consul
Besides, message queues(federated or not) and customization on client-side may require to further optimize reads and writes across servers.
It would be really nice to enhance both Java client and JDBC driver to support multiple endpoints, which provides more options for people to choose. Yes, we have
BalancedClickhouseDataSource in JDBC driver, and maybe
ClickHouseCluster in Java client, but unfortunately none of them is good enough and the latter was not even tested on a clustered environment.
Endpoint - essentially a combination of host, port and protocol. It may contain additional information like status, role, tags, weight, server revision/version/timezone, and credentials for authentication.
(localhost, 8123, http)and
(127.0.0.1, 8123, http)are different endpoints.
Server - an instance of ClickHouse server, which exposes multiple endpoints for client to connect to.
clickhouse-localis a special type of server, which only exists when needed.
Cluster - a group of ClickHouse servers under same name, as described in
|1.1||Connect||an endpoint||memorize which exact server was connected to, based on
|1.2||Connect||multiple endpoints||either one of them based on load balancing policy, or multiple based on property matching(e.g. tags/status/distance)|
|1.3||Connect||a server||either one of discovered endpoints, or multiple|
|1.4||Connect||a cluster||either one of discovered endpoints, or multiple|
|2.1||Read||an endpoint||memorize which exact server was queried against, based on
|2.2||Read||multiple endpoints||either one of them based on load balancing policy, or multiple based on property matching and the query(prefer to query multiple nodes in parallel)|
|2.3||Read||a server||one of discovered endpoints, perhaps the best protocol available|
|2.4||Read||a cluster||either one of discovered endpoints, or multiple with consideration of sharding key and replicas|
|3.1||Write||an endpoint||memorize which exact server was queried against, based on
|3.2||Write||multiple endpoints||either one of them based on load balancing policy, or multiple based on property matching and the target table(prefer to write into local table)|
|3.3||Write||a server||one of discovered endpoints, perhaps the best protocol available|
|3.4||Write||a cluster||either one of discovered endpoints, or multiple with consideration of sharding key and replicas|
In order to support above user scenarios, we need to implement below features:
|1||Auto discovery||The ability to automatically discover: 1) exposed ports of a ClickHouse server; 2) servers of a cluster; 3) connections between configured endpoint and actually connected server(
|2||Load balancing||Pick endpoint from a sorted and filtered list using one of two policies: 1) first alive; and 2) round robin|
|3||Failover||Retry only when there’s network issue(happens before sending query) or failure of select query|
|4||Protocol detection||Probe given host + port to figure out the protocol - example|
|5||Health check||Besides liveness detection like ping, we need to understand “distance” between client and each endpoint|
|6||Parallel query||Query partitioned data stored on multiple servers and return consolidated response(unordered) to the caller|
|7||Dual write||Write same or distributed data into multiple endpoints. Might need to introduce configurable client-side consistency-level|
Potential changes to existing code:
- ClickHouseCluster - represents a cluster, which contains list of endpoints. Needs to rewrite the class accordingly and add background thread for auto-discovery.
- ClickHouseEndpoint - represents an endpoint. Replacement of
- ClickHouseEndpoints - represents list of endpoints. It contains methods to add
ClickHouseCluster, and a background thread for health check.
- LoadBalancingPolicy - load balancing policy can be used in
ClickHouseEndpointsfor picking an endpoint from a sorted(based on status, weight and maybe distance) and filtered(based on tags etc.) list
- JDBC URL -
- JDBC Connection Properties
Property Description Example cluster name of the cluster cluster=mycluster tags comma separated tags for categorization tags=dc1,readonly,r1s1 lbPolicy either
lbPolicy=roundRobin failover either
false(default), only retry on network issue or failure of select query
failover=true autoDiscovery either
autoDiscovery=true healthCheck either
- Created a year ago
Top GitHub Comments
Any update @dynaxis? Just trying to avoid duplicated efforts here. I’m thinking to add an alternative of
BalancedClickhouseDataSource in the coming weekend. Of course, it’s not going to be a complete implementation of all above, but just bare-bone with single load-balancing policy and no auto-discovery and fail-over.
A few things to start with:
- 1. the ability to probe secure and insecure ports(to figure out protocol) - feature 4
- 2. new classes: ClickHouseEndpoint and ClickHouseEndpoints(or ClickHouseNode, ClickHouseNodes instead for minimum changes)
- 3. ClickHouseEndpoint can carry ClickHouseConfig like timeout and buffer size etc., which will override the ones in ClickHouseClient when creating ClickHouseRequest
- 4. the ability to connect to a URL in Java client
// host=localhost, protocol=tcp, port=9440, ssl=true, sslmode=NONE client.connect(ClickHouseEndpoint.of("tcps://localhost/system")); // tcps is short for TCP secure // host=localhost, protocol=http, port=8123, ssl=false client.connect("http://localhost/system"); // port is NOT 80 as we're connecting to ClickHouse, not a web server // host=localhost, protocol=grpc, port=9100, ssl=false client.connect("localhost:9100/system"); // probe port 9100 to figure out protocol // host=localhost, protocol=http, port=8123, ssl=false, database=default client.connect("localhost"); // probe default port 8123 to figure out protocol, try more ports only when autoDiscovery is enabled // may cover below case later - cached ClickHouseEndpoints will be shared among clients // client.connect("localhost, https://localhost:443,grpc://localhost,tcp://localhost");
- 5. the ability to define nested URLs in JDBC driver
// the outer-most protocol(defaults to ANY) and parameters are defaults, which can be override by inner URL Connection conn = DriverManager.getConnection( "jdbc:ch://server1?connection_timeout=3000,grpc://server2,(tcp://server3:9000/default)/?connection_timeout=5000", "default", "default"); // change default protocol to TCP(instead of ANY), so we'll probe port 9000 for all 3 servers conn = DriverManager.getConnection("jdbc:ch:tcp://server1,server2,server3/?connection_timeout=5000", "default", "");
- 6. background thread for health check when connecting to multiple endpoints - and maybe options to set check intervals and threads(defaults to single-thread)
Please feel free to submit PR or review/merge changes later.