ZooKeeperNetEx connection loss issue: an acquired lock seems is not released
See original GitHub issueI’ve implemented a lock with the Zookeeper with this configuration :
- DistributedLock.ZooKeeper - Version=“1.0.0”
- dotnet version 6.0
- Hosted on K8s (one pod, there is no concurrent request)
- Zookeeper server configuration on K8s :
version: “3.9” services: zk1: container_name: zk1 hostname: zk1 image: bitnami/zookeeper:3.8.0-debian-11-r57 ports: - 2181:2181 environment: - ALLOW_ANONYMOUS_LOGIN=yes - ZOO_SERVER_ID=1 - ZOO_SERVERS=0.0.0.0:2888:3888 - ZOO_MAX_CLIENT_CNXNS=500
There are several worker services inside the application, each of them working with a different lock key.
periodically it tries to accuqire the lock and do some processes. It seems they are working without problem, but after a while, I get this exception
Locking failed.Exception of type 'org.apache.zookeeper.KeeperException+ConnectionLossException' was thrown. org.apache.zookeeper.KeeperException+ConnectionLossException: Exception of type 'org.apache.zookeeper.KeeperException+ConnectionLossException' was thrown.
It seems the lock cannot be acquired because it has not been released, although there is no concurrent request for the lock key.
The LockService code in dotnet :
`
private TimeSpan _connectionTimeoutInSecond = TimeSpan.FromSeconds(30);
private TimeSpan _waitingForLockInSecond = TimeSpan.FromSeconds(30);
public async Task<LockProcessResult> DoActionWithLockAsync(string lockKey, Func<Task> func)
{
var processResult = new LockProcessResult();
try
{
var @lock = new ZooKeeperDistributedLock(lockKey, _configuration.ConnectionString, opt =>
{
opt.ConnectTimeout(_connectionTimeoutInSecond);
});
await using (var handle = await @lock.TryAcquireAsync(timeout: _waitingForLockInSecond))
{
if (handle != null)
{
// I have the lock
await func();
}
else
{
processResult.SetException(new LockAcquisitionFailedException(lockKey));
}
}
}
catch (Exception ex)
{
//I got the exceptions here
processResult.SetException(ex);
}
return processResult;
}`
I appreciate any suggestion
Issue Analytics
- State:
- Created 10 months ago
- Comments:50 (22 by maintainers)
Top GitHub Comments
Unfortunately we are getting Connection Loss sometimes, but it will be gone in a minute.
Exception of type 'org.apache.zookeeper.KeeperException+ConnectionLossException' was thrown.
@madelson we have just tested your change locally and in a K8 cluster and that code change fixed the issue - could you issue a PR for this change against the main repo?