exp: Queuing experiment causes "Unable to acquire lock" error although no lock file exists at first

Bug Report

Description

I am trying to queue an experiment. There is no .dvc/tmp/lock file and no .dvc/tmp/rwlock.lock file or any other lock file in the .dvc/tmp directory before running dvc exp run --queue. During the execution of this command, different lock files are being created (lock, exp_scm_lock, and multiple <sha>.lock files) in .dvc/tmp/. Eventually, the command fails with error message:

ERROR: Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly. Check the page <https://dvc.org/doc/user-guide/troubleshooting#lock-issue> for other possible reasons and to learn how to resolve this.

Note that I am in a larger mono-repo with a local .dvc project folder. Git operations generally seem to take a bit longer and I am just wondering if this has to do with the observed error?

Also, I previously had a queue running and stopped it with dvc queue stop. This is a long time ago (~1h). I since had uninstalled DVC version 2.31.0 and installed a newer version (tried 2.33.2 and 2.34.0, both installed on CentOS 7 using yum).

Downgrading to DVC version 2.31.0 resolved the issue.

I had set core.hardlink_lock = true in .dvc/config previously following the guidance in the error message. This setting made no difference to the observed issue.

Reproduce

Expected

Environment information

Output of dvc doctor:

$ dvc doctor

Additional Information (if any):

[aschuh-hf]

Might this issue be caused by long running commands issued by the VS Code extension? Observing it again after updating DVC to the latest version 2.35.1.

[aschuh-hf]

In https://github.com/iterative/dvc/pull/8623, we introduce a double-check to check if the process in .dvc/tmp/rwlock still exists, it can solve this issue.

I’ve seen this PR, but nonetheless I experienced this issue only with DVC versions which include this PR. Could it be that the feature is not working as intended?

Could you please elaborate what do you mean here Downgrading to DVC version 2.31.0 resolved the issue. Does it mean that when you get the error occur, and switch back to version 2.31.0 the command will success, or in 2.31.0 the problem will not be triggered?

By downgrading I mean installing an earlier version after I encountered the issue. I still had to manually delete lock files, I think, but with the older DVC version I did then no longer encounter this issue.