Make queued experiments initialization progress run in parallel.

For now, when we run queued experiments we would lock the whole scm during the whole initialization progress.

https://github.com/iterative/dvc/blob/29b3dc1f9ec96fbd4116f8adabaeb17c9633afb2/dvc/repo/experiments/queue/base.py#L566-L568.

This makes the initialization progress only run sequentially. We can do a more precise control on this progress to make the

1. We should only lock some of the exp refs or an exp reference namespace.
2. We should only add the lock on a few steps instead of on the whole progress.
3. We do not need to set EXEC_* branches to the before pushing them to the remote.

[karajan1001]

The queue stashes are different for different workers

Is this the way it’s supposed to work? I would think they would consume tasks from the same queue.

The queue itself is thread safety, but no one can guarantee that the stash queue is too. For example, If I use git stash list first to find the desired stash, and then want to git stash pop stash@{n} it. But in the interval time, some other processes did some operation on the stash queue.

[karajan1001]

I’m starting from the command line using that repo. Are you seeing slowdown during exp run --queue or during queue start?

It is not about slowdown but about the experiments running in sequence even if I started several workers. The reason for this is that we use an SCM lock during the initialization progress. And because 1. the workers are only reading each of the stashes, 2. The queue stashes are different for different workers. So we can make some modifications and then release the lock, and make the workers run in parallel.