checkout: use more than one core to check checksum when checking into another branch

I’m working with a few million images that sum up to ~11GB. When I run a dvc checkout, dvc checks that files haven’t changed, and if they did, it computes the new md5. This, however, could take more than 15 minutes when we are working with lots of images.

htop shows that during this process only one core is being used. Perhaps we could speed up the process using multiprocessing and distributing the check of such files across all available cores in the machine.

[efiop]

Hi @IamGianluca !

Great idea! We will need to modify collect_dir_cache method in https://github.com/iterative/dvc/blob/master/dvc/remote/local.py#L187 for that. Thanks to @mroutis , we also have a selective checkout optimization coming in 0.20.4, which should help with that, since it won’t remove unchanged files inside directories tracked by dvc. Also, I am suspecting that you are running into state db size limit(10 million), which also makes things slower. We’ve increased it to 100M in 0.20.4, which is coming today. I will notify you when it is ready, so you could give it a try to see if it makes a difference for you. We will look into parallelization soon as well.

Thank you for the feedback!

[efiop]

Currently checksums for directories are computed in multiple threads, utilizing multiple cores. Closing. Please feel free to reopen if the issue still persists.