add: incredibly slow

Bug Report

add: incredibly slow

Description

I have a dataset that consists of videos (large files) that sit along some metadata in json files. If the metadata (json files) are updated, if the dataset directory is re-added, it seems to be rehashing everything again?

dataset

• A a_1.avi a_2.avi a.json
• B b_1.avi b_2.avi b.json …

Updating a handful of json files, dvc add of dataset takes ~1 hour to re-compute the md5 hashes? Which doesn’t make sense given all the large files are untouched, and already in a local dvc cache?

Reproduce

1. Existing DVC dataset directory with several large files
2. update metatadata json files (along side the large files)
3. dvc add dataset

Expected

would expect it to take just a few minutes; however, its taking an hour…

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 2.8.2 (pip)
---------------------------------
Platform: Python 3.8.0 on Linux-3.10.0-1160.45.1.el7.x86_64-x86_64-with-glibc2.27
Supports:
	webhdfs (fsspec = 2021.10.1),
	http (aiohttp = 3.8.0, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.0, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2021.10.1, boto3 = 1.17.106)
Cache types: hardlink, symlink
Cache directory: nfs on LEB1MLNAS.comany.com:/leb1mlnas_projects
Caches: local
Remotes: s3
Workspace directory: nfs on LEB1MLNAS.company.com:/leb1mlnas_projects
Repo: dvc, git

[wdixon]

We can close, thank you for the work and the response!

[Mastergalen]

I copied the directory to the local file system and ran the same dvc add command. This time it only took 1 minute.

Seems like it’s something to do with nfs?

EDIT: Looks like the team is already of this issue #5562