Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
`exp run`: fails for second run on any commit in Windows environment (VS Code demo project)
See original GitHub issueBug Report
Description
It appears that experiments cannot be resumed on Windows. Bug relates to Windows only and found inside the VS Code demo project. Both Mac and Linux have none of these issues.
Reproduce
- Open the VS code demo project on a Windows machine
- checkout master
- Run
dvc exp run -v:
2021-08-09 23:51:51,773 DEBUG: Check for update is disabled.
2021-08-09 23:51:52,001 DEBUG: Detaching HEAD at 'HEAD'
2021-08-09 23:51:52,055 DEBUG: Stashing workspace
2021-08-09 23:51:52,281 DEBUG: Stashing changes in 'refs/stash'
2021-08-09 23:51:52,305 DEBUG: Restore HEAD to 'master'
2021-08-09 23:51:52,346 ERROR: unexpected error - 'cannot stash changes - there is nothing to stash.'
------------------------------------------------------------
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\main.py", line 55, in main
ret = cmd.do_run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\base.py", line 50, in do_run
return self.run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\experiments.py", line 581, in run
results = self.repo.experiments.run(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 957, in run
return run(self.repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
return f(repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\run.py", line 32, in run
return repo.experiments.reproduce_one(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 410, in reproduce_one
stash_rev = self.new(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 46, in wrapper
return f(exp, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 473, in new
return self._stash_exp(*args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 170, in _stash_exp
with self.scm.stash_workspace() as workspace:
File "C:\Python39\lib\contextlib.py", line 117, in __enter__
return next(self.gen)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\__init__.py", line 471, in stash_workspace
rev = self.stash.push(**kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\stash.py", line 40, in push
rev, reset = self.scm._stash_push( # pylint: disable=protected-access
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\__init__.py", line 343, in _backend_func
return func(*args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\backend\pygit2.py", line 374, in _stash_push
oid = self.repo.stash(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\pygit2\repository.py", line 1089, in stash
check_error(err)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\pygit2\errors.py", line 56, in check_error
raise KeyError(message)
KeyError: 'cannot stash changes - there is nothing to stash.'
------------------------------------------------------------
2021-08-09 23:51:52,588 DEBUG: Version info for developers:
DVC version: 2.5.4+63b710
---------------------------------
Platform: Python 3.9.6 on Windows-10-10.0.19042-SP0
Supports:
http (requests = 2.26.0),
https (requests = 2.26.0)
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: https
Workspace directory: NTFS on C:\
Repo: dvc (subdir), git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
- Change input in
params.yml - run
dvc exp run-> runs as expected - run
dvc exp run:
2021-08-09 23:55:06,888 DEBUG: Check for update is disabled.
2021-08-09 23:55:07,247 DEBUG: Checkpoint run from '1ea92f3' with baseline '7a957c08afdf1a3cf8276271ebe99f5f35ef321b'
2021-08-09 23:55:07,247 DEBUG: Detaching HEAD at 'HEAD'
2021-08-09 23:55:07,487 DEBUG: Stashing workspace
2021-08-09 23:55:07,720 DEBUG: Stashing changes in 'refs/stash'
2021-08-09 23:55:09,057 DEBUG: Applying stash commit 'd4d2919538a6da2f91f01bd006883718bed4f096'
2021-08-09 23:55:09,648 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
2021-08-09 23:55:09,656 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-09 23:55:09,658 DEBUG: Checking out 'data\MNIST\raw' with cache 'object md5: 8c257df187855c681f88bde92d721ccd.dir'.
2021-08-09 23:55:09,663 DEBUG: Linking directory 'data\MNIST\raw'.
2021-08-09 23:55:09,691 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-09 23:55:09,698 DEBUG: Created 'copy': .dvc\cache\.cache_type_test_file -> data\MNIST\raw\.Zr7K654VF4dewwbd36Cmuf
2021-08-09 23:55:09,708 DEBUG: Removing 'data\MNIST\raw\.Zr7K654VF4dewwbd36Cmuf'
2021-08-09 23:55:09,710 DEBUG: Removing '.dvc\cache\.cache_type_test_file'
2021-08-09 23:55:09,714 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,723 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,727 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,730 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,733 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,735 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,739 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,742 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-09 23:55:09,746 DEBUG: state save (8782108282916533484, 2ad916ff24d741373e0dafb9546e467f, 66544770) 8c257df187855c681f88bde92d721ccd.dir
2021-08-09 23:55:09,761 DEBUG: Saving information to 'data\MNIST\raw.dvc'.
Existing checkpoint experiment 'exp-a59e4' will be resumed
2021-08-09 23:55:11,421 DEBUG: Stashing changes in 'refs/exps/stash'
2021-08-09 23:55:12,926 DEBUG: Stashed experiment 'bbfb86f' with baseline '7a957c0' for future execution.
2021-08-09 23:55:13,662 DEBUG: Restoring stashed workspace
2021-08-09 23:55:13,664 DEBUG: Popping from stash 'refs/stash'
2021-08-09 23:55:13,669 DEBUG: Applying stash commit 'd4d2919538a6da2f91f01bd006883718bed4f096'
2021-08-09 23:55:14,291 DEBUG: Dropping 'refs/stash@{0}'
2021-08-09 23:55:14,309 DEBUG: Restore HEAD to 'master'
2021-08-09 23:55:14,401 DEBUG: Detaching HEAD at '1ea92f346cc48cf50edb1769bb4984d4ab9a1c3d'
2021-08-09 23:55:16,547 ERROR: unexpected error - could not rmdir 'C:/Users/Administrator/code/vscode-dvc/demo/': The process cannot access the file because it is being used by another process.
------------------------------------------------------------
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\main.py", line 55, in main
ret = cmd.do_run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\base.py", line 50, in do_run
return self.run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\experiments.py", line 581, in run
results = self.repo.experiments.run(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 957, in run
return run(self.repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
return f(repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\run.py", line 32, in run
return repo.experiments.reproduce_one(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 421, in reproduce_one
results = self._workspace_repro()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 57, in wrapper
ret = f(exp, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 774, in _workspace_repro
with self.scm.detach_head(entry.rev, force=True):
File "C:\Python39\lib\contextlib.py", line 117, in __enter__
return next(self.gen)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\__init__.py", line 443, in detach_head
self.checkout(rev, detach=True, force=force)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\__init__.py", line 343, in _backend_func
return func(*args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\backend\pygit2.py", line 168, in checkout
self.repo.checkout_tree(commit, strategy=checkout_strategy)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\pygit2\repository.py", line 373, in checkout_tree
check_error(C.git_checkout_tree(self._repo, cptr[0], copts))
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\pygit2\errors.py", line 65, in check_error
raise GitError(message)
_pygit2.GitError: could not rmdir 'C:/Users/Administrator/code/vscode-dvc/demo/': The process cannot access the file because it is being used by another process.
------------------------------------------------------------
2021-08-09 23:55:17,117 DEBUG: Version info for developers:
DVC version: 2.5.4+63b710
---------------------------------
Platform: Python 3.9.6 on Windows-10-10.0.19042-SP0
Supports:
http (requests = 2.26.0),
https (requests = 2.26.0)
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: https
Workspace directory: NTFS on C:\
Repo: dvc (subdir), git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
The above command leaves the workspace in this state (all files tracked by git appear modified):
- all subsequent runs will fail with the same message
- run
git reset --hard HEAD && dvc checkout - run
dvc exp run-> fails withKeyError: 'cannot stash changes - there is nothing to stash.'again - update
params.yml - run
dvc exp run-> fails with_pygit2.GitError: could not rmdir 'C:/Users/Administrator/code/vscode-dvc/demo/': The process cannot access the file because it is being used by another process. - remove
check_errorfromcheckout_treeinpygit2/repository.py - run
dvc exp run -v:
2021-08-10 00:03:06,934 DEBUG: Check for update is disabled.
2021-08-10 00:03:07,746 DEBUG: Checkpoint run from '1ea92f3' with baseline '7a957c08afdf1a3cf8276271ebe99f5f35ef321b'
2021-08-10 00:03:07,746 DEBUG: Detaching HEAD at 'HEAD'
2021-08-10 00:03:07,796 DEBUG: Stashing workspace
2021-08-10 00:03:08,025 DEBUG: Stashing changes in 'refs/stash'
2021-08-10 00:03:08,107 DEBUG: Applying stash commit '06880c75d1dada1f6c6eab94ba39748853a77e9f'
2021-08-10 00:03:08,334 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
2021-08-10 00:03:08,342 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:03:08,344 DEBUG: Checking out 'data\MNIST\raw' with cache 'object md5: 8c257df187855c681f88bde92d721ccd.dir'.
2021-08-10 00:03:08,345 DEBUG: Linking directory 'data\MNIST\raw'.
2021-08-10 00:03:08,355 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-10 00:03:08,362 DEBUG: Created 'copy': .dvc\cache\.cache_type_test_file -> data\MNIST\raw\.8c7ST8PbsPAAnxy8tkgqXh
2021-08-10 00:03:08,367 DEBUG: Removing 'data\MNIST\raw\.8c7ST8PbsPAAnxy8tkgqXh'
2021-08-10 00:03:08,373 DEBUG: Removing '.dvc\cache\.cache_type_test_file'
2021-08-10 00:03:08,377 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,380 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,383 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,386 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,388 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,391 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,394 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,397 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:03:08,400 DEBUG: state save (8782108282916533484, 2ad916ff24d741373e0dafb9546e467f, 66544770) 8c257df187855c681f88bde92d721ccd.dir
2021-08-10 00:03:08,421 DEBUG: Saving information to 'data\MNIST\raw.dvc'.
Modified checkpoint experiment based on 'exp-a59e4' will be created
2021-08-10 00:03:09,215 DEBUG: Stashing changes in 'refs/exps/stash'
2021-08-10 00:03:10,915 DEBUG: Stashed experiment '429b127' with baseline '7a957c0' for future execution.
2021-08-10 00:03:12,152 DEBUG: Restoring stashed workspace
2021-08-10 00:03:12,165 DEBUG: Popping from stash 'refs/stash'
2021-08-10 00:03:12,167 DEBUG: Applying stash commit '06880c75d1dada1f6c6eab94ba39748853a77e9f'
2021-08-10 00:03:12,238 DEBUG: Dropping 'refs/stash@{0}'
2021-08-10 00:03:12,331 DEBUG: Restore HEAD to 'master'
2021-08-10 00:03:12,567 DEBUG: Detaching HEAD at '1ea92f346cc48cf50edb1769bb4984d4ab9a1c3d'
2021-08-10 00:03:13,061 DEBUG: Popping from stash 'refs/exps/stash'
2021-08-10 00:03:13,331 DEBUG: Applying stash commit '429b12727236cfdc175781fd5506aea39f23a582'
2021-08-10 00:03:13,519 DEBUG: Restore HEAD to 'master'
2021-08-10 00:03:13,587 ERROR: Could not apply stash commit: Stash apply resulted in merge conflicts: 241 uncommitted changes exist in the index
------------------------------------------------------------
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\backend\pygit2.py", line 398, in _apply
self.repo.stash_apply(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\pygit2\repository.py", line 1129, in stash_apply
check_error(C.git_stash_apply(self._repo, index, stash_opts))
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\pygit2\errors.py", line 65, in check_error
raise GitError(message)
_pygit2.GitError: 241 uncommitted changes exist in the index
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\stash.py", line 52, in pop
self.apply(rev)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\stash.py", line 60, in apply
self.scm._stash_apply(rev) # pylint: disable=protected-access
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\__init__.py", line 343, in _backend_func
return func(*args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\backend\pygit2.py", line 419, in _stash_apply
_apply(0)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\backend\pygit2.py", line 403, in _apply
raise MergeConflictError(
dvc.scm.base.MergeConflictError: Stash apply resulted in merge conflicts
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\main.py", line 55, in main
ret = cmd.do_run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\base.py", line 50, in do_run
return self.run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\experiments.py", line 581, in run
results = self.repo.experiments.run(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 957, in run
return run(self.repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
return f(repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\run.py", line 32, in run
return repo.experiments.reproduce_one(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 421, in reproduce_one
results = self._workspace_repro()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 57, in wrapper
ret = f(exp, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 775, in _workspace_repro
rev = self.stash.pop()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\scm\git\stash.py", line 54, in pop
raise SCMError("Could not apply stash commit") from exc
dvc.scm.base.SCMError: Could not apply stash commit
------------------------------------------------------------
2021-08-10 00:03:13,943 DEBUG: Analytics is disabled.
- remove
check_errorfromstash_applyinpygit2/repository.py: - run
dvc exp run -v(runs without error):
2021-08-10 00:06:18,684 DEBUG: Check for update is disabled.
2021-08-10 00:06:19,488 DEBUG: Checkpoint run from '1ea92f3' with baseline '7a957c08afdf1a3cf8276271ebe99f5f35ef321b'
2021-08-10 00:06:19,488 DEBUG: Detaching HEAD at 'HEAD'
2021-08-10 00:06:19,542 DEBUG: Stashing workspace
2021-08-10 00:06:19,813 DEBUG: Stashing changes in 'refs/stash'
2021-08-10 00:06:19,896 DEBUG: Applying stash commit '0c80a62141d775b21ba789c6731207989d69d80e'
2021-08-10 00:06:20,155 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
2021-08-10 00:06:20,162 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:06:20,164 DEBUG: Checking out 'data\MNIST\raw' with cache 'object md5: 8c257df187855c681f88bde92d721ccd.dir'.
2021-08-10 00:06:20,166 DEBUG: Linking directory 'data\MNIST\raw'.
2021-08-10 00:06:20,171 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-10 00:06:20,175 DEBUG: Created 'copy': .dvc\cache\.cache_type_test_file -> data\MNIST\raw\.nSAq3Up6DAb59zWnwMUiab
2021-08-10 00:06:20,176 DEBUG: Removing 'data\MNIST\raw\.nSAq3Up6DAb59zWnwMUiab'
2021-08-10 00:06:20,178 DEBUG: Removing '.dvc\cache\.cache_type_test_file'
2021-08-10 00:06:20,181 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,183 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,186 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,189 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,192 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,194 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,197 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,199 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:06:20,202 DEBUG: state save (8782108282916533484, 2ad916ff24d741373e0dafb9546e467f, 66544770) 8c257df187855c681f88bde92d721ccd.dir
2021-08-10 00:06:20,223 DEBUG: Saving information to 'data\MNIST\raw.dvc'.
Modified checkpoint experiment based on 'exp-a59e4' will be created
2021-08-10 00:06:21,050 DEBUG: Stashing changes in 'refs/exps/stash'
2021-08-10 00:06:22,752 DEBUG: Stashed experiment '43e44f5' with baseline '7a957c0' for future execution.
2021-08-10 00:06:24,003 DEBUG: Restoring stashed workspace
2021-08-10 00:06:24,055 DEBUG: Popping from stash 'refs/stash'
2021-08-10 00:06:24,057 DEBUG: Applying stash commit '0c80a62141d775b21ba789c6731207989d69d80e'
2021-08-10 00:06:24,154 DEBUG: Dropping 'refs/stash@{0}'
2021-08-10 00:06:24,208 DEBUG: Restore HEAD to 'master'
2021-08-10 00:06:24,236 DEBUG: Detaching HEAD at '1ea92f346cc48cf50edb1769bb4984d4ab9a1c3d'
2021-08-10 00:06:24,263 DEBUG: Popping from stash 'refs/exps/stash'
2021-08-10 00:06:24,274 DEBUG: Applying stash commit '43e44f5cc2ba5e0255861f2457b02fc545c99529'
2021-08-10 00:06:24,330 DEBUG: Dropping 'refs/exps/stash@{0}'
2021-08-10 00:06:24,708 DEBUG: Running repro in 'C:\Users\Administrator\code\vscode-dvc\demo'
2021-08-10 00:06:26,022 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
'data\MNIST\raw.dvc' didn't change, skipping
2021-08-10 00:06:26,214 DEBUG: stage: 'train' changed.
2021-08-10 00:06:26,261 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:06:26,273 DEBUG: Removing output 'predictions.json' of stage: 'train'.
2021-08-10 00:06:26,277 DEBUG: Removing 'predictions.json'
2021-08-10 00:06:26,290 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:26,293 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
Running stage 'train':
> python train.py
C:\Users\Administrator\code\vscode-dvc\demo\.env\lib\site-packages\torchvision\datasets\mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:180.)
return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)
C:\Users\Administrator\code\vscode-dvc\demo\.env\lib\site-packages\torch\nn\functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at ..\c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
2021-08-10 00:06:39,833 DEBUG: state save (3032210458725577634, 1628553983697490688, 5) 9a1093a5daf806d8bbaef3273ca12a24
2021-08-10 00:06:39,867 DEBUG: state save (2538292536843309730, 1628553983701488896, 91) 36ba03665aca210beecd3a3d1fb01a7a
2021-08-10 00:06:39,888 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:39,890 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:39,895 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:39,897 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:39,955 DEBUG: state save (5136252253359496632, 1628553998096464128, 410191) abff23f1e32f2621260506433042c992
2021-08-10 00:06:39,973 DEBUG: state save (5136252253359496632, 1628553998096464128, 410191) abff23f1e32f2621260506433042c992
2021-08-10 00:06:39,975 DEBUG: {'model.pt': 'modified'}
2021-08-10 00:06:40,030 DEBUG: state save (6094399142862152409, 1628553999676563456, 310000) e5043dc3efea46e3960b8d2279f6a1ba
2021-08-10 00:06:40,040 DEBUG: state save (6094399142862152409, 1628553999676563456, 310000) e5043dc3efea46e3960b8d2279f6a1ba
2021-08-10 00:06:40,046 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 00:06:40,055 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 00:06:40,115 DEBUG: state save (2735456330190799666, 1628553999737544704, 59) 31c025dc2028faafea504863a4504957
2021-08-10 00:06:40,141 DEBUG: state save (3308320455044055458, 1628553999738542848, 46) beb238213c0fbb9f45434cf6ee50c658
2021-08-10 00:06:40,150 DEBUG: staged tree 'object md5: edfd2c1434c4f73118421fa669ea361d.dir'
2021-08-10 00:06:40,154 DEBUG: state save (1457375764667085341, abccdbe281707c275fcaac3c98796f44, 105) edfd2c1434c4f73118421fa669ea361d.dir
2021-08-10 00:06:40,156 DEBUG: {'logs': 'modified'}
2021-08-10 00:06:40,191 DEBUG: Computed stage: 'train' md5: 'abb5a00e174d2d4c26eb30717627076d'
2021-08-10 00:06:40,268 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:40,272 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:40,301 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:40,636 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:40,638 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:40,642 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:40,658 DEBUG: state save (6094399142862152409, 1628553999676563456, 310000) e5043dc3efea46e3960b8d2279f6a1ba
2021-08-10 00:06:40,660 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:06:40,664 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:06:40,666 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:06:40,669 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:06:40,681 DEBUG: state save (6094399142862152409, 1628553999676563456, 310000) e5043dc3efea46e3960b8d2279f6a1ba
2021-08-10 00:06:40,693 DEBUG: Checking out 'predictions.json' with cache 'ref object md5: e5043dc3efea46e3960b8d2279f6a1ba -> predictions.json'.
2021-08-10 00:06:40,727 DEBUG: Created 'copy': .dvc\cache\e5\043dc3efea46e3960b8d2279f6a1ba -> predictions.json
2021-08-10 00:06:40,740 DEBUG: state save (7228765298518594407, 1628554000724508160, 310000) e5043dc3efea46e3960b8d2279f6a1ba
2021-08-10 00:06:40,792 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:06:40,796 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:06:40,825 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:06:40,842 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:06:40,847 DEBUG: state save (5136252253359496632, 1628553998096464128, 410191) abff23f1e32f2621260506433042c992
2021-08-10 00:06:40,867 DEBUG: Checking out 'model.pt' with cache 'ref object md5: abff23f1e32f2621260506433042c992 -> model.pt'.
2021-08-10 00:06:40,889 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-10 00:06:40,926 DEBUG: Created 'copy': .dvc\cache\ab\ff23f1e32f2621260506433042c992 -> model.pt
2021-08-10 00:06:40,933 DEBUG: state save (7808676096832632822, 1628554000925509888, 410191) abff23f1e32f2621260506433042c992
2021-08-10 00:06:41,021 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:06:41,023 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:06:41,034 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:06:41,050 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:06:41,083 DEBUG: state save (3308320455044055458, 1628553999738542848, 46) beb238213c0fbb9f45434cf6ee50c658
2021-08-10 00:06:41,104 DEBUG: state save (2735456330190799666, 1628553999737544704, 59) 31c025dc2028faafea504863a4504957
2021-08-10 00:06:41,189 DEBUG: Downloading 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f/ed/fd2c1434c4f73118421fa669ea361d.dir' to '.dvc\cache\ed\fd2c1434c4f73118421fa669ea361d.dir'
2021-08-10 00:06:41,206 DEBUG: state save (2961203292049156359, 1628554001203508480, 135) edfd2c1434c4f73118421fa669ea361d.dir
2021-08-10 00:06:41,214 DEBUG: Checking out 'logs' with cache 'object md5: edfd2c1434c4f73118421fa669ea361d.dir'.
2021-08-10 00:06:41,221 DEBUG: Linking directory 'logs'.
2021-08-10 00:06:41,260 DEBUG: Created 'copy': .dvc\cache\be\b238213c0fbb9f45434cf6ee50c658 -> logs\acc.tsv
2021-08-10 00:06:41,294 DEBUG: Created 'copy': .dvc\cache\31\c025dc2028faafea504863a4504957 -> logs\loss.tsv
2021-08-10 00:06:41,297 DEBUG: state save (1457375764667085341, f1f407328ae0ba66db8bc7e593c77d16, 105) edfd2c1434c4f73118421fa669ea361d.dir
2021-08-10 00:06:41,379 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:06:41,382 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 00:06:41,384 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:41,392 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:41,393 DEBUG: Running checkpoint callback for stage 'stage: 'train''
Updating lock file 'dvc.lock'
2021-08-10 00:06:42,752 DEBUG: Commit to new experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-c730b'
Checkpoint experiment iteration 'e83dbdb'.
2021-08-10 00:06:43,072 DEBUG: Removing signal file for 'checkpoint' task
file://C:\Users\Administrator\code\vscode-dvc\demo\logs.html
2021-08-10 00:06:43,596 DEBUG: Removing signal file for 'live' task
2021-08-10 00:06:51,237 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:51,239 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:51,243 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:51,257 DEBUG: state save (7808676096832632822, 1628554008624119040, 410191) e53c3bc6abe7418c4b16aede95143e6f
2021-08-10 00:06:51,261 DEBUG: {'model.pt': 'modified'}
2021-08-10 00:06:51,273 DEBUG: state save (7228765298518594407, 1628554009081577216, 310000) 8b04313a209e7a594fc43783acecddeb
2021-08-10 00:06:51,276 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 00:06:51,294 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 00:06:51,325 DEBUG: staged tree 'object md5: 574ead971262f01a0d2ba766117eb5f3.dir'
2021-08-10 00:06:51,329 DEBUG: state save (1457375764667085341, 7873c10da37da502af8043a2d7bfb85f, 166) 574ead971262f01a0d2ba766117eb5f3.dir
2021-08-10 00:06:51,332 DEBUG: {'logs': 'modified'}
2021-08-10 00:06:51,353 DEBUG: Computed stage: 'train' md5: 'b068f7256a3cf9a154b3c6e99776a9af'
2021-08-10 00:06:51,415 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:51,420 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:51,425 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:51,511 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:51,516 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:51,522 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:51,546 DEBUG: state save (7228765298518594407, 1628554009081577216, 310000) 8b04313a209e7a594fc43783acecddeb
2021-08-10 00:06:51,547 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:06:51,548 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:06:51,550 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:06:51,553 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:06:51,558 DEBUG: state save (7228765298518594407, 1628554009081577216, 310000) 8b04313a209e7a594fc43783acecddeb
2021-08-10 00:06:51,571 DEBUG: Checking out 'predictions.json' with cache 'ref object md5: 8b04313a209e7a594fc43783acecddeb -> predictions.json'.
2021-08-10 00:06:51,585 DEBUG: Created 'copy': .dvc\cache\8b\04313a209e7a594fc43783acecddeb -> predictions.json
2021-08-10 00:06:51,601 DEBUG: state save (5759430429627810359, 1628554011581470208, 310000) 8b04313a209e7a594fc43783acecddeb
2021-08-10 00:06:51,655 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:06:51,658 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:06:51,660 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:06:51,668 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:06:51,676 DEBUG: state save (7808676096832632822, 1628554008624119040, 410191) e53c3bc6abe7418c4b16aede95143e6f
2021-08-10 00:06:51,699 DEBUG: Checking out 'model.pt' with cache 'ref object md5: e53c3bc6abe7418c4b16aede95143e6f -> model.pt'.
2021-08-10 00:06:51,713 DEBUG: Created 'copy': .dvc\cache\e5\3c3bc6abe7418c4b16aede95143e6f -> model.pt
2021-08-10 00:06:51,716 DEBUG: state save (828754278277400517, 1628554011710471936, 410191) e53c3bc6abe7418c4b16aede95143e6f
2021-08-10 00:06:51,739 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:06:51,740 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:06:51,755 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:06:51,766 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:06:51,778 DEBUG: state save (2470259269456398407, 1628554009087578368, 95) 696f61c901a72a2abf76a8cfcb9c305d
2021-08-10 00:06:51,780 DEBUG: state save (8977900025558396832, 1628554009088577024, 71) 7a90ffbc0837b03c0cba063e22f75398
2021-08-10 00:06:51,808 DEBUG: Downloading 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f/57/4ead971262f01a0d2ba766117eb5f3.dir' to '.dvc\cache\57\4ead971262f01a0d2ba766117eb5f3.dir'
2021-08-10 00:06:51,820 DEBUG: state save (4253559800715240344, 1628554011816470272, 135) 574ead971262f01a0d2ba766117eb5f3.dir
2021-08-10 00:06:51,824 DEBUG: Checking out 'logs' with cache 'object md5: 574ead971262f01a0d2ba766117eb5f3.dir'.
2021-08-10 00:06:51,827 DEBUG: Linking directory 'logs'.
2021-08-10 00:06:51,831 DEBUG: Created 'copy': .dvc\cache\7a\90ffbc0837b03c0cba063e22f75398 -> logs\acc.tsv
2021-08-10 00:06:51,836 DEBUG: Created 'copy': .dvc\cache\69\6f61c901a72a2abf76a8cfcb9c305d -> logs\loss.tsv
2021-08-10 00:06:51,839 DEBUG: state save (1457375764667085341, dfd413eb30a0957808f558de893d6057, 166) 574ead971262f01a0d2ba766117eb5f3.dir
2021-08-10 00:06:51,874 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:06:51,894 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 00:06:51,919 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:51,930 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:51,932 DEBUG: Running checkpoint callback for stage 'stage: 'train''
Updating lock file 'dvc.lock'
2021-08-10 00:06:52,809 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-c730b'
Checkpoint experiment iteration 'af28688'.
2021-08-10 00:06:53,150 DEBUG: Removing signal file for 'checkpoint' task
file://C:\Users\Administrator\code\vscode-dvc\demo\logs.html
2021-08-10 00:06:53,238 DEBUG: Removing signal file for 'live' task
file://C:\Users\Administrator\code\vscode-dvc\demo\logs.html
2021-08-10 00:06:53,490 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:53,492 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:53,498 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:53,518 DEBUG: {}
2021-08-10 00:06:53,519 DEBUG: Output 'model.pt' didn't change. Skipping saving.
2021-08-10 00:06:53,525 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 00:06:53,539 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 00:06:53,552 DEBUG: {}
2021-08-10 00:06:53,553 DEBUG: Output 'logs' didn't change. Skipping saving.
2021-08-10 00:06:53,559 DEBUG: Computed stage: 'train' md5: 'b068f7256a3cf9a154b3c6e99776a9af'
2021-08-10 00:06:53,613 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:53,616 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:53,619 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:53,689 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:06:53,692 DEBUG: state save (8796477465494994334, 361c941d3de1055986ab5194bc243f96, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:06:53,695 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:06:53,713 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:06:53,715 DEBUG: Checking out 'predictions.json' with cache 'object md5: 8b04313a209e7a594fc43783acecddeb'.
2021-08-10 00:06:53,717 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 00:06:53,724 DEBUG: state save (5759430429627810359, 1628554011581470208, 310000) 8b04313a209e7a594fc43783acecddeb
2021-08-10 00:06:53,767 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:06:53,769 DEBUG: Checking out 'model.pt' with cache 'object md5: e53c3bc6abe7418c4b16aede95143e6f'.
2021-08-10 00:06:53,772 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:06:53,774 DEBUG: state save (828754278277400517, 1628554011710471936, 410191) e53c3bc6abe7418c4b16aede95143e6f
2021-08-10 00:06:53,797 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:06:53,800 DEBUG: Checking out 'logs' with cache 'object md5: 574ead971262f01a0d2ba766117eb5f3.dir'.
2021-08-10 00:06:53,803 DEBUG: Linking directory 'logs'.
2021-08-10 00:06:53,809 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:53,814 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:06:53,820 DEBUG: state save (1457375764667085341, dfd413eb30a0957808f558de893d6057, 166) 574ead971262f01a0d2ba766117eb5f3.dir
2021-08-10 00:06:53,851 DEBUG: stage: 'train' was reproduced
To track the changes with git, run:
git add predictions.json 'data\MNIST' .gitignore train.py dvc.yaml params.yaml dvc.lock logs.json
2021-08-10 00:06:54,567 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-c730b'
2021-08-10 00:06:54,886 DEBUG: Removing 'C:\Users\Administrator\code\vscode-dvc\demo\.dvc\tmp\exps\run\workspace.run'
2021-08-10 00:06:54,952 DEBUG: Restore HEAD to 'master'
Reproduced experiment(s): exp-c730b
Experiment results have been applied to your workspace.
To promote an experiment to a Git branch run:
dvc exp branch <exp>
2021-08-10 00:06:54,977 DEBUG: Analytics is disabled.
- run
dvc exp run -v(fails again):
2021-08-10 00:13:20,933 DEBUG: Check for update is disabled.
2021-08-10 00:13:21,215 DEBUG: Checkpoint run from 'b0e55cd' with baseline '7a957c08afdf1a3cf8276271ebe99f5f35ef321b'
2021-08-10 00:13:21,215 DEBUG: Detaching HEAD at 'HEAD'
2021-08-10 00:13:21,416 DEBUG: Stashing workspace
2021-08-10 00:13:21,589 DEBUG: Stashing changes in 'refs/stash'
2021-08-10 00:13:22,183 DEBUG: Applying stash commit '1d9984739b9fbc785c56a119ff45188d844d4dbb'
2021-08-10 00:13:22,605 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
2021-08-10 00:13:22,613 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:13:22,614 DEBUG: Checking out 'data\MNIST\raw' with cache 'object md5: 8c257df187855c681f88bde92d721ccd.dir'.
2021-08-10 00:13:22,615 DEBUG: Linking directory 'data\MNIST\raw'.
2021-08-10 00:13:22,620 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-10 00:13:22,625 DEBUG: Created 'copy': .dvc\cache\.cache_type_test_file -> data\MNIST\raw\.8aEJiCkacxTh6xNAURf3nn
2021-08-10 00:13:22,627 DEBUG: Removing 'data\MNIST\raw\.8aEJiCkacxTh6xNAURf3nn'
2021-08-10 00:13:22,630 DEBUG: Removing '.dvc\cache\.cache_type_test_file'
2021-08-10 00:13:22,635 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,641 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,646 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,649 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,652 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,655 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,657 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,660 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:13:22,663 DEBUG: state save (8782108282916533484, 2ad916ff24d741373e0dafb9546e467f, 66544770) 8c257df187855c681f88bde92d721ccd.dir
2021-08-10 00:13:22,686 DEBUG: Saving information to 'data\MNIST\raw.dvc'.
Modified checkpoint experiment based on 'exp-c730b' will be created
2021-08-10 00:13:23,593 DEBUG: Stashing changes in 'refs/exps/stash'
2021-08-10 00:13:25,063 DEBUG: Stashed experiment '6c78182' with baseline '7a957c0' for future execution.
2021-08-10 00:13:25,375 DEBUG: Restoring stashed workspace
2021-08-10 00:13:25,376 DEBUG: Popping from stash 'refs/stash'
2021-08-10 00:13:25,378 DEBUG: Applying stash commit '1d9984739b9fbc785c56a119ff45188d844d4dbb'
2021-08-10 00:13:25,669 DEBUG: Dropping 'refs/stash@{0}'
2021-08-10 00:13:25,679 DEBUG: Restore HEAD to 'master'
2021-08-10 00:13:25,699 DEBUG: Detaching HEAD at 'b0e55cd08d60f05fedf4eba569d31e0fcb6cee31'
2021-08-10 00:13:26,022 DEBUG: Popping from stash 'refs/exps/stash'
2021-08-10 00:13:26,070 DEBUG: Applying stash commit '6c781829e035412e273cdd8e5eb420d30de089c2'
2021-08-10 00:13:26,167 DEBUG: Dropping 'refs/exps/stash@{0}'
2021-08-10 00:13:26,377 DEBUG: Running repro in 'C:\Users\Administrator\code\vscode-dvc\demo'
2021-08-10 00:13:28,960 DEBUG: Removing 'C:\Users\Administrator\code\vscode-dvc\demo\.dvc\tmp\exps\run\workspace.run'
2021-08-10 00:13:28,971 DEBUG: Restore HEAD to 'master'
2021-08-10 00:13:30,502 ERROR: Failed to reproduce experiment '6c78182': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Administrator\\code\\vscode-dvc\\demo\\.dvc\\tmp\\exps\\run\\workspace.run'
------------------------------------------------------------
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\zc\lockfile\__init__.py", line 42, in _lock_file
msvcrt.locking(file.fileno(), msvcrt.LK_NBLCK, 1)
PermissionError: [Errno 13] Permission denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\lock.py", line 112, in _do_lock
self._lock = zc.lockfile.LockFile(self._lockfile)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\zc\lockfile\__init__.py", line 117, in __init__
super(LockFile, self).__init__(path)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\zc\lockfile\__init__.py", line 90, in __init__
_lock_file(fp)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\zc\lockfile\__init__.py", line 44, in _lock_file
raise LockError("Couldn't lock %r" % file.name)
zc.lockfile.LockError: Couldn't lock 'C:\\Users\\Administrator\\code\\vscode-dvc\\demo\\.dvc\\tmp\\lock'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\executor\base.py", line 422, in _repro_dvc
yield dvc
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\executor\base.py", line 322, in reproduce
dvc_checkout(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\__init__.py", line 48, in wrapper
with lock_repo(repo):
File "C:\Python39\lib\contextlib.py", line 117, in __enter__
return next(self.gen)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\__init__.py", line 36, in lock_repo
with repo.lock:
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\lock.py", line 131, in __enter__
self.lock()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\lock.py", line 120, in lock
lock_retry()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\funcy\decorators.py", line 45, in wrapper
return deco(call, *dargs, **dkwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\funcy\flow.py", line 127, in retry
return call()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\funcy\decorators.py", line 66, in __call__
return self._func(*self._args, **self._kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\lock.py", line 114, in _do_lock
raise LockError(FAILED_TO_LOCK_MESSAGE)
dvc.lock.LockError: Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly. Check the page <https://dvc.org/doc/user-guide/troubleshooting#lock-issue>
for other possible reasons and to learn how to resolve this.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\utils\fs.py", line 132, in _unlink
os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Administrator\\code\\vscode-dvc\\demo\\.dvc\\tmp\\exps\\run\\workspace.run'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 789, in _workspace_repro
exec_result = BaseExecutor.reproduce(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\executor\base.py", line 372, in reproduce
logger.warning(
File "C:\Python39\lib\contextlib.py", line 135, in __exit__
self.gen.throw(type, value, traceback)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\executor\base.py", line 435, in _repro_dvc
remove(pidfile)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\utils\fs.py", line 144, in remove
_unlink(path, _chmod)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\utils\fs.py", line 134, in _unlink
onerror(os.unlink, path, sys.exc_info())
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\utils\fs.py", line 127, in _chmod
func(p)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\Administrator\\code\\vscode-dvc\\demo\\.dvc\\tmp\\exps\\run\\workspace.run'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\main.py", line 55, in main
ret = cmd.do_run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\base.py", line 50, in do_run
return self.run()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\command\experiments.py", line 581, in run
results = self.repo.experiments.run(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 957, in run
return run(self.repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\__init__.py", line 49, in wrapper
return f(repo, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\run.py", line 32, in run
return repo.experiments.reproduce_one(
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 421, in reproduce_one
results = self._workspace_repro()
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 57, in wrapper
ret = f(exp, *args, **kwargs)
File "c:\users\administrator\code\vscode-dvc\demo\.env\lib\site-packages\dvc\repo\experiments\__init__.py", line 815, in _workspace_repro
raise DvcException(
dvc.exceptions.DvcException: Failed to reproduce experiment '6c78182'
subsequent runs all seem to pass but the experiment never “resumes”, a new experiment is always started. Here is an example log:
2021-08-10 00:15:28,308 DEBUG: Check for update is disabled.
2021-08-10 00:15:28,622 DEBUG: Checkpoint run from '01103a9' with baseline '7a957c08afdf1a3cf8276271ebe99f5f35ef321b'
2021-08-10 00:15:28,623 DEBUG: Detaching HEAD at 'HEAD'
2021-08-10 00:15:28,826 DEBUG: Stashing workspace
2021-08-10 00:15:29,053 DEBUG: Stashing changes in 'refs/stash'
2021-08-10 00:15:29,684 DEBUG: Applying stash commit 'ee68b123d5390cbcbc43713e4a7869e97057d878'
2021-08-10 00:15:30,151 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
2021-08-10 00:15:30,159 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:15:30,161 DEBUG: Checking out 'data\MNIST\raw' with cache 'object md5: 8c257df187855c681f88bde92d721ccd.dir'.
2021-08-10 00:15:30,163 DEBUG: Linking directory 'data\MNIST\raw'.
2021-08-10 00:15:30,168 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-10 00:15:30,172 DEBUG: Created 'copy': .dvc\cache\.cache_type_test_file -> data\MNIST\raw\.WBdj3ejTWfViGnkRpD4bGb
2021-08-10 00:15:30,173 DEBUG: Removing 'data\MNIST\raw\.WBdj3ejTWfViGnkRpD4bGb'
2021-08-10 00:15:30,175 DEBUG: Removing '.dvc\cache\.cache_type_test_file'
2021-08-10 00:15:30,177 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,180 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,183 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,187 DEBUG: Skipping copying for 'data\MNIST\raw\t10k-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,190 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,197 DEBUG: Skipping copying for 'data\MNIST\raw\train-images-idx3-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,205 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,212 DEBUG: Skipping copying for 'data\MNIST\raw\train-labels-idx1-ubyte.gz', since it is not a symlink or a hardlink.
2021-08-10 00:15:30,221 DEBUG: state save (8782108282916533484, 2ad916ff24d741373e0dafb9546e467f, 66544770) 8c257df187855c681f88bde92d721ccd.dir
2021-08-10 00:15:30,248 DEBUG: Saving information to 'data\MNIST\raw.dvc'.
Modified checkpoint experiment based on 'exp-d1392' will be created
2021-08-10 00:15:31,174 DEBUG: Stashing changes in 'refs/exps/stash'
2021-08-10 00:15:32,671 DEBUG: Stashed experiment 'ecd72a3' with baseline '7a957c0' for future execution.
2021-08-10 00:15:33,034 DEBUG: Restoring stashed workspace
2021-08-10 00:15:33,035 DEBUG: Popping from stash 'refs/stash'
2021-08-10 00:15:33,036 DEBUG: Applying stash commit 'ee68b123d5390cbcbc43713e4a7869e97057d878'
2021-08-10 00:15:33,353 DEBUG: Dropping 'refs/stash@{0}'
2021-08-10 00:15:33,363 DEBUG: Restore HEAD to 'master'
2021-08-10 00:15:33,389 DEBUG: Detaching HEAD at '01103a9b2c2e59e8a873624a3dd28586b5926998'
2021-08-10 00:15:33,784 DEBUG: Popping from stash 'refs/exps/stash'
2021-08-10 00:15:33,799 DEBUG: Applying stash commit 'ecd72a3c123613793282e5ea0f9af794c2e58a28'
2021-08-10 00:15:33,882 DEBUG: Dropping 'refs/exps/stash@{0}'
2021-08-10 00:15:34,036 DEBUG: Running repro in 'C:\Users\Administrator\code\vscode-dvc\demo'
2021-08-10 00:15:35,964 DEBUG: Computed stage: 'data\MNIST\raw.dvc' md5: 'None'
'data\MNIST\raw.dvc' didn't change, skipping
2021-08-10 00:15:35,995 DEBUG: stage: 'train' changed.
2021-08-10 00:15:36,011 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:15:36,014 DEBUG: Removing output 'predictions.json' of stage: 'train'.
2021-08-10 00:15:36,016 DEBUG: Removing 'predictions.json'
2021-08-10 00:15:36,024 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:36,028 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
Running stage 'train':
> python train.py
C:\Users\Administrator\code\vscode-dvc\demo\.env\lib\site-packages\torchvision\datasets\mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:180.)
return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)
C:\Users\Administrator\code\vscode-dvc\demo\.env\lib\site-packages\torch\nn\functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at ..\c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
2021-08-10 00:15:44,278 DEBUG: state save (5790250872895244108, 1628554532665517056, 91) 36ba03665aca210beecd3a3d1fb01a7a
2021-08-10 00:15:44,285 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:44,288 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:44,292 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:44,321 DEBUG: state save (7289658300788867685, 1628554542933499136, 410191) 3e80090efe40c647078bf54c52bd5f54
2021-08-10 00:15:44,334 DEBUG: state save (7289658300788867685, 1628554542933499136, 410191) 3e80090efe40c647078bf54c52bd5f54
2021-08-10 00:15:44,336 DEBUG: {'model.pt': 'modified'}
2021-08-10 00:15:44,365 DEBUG: state save (616998489237124496, 1628554543374497280, 310000) 6f24e48076cf9a5836138c89eda6948e
2021-08-10 00:15:44,368 DEBUG: state save (616998489237124496, 1628554543374497280, 310000) 6f24e48076cf9a5836138c89eda6948e
2021-08-10 00:15:44,370 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 00:15:44,381 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 00:15:44,414 DEBUG: state save (1753618898336630839, 1628554543383501824, 46) 88fa5e43feea85504d9f043728bc6956
2021-08-10 00:15:44,414 DEBUG: state save (483538167437174312, 1628554543382503680, 59) 2440e16142735656730c499df3b102e4
2021-08-10 00:15:44,432 DEBUG: staged tree 'object md5: 4d87a14a751ad7feed61a8fe4ad4fdf5.dir'
2021-08-10 00:15:44,435 DEBUG: state save (1457375764667085341, d98d6db67b3b7306080d5cef8e63542e, 105) 4d87a14a751ad7feed61a8fe4ad4fdf5.dir
2021-08-10 00:15:44,438 DEBUG: {'logs': 'modified'}
2021-08-10 00:15:44,453 DEBUG: Computed stage: 'train' md5: '947d631d2c00f8deb8e399240e9a84e8'
2021-08-10 00:15:44,512 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:44,516 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:44,521 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:44,588 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:44,595 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:44,602 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:44,629 DEBUG: state save (616998489237124496, 1628554543374497280, 310000) 6f24e48076cf9a5836138c89eda6948e
2021-08-10 00:15:44,630 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:15:44,632 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:15:44,633 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:15:44,636 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:15:44,641 DEBUG: state save (616998489237124496, 1628554543374497280, 310000) 6f24e48076cf9a5836138c89eda6948e
2021-08-10 00:15:44,655 DEBUG: Checking out 'predictions.json' with cache 'ref object md5: 6f24e48076cf9a5836138c89eda6948e -> predictions.json'.
2021-08-10 00:15:44,672 DEBUG: Created 'copy': .dvc\cache\6f\24e48076cf9a5836138c89eda6948e -> predictions.json
2021-08-10 00:15:44,677 DEBUG: state save (3047864459953640844, 1628554544669497600, 310000) 6f24e48076cf9a5836138c89eda6948e
2021-08-10 00:15:44,713 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:15:44,716 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:15:44,717 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:15:44,720 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:15:44,724 DEBUG: state save (7289658300788867685, 1628554542933499136, 410191) 3e80090efe40c647078bf54c52bd5f54
2021-08-10 00:15:44,732 DEBUG: Checking out 'model.pt' with cache 'ref object md5: 3e80090efe40c647078bf54c52bd5f54 -> model.pt'.
2021-08-10 00:15:44,734 DEBUG: Cache type 'reflink' is not supported: reflink is not supported
2021-08-10 00:15:44,743 DEBUG: Created 'copy': .dvc\cache\3e\80090efe40c647078bf54c52bd5f54 -> model.pt
2021-08-10 00:15:44,749 DEBUG: state save (1433297732824338785, 1628554544741499904, 410191) 3e80090efe40c647078bf54c52bd5f54
2021-08-10 00:15:44,780 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:15:44,782 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:15:44,785 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:15:44,791 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:15:44,798 DEBUG: state save (1753618898336630839, 1628554543383501824, 46) 88fa5e43feea85504d9f043728bc6956
2021-08-10 00:15:44,801 DEBUG: state save (483538167437174312, 1628554543382503680, 59) 2440e16142735656730c499df3b102e4
2021-08-10 00:15:44,819 DEBUG: Downloading 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f/4d/87a14a751ad7feed61a8fe4ad4fdf5.dir' to '.dvc\cache\4d\87a14a751ad7feed61a8fe4ad4fdf5.dir'
2021-08-10 00:15:44,829 DEBUG: state save (2290552367422593571, 1628554544825498624, 135) 4d87a14a751ad7feed61a8fe4ad4fdf5.dir
2021-08-10 00:15:44,833 DEBUG: Checking out 'logs' with cache 'object md5: 4d87a14a751ad7feed61a8fe4ad4fdf5.dir'.
2021-08-10 00:15:44,836 DEBUG: Linking directory 'logs'.
2021-08-10 00:15:44,847 DEBUG: Created 'copy': .dvc\cache\88\fa5e43feea85504d9f043728bc6956 -> logs\acc.tsv
2021-08-10 00:15:44,864 DEBUG: Created 'copy': .dvc\cache\24\40e16142735656730c499df3b102e4 -> logs\loss.tsv
2021-08-10 00:15:44,878 DEBUG: state save (1457375764667085341, 5b751a1a0274427e7868e84f619585da, 105) 4d87a14a751ad7feed61a8fe4ad4fdf5.dir
2021-08-10 00:15:44,905 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:15:44,908 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 00:15:44,910 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:44,913 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:44,918 DEBUG: Running checkpoint callback for stage 'stage: 'train''
Updating lock file 'dvc.lock'
2021-08-10 00:15:45,620 DEBUG: Commit to new experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-9bb5a'
Checkpoint experiment iteration '70f89bf'.
2021-08-10 00:15:45,985 DEBUG: Removing signal file for 'checkpoint' task
file://C:\Users\Administrator\code\vscode-dvc\demo\logs.html
2021-08-10 00:15:46,632 DEBUG: Removing signal file for 'live' task
2021-08-10 00:15:52,959 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:52,961 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:52,965 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:52,987 DEBUG: state save (1433297732824338785, 1628554552234487808, 410191) d05ac82f4c284531f86db5d9fcdf29a2
2021-08-10 00:15:52,990 DEBUG: state save (1433297732824338785, 1628554552234487808, 410191) d05ac82f4c284531f86db5d9fcdf29a2
2021-08-10 00:15:52,992 DEBUG: {'model.pt': 'modified'}
2021-08-10 00:15:53,014 DEBUG: state save (3047864459953640844, 1628554552780473856, 310000) f43f0fd3621297080b311aa4ccb08ea3
2021-08-10 00:15:53,018 DEBUG: state save (3047864459953640844, 1628554552780473856, 310000) f43f0fd3621297080b311aa4ccb08ea3
2021-08-10 00:15:53,022 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 00:15:53,043 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 00:15:53,075 DEBUG: state save (8798193123524591996, 1628554552785486336, 71) ffd4ca3f4caefcd3b1c7470517c8ca10
2021-08-10 00:15:53,086 DEBUG: state save (3905119094551895630, 1628554552784489984, 96) acc1b5fdf9aadec17fa600c5b2be5832
2021-08-10 00:15:53,105 DEBUG: staged tree 'object md5: d6b00174e60d73e40336502a15f082fc.dir'
2021-08-10 00:15:53,108 DEBUG: state save (1457375764667085341, 8033ffe91038c9f0fa12dda4f449682d, 167) d6b00174e60d73e40336502a15f082fc.dir
2021-08-10 00:15:53,111 DEBUG: {'logs': 'modified'}
2021-08-10 00:15:53,132 DEBUG: Computed stage: 'train' md5: '750c12bdda625f4ed5c2f6ecb7b24ac4'
2021-08-10 00:15:53,183 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:53,187 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:53,197 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:53,264 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:53,268 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:53,274 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:53,294 DEBUG: state save (3047864459953640844, 1628554552780473856, 310000) f43f0fd3621297080b311aa4ccb08ea3
2021-08-10 00:15:53,295 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:15:53,297 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:15:53,298 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:15:53,301 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:15:53,306 DEBUG: state save (3047864459953640844, 1628554552780473856, 310000) f43f0fd3621297080b311aa4ccb08ea3
2021-08-10 00:15:53,320 DEBUG: Checking out 'predictions.json' with cache 'ref object md5: f43f0fd3621297080b311aa4ccb08ea3 -> predictions.json'.
2021-08-10 00:15:53,331 DEBUG: Created 'copy': .dvc\cache\f4\3f0fd3621297080b311aa4ccb08ea3 -> predictions.json
2021-08-10 00:15:53,337 DEBUG: state save (2384021035544222717, 1628554553329487104, 310000) f43f0fd3621297080b311aa4ccb08ea3
2021-08-10 00:15:53,377 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:15:53,379 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:15:53,380 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:15:53,384 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:15:53,393 DEBUG: state save (1433297732824338785, 1628554552234487808, 410191) d05ac82f4c284531f86db5d9fcdf29a2
2021-08-10 00:15:53,414 DEBUG: Checking out 'model.pt' with cache 'ref object md5: d05ac82f4c284531f86db5d9fcdf29a2 -> model.pt'.
2021-08-10 00:15:53,427 DEBUG: Created 'copy': .dvc\cache\d0\5ac82f4c284531f86db5d9fcdf29a2 -> model.pt
2021-08-10 00:15:53,433 DEBUG: state save (6349537402451160696, 1628554553424481536, 410191) d05ac82f4c284531f86db5d9fcdf29a2
2021-08-10 00:15:53,468 DEBUG: Preparing to transfer data from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f' to '.dvc\cache'
2021-08-10 00:15:53,471 DEBUG: Preparing to collect status from '.dvc\cache'
2021-08-10 00:15:53,472 DEBUG: Collecting status from '.dvc\cache'
2021-08-10 00:15:53,485 DEBUG: Preparing to collect status from 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f'
2021-08-10 00:15:53,499 DEBUG: state save (8798193123524591996, 1628554552785486336, 71) ffd4ca3f4caefcd3b1c7470517c8ca10
2021-08-10 00:15:53,502 DEBUG: state save (3905119094551895630, 1628554552784489984, 96) acc1b5fdf9aadec17fa600c5b2be5832
2021-08-10 00:15:53,545 DEBUG: Downloading 'memory://dvc-staging/8c14938f07de99a956e292f08bbf45f184096c4d4237001a7093794b8405678f/d6/b00174e60d73e40336502a15f082fc.dir' to '.dvc\cache\d6\b00174e60d73e40336502a15f082fc.dir'
2021-08-10 00:15:53,552 DEBUG: state save (2924731880013320869, 1628554553550481664, 135) d6b00174e60d73e40336502a15f082fc.dir
2021-08-10 00:15:53,555 DEBUG: Checking out 'logs' with cache 'object md5: d6b00174e60d73e40336502a15f082fc.dir'.
2021-08-10 00:15:53,558 DEBUG: Linking directory 'logs'.
2021-08-10 00:15:53,584 DEBUG: Created 'copy': .dvc\cache\ff\d4ca3f4caefcd3b1c7470517c8ca10 -> logs\acc.tsv
2021-08-10 00:15:53,602 DEBUG: Created 'copy': .dvc\cache\ac\c1b5fdf9aadec17fa600c5b2be5832 -> logs\loss.tsv
2021-08-10 00:15:53,609 DEBUG: state save (1457375764667085341, 33843b4f0c48238a234e7aa834791616, 167) d6b00174e60d73e40336502a15f082fc.dir
2021-08-10 00:15:53,643 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:15:53,646 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 00:15:53,649 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:53,651 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:53,652 DEBUG: Running checkpoint callback for stage 'stage: 'train''
Updating lock file 'dvc.lock'
2021-08-10 00:15:54,347 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-9bb5a'
Checkpoint experiment iteration 'd8e6b1f'.
2021-08-10 00:15:54,759 DEBUG: Removing signal file for 'checkpoint' task
file://C:\Users\Administrator\code\vscode-dvc\demo\logs.html
2021-08-10 00:15:54,837 DEBUG: Removing signal file for 'live' task
file://C:\Users\Administrator\code\vscode-dvc\demo\logs.html
2021-08-10 00:15:55,143 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:55,145 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:55,150 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:55,172 DEBUG: {}
2021-08-10 00:15:55,173 DEBUG: Output 'model.pt' didn't change. Skipping saving.
2021-08-10 00:15:55,182 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 00:15:55,199 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 00:15:55,210 DEBUG: {}
2021-08-10 00:15:55,211 DEBUG: Output 'logs' didn't change. Skipping saving.
2021-08-10 00:15:55,223 DEBUG: Computed stage: 'train' md5: '750c12bdda625f4ed5c2f6ecb7b24ac4'
2021-08-10 00:15:55,296 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:55,300 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:55,306 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:55,371 DEBUG: staged tree 'object md5: 0aed307494600d178fbdc0d000d6db38.dir'
2021-08-10 00:15:55,377 DEBUG: state save (8796477465494994334, bc9a18be98bf83123f83e0840190e033, 66544866) 0aed307494600d178fbdc0d000d6db38.dir
2021-08-10 00:15:55,386 DEBUG: state save (4592395663258618475, 1628553983714489344, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 00:15:55,413 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:15:55,415 DEBUG: Checking out 'predictions.json' with cache 'object md5: f43f0fd3621297080b311aa4ccb08ea3'.
2021-08-10 00:15:55,419 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 00:15:55,421 DEBUG: state save (2384021035544222717, 1628554553329487104, 310000) f43f0fd3621297080b311aa4ccb08ea3
2021-08-10 00:15:55,453 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:15:55,456 DEBUG: Checking out 'model.pt' with cache 'object md5: d05ac82f4c284531f86db5d9fcdf29a2'.
2021-08-10 00:15:55,461 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 00:15:55,463 DEBUG: state save (6349537402451160696, 1628554553424481536, 410191) d05ac82f4c284531f86db5d9fcdf29a2
2021-08-10 00:15:55,491 DEBUG: Preparing to transfer data from '.dvc\cache' to '.dvc\cache'
2021-08-10 00:15:55,495 DEBUG: Checking out 'logs' with cache 'object md5: d6b00174e60d73e40336502a15f082fc.dir'.
2021-08-10 00:15:55,499 DEBUG: Linking directory 'logs'.
2021-08-10 00:15:55,514 DEBUG: Skipping copying for 'logs\acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:55,521 DEBUG: Skipping copying for 'logs\loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 00:15:55,528 DEBUG: state save (1457375764667085341, 33843b4f0c48238a234e7aa834791616, 167) d6b00174e60d73e40336502a15f082fc.dir
2021-08-10 00:15:55,552 DEBUG: stage: 'train' was reproduced
To track the changes with git, run:
git add predictions.json dvc.yaml dvc.lock .gitignore 'data\MNIST' params.yaml train.py logs.json
2021-08-10 00:15:56,332 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-9bb5a'
2021-08-10 00:15:56,741 DEBUG: Removing 'C:\Users\Administrator\code\vscode-dvc\demo\.dvc\tmp\exps\run\workspace.run'
2021-08-10 00:15:56,818 DEBUG: Restore HEAD to 'master'
Reproduced experiment(s): exp-9bb5a
Experiment results have been applied to your workspace.
To promote an experiment to a Git branch run:
dvc exp branch <exp>
2021-08-10 00:15:58,346 DEBUG: Analytics is disabled.
Comparing the above with Mac:
2021-08-10 10:20:02,103 DEBUG: Check for update is disabled.
2021-08-10 10:20:02,349 DEBUG: Checkpoint run from 'd4e7040' with baseline '7a957c08afdf1a3cf8276271ebe99f5f35ef321b'
2021-08-10 10:20:02,350 DEBUG: Detaching HEAD at 'HEAD'
2021-08-10 10:20:02,429 DEBUG: Stashing workspace
2021-08-10 10:20:02,575 DEBUG: Stashing changes in 'refs/stash'
2021-08-10 10:20:02,839 DEBUG: Applying stash commit 'fa9ae7eb4442c5a1d58678b522445d6fc34f15d4'
2021-08-10 10:20:02,967 DEBUG: Computed stage: 'data/MNIST/raw.dvc' md5: 'None'
2021-08-10 10:20:02,972 DEBUG: Checking out 'data/MNIST/raw' with cache 'object md5: 8c257df187855c681f88bde92d721ccd.dir'.
2021-08-10 10:20:02,972 DEBUG: Linking directory 'data/MNIST/raw'.
2021-08-10 10:20:02,973 DEBUG: Created 'reflink': .dvc/cache/.cache_type_test_file -> data/MNIST/raw/.ZWSkfnyswW8ubGrixG9Qrh
2021-08-10 10:20:02,973 DEBUG: Removing 'data/MNIST/raw/.ZWSkfnyswW8ubGrixG9Qrh'
2021-08-10 10:20:02,973 DEBUG: Removing '.dvc/cache/.cache_type_test_file'
2021-08-10 10:20:02,974 DEBUG: Removing 'data/MNIST/raw/t10k-images-idx3-ubyte'
2021-08-10 10:20:02,975 DEBUG: Created 'reflink': .dvc/cache/26/46ac647ad5339dbf082846283269ea -> data/MNIST/raw/t10k-images-idx3-ubyte
2021-08-10 10:20:02,976 DEBUG: Removing 'data/MNIST/raw/t10k-images-idx3-ubyte.gz'
2021-08-10 10:20:02,976 DEBUG: Created 'reflink': .dvc/cache/9f/b629c4189551a2d022fa330f9573f3 -> data/MNIST/raw/t10k-images-idx3-ubyte.gz
2021-08-10 10:20:02,977 DEBUG: Removing 'data/MNIST/raw/t10k-labels-idx1-ubyte'
2021-08-10 10:20:02,978 DEBUG: Created 'reflink': .dvc/cache/27/ae3e4e09519cfbb04c329615203637 -> data/MNIST/raw/t10k-labels-idx1-ubyte
2021-08-10 10:20:02,978 DEBUG: Removing 'data/MNIST/raw/t10k-labels-idx1-ubyte.gz'
2021-08-10 10:20:02,979 DEBUG: Created 'reflink': .dvc/cache/ec/29112dd5afa0611ce80d1b7f02629c -> data/MNIST/raw/t10k-labels-idx1-ubyte.gz
2021-08-10 10:20:02,980 DEBUG: Removing 'data/MNIST/raw/train-images-idx3-ubyte'
2021-08-10 10:20:02,982 DEBUG: Created 'reflink': .dvc/cache/6b/bc9ace898e44ae57da46a324031adb -> data/MNIST/raw/train-images-idx3-ubyte
2021-08-10 10:20:02,982 DEBUG: Removing 'data/MNIST/raw/train-images-idx3-ubyte.gz'
2021-08-10 10:20:02,984 DEBUG: Created 'reflink': .dvc/cache/f6/8b3c2dcbeaaa9fbdd348bbdeb94873 -> data/MNIST/raw/train-images-idx3-ubyte.gz
2021-08-10 10:20:02,984 DEBUG: Removing 'data/MNIST/raw/train-labels-idx1-ubyte'
2021-08-10 10:20:02,985 DEBUG: Created 'reflink': .dvc/cache/a2/5bea736e30d166cdddb491f175f624 -> data/MNIST/raw/train-labels-idx1-ubyte
2021-08-10 10:20:02,986 DEBUG: Removing 'data/MNIST/raw/train-labels-idx1-ubyte.gz'
2021-08-10 10:20:02,987 DEBUG: Created 'reflink': .dvc/cache/d5/3e105ee54ea40749a09fcbcd1e9432 -> data/MNIST/raw/train-labels-idx1-ubyte.gz
2021-08-10 10:20:02,988 DEBUG: state save (118981590, ebf8445244fbfbda0afe7f90695c062f, 66544770) 8c257df187855c681f88bde92d721ccd.dir
2021-08-10 10:20:02,991 DEBUG: Saving information to 'data/MNIST/raw.dvc'.
Existing checkpoint experiment 'exp-b7ef0' will be resumed
2021-08-10 10:20:03,426 DEBUG: Stashing changes in 'refs/exps/stash'
2021-08-10 10:20:04,248 DEBUG: Stashed experiment '8b4fa8c' with baseline '7a957c0' for future execution.
2021-08-10 10:20:04,334 DEBUG: Restoring stashed workspace
2021-08-10 10:20:04,334 DEBUG: Popping from stash 'refs/stash'
2021-08-10 10:20:04,335 DEBUG: Applying stash commit 'fa9ae7eb4442c5a1d58678b522445d6fc34f15d4'
2021-08-10 10:20:04,416 DEBUG: Dropping 'refs/stash@{0}'
2021-08-10 10:20:04,421 DEBUG: Restore HEAD to 'master'
2021-08-10 10:20:04,424 DEBUG: Detaching HEAD at 'd4e7040cc794510b21414cb222f2cf9d7e3ec3df'
2021-08-10 10:20:05,396 DEBUG: Popping from stash 'refs/exps/stash'
2021-08-10 10:20:05,398 DEBUG: Applying stash commit '8b4fa8c57d477e51d2b3bb26bbe84f9d4395bf2a'
2021-08-10 10:20:05,479 DEBUG: Dropping 'refs/exps/stash@{0}'
2021-08-10 10:20:05,509 DEBUG: Running repro in '/Users/mattseddon/PP/vscode-dvc/demo'
2021-08-10 10:20:05,509 DEBUG: Removing '/Users/mattseddon/PP/vscode-dvc/demo/.dvc/tmp/repro.dat'
2021-08-10 10:20:05,815 DEBUG: Computed stage: 'data/MNIST/raw.dvc' md5: 'None'
'data/MNIST/raw.dvc' didn't change, skipping
2021-08-10 10:20:05,821 DEBUG: stage: 'train' changed.
2021-08-10 10:20:05,824 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 10:20:05,824 DEBUG: Removing output 'predictions.json' of stage: 'train'.
2021-08-10 10:20:05,824 DEBUG: Removing 'predictions.json'
2021-08-10 10:20:05,825 DEBUG: Skipping copying for 'logs/loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 10:20:05,825 DEBUG: Skipping copying for 'logs/acc.tsv', since it is not a symlink or a hardlink.
Running stage 'train':
> python train.py
/Users/mattseddon/PP/vscode-dvc/demo/.env/lib/python3.9/site-packages/torchvision/datasets/mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:180.)
return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)
/Users/mattseddon/PP/vscode-dvc/demo/.env/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at ../c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
2021-08-10 10:20:19,872 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:19,885 DEBUG: state save (121870457, 1628554818945782784, 410191) dafd1a34fa7b2c4c6abe5a8cc8d3693e
2021-08-10 10:20:19,885 DEBUG: state save (121870457, 1628554818945782784, 410191) dafd1a34fa7b2c4c6abe5a8cc8d3693e
2021-08-10 10:20:19,886 DEBUG: {'model.pt': 'modified'}
2021-08-10 10:20:19,889 DEBUG: state save (121870708, 1628554819645959168, 310000) 1ff05c79f3f0bcec32e269f997c1cdfb
2021-08-10 10:20:19,889 DEBUG: state save (121870708, 1628554819645959168, 310000) 1ff05c79f3f0bcec32e269f997c1cdfb
2021-08-10 10:20:19,890 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 10:20:19,893 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 10:20:19,898 DEBUG: state save (121870459, 1628554819648622336, 129) 5b5c195c3c119abb6587197d7c821388
2021-08-10 10:20:19,898 DEBUG: state save (121870458, 1628554819648974080, 92) edd15f1ee541e324e567d5ec7bff0fb6
2021-08-10 10:20:19,903 DEBUG: Downloading 'memory://.ljbthbsdsofxaq8ghetl4e.tmp' to '.dvc/tmp/staging/93/2fa33490a2dedd10a9038d8cbaeb28.dir'
2021-08-10 10:20:19,905 DEBUG: state save (121870714, 1628554819904773376, 135) 932fa33490a2dedd10a9038d8cbaeb28.dir
2021-08-10 10:20:19,906 DEBUG: state save (120697374, 546ae19a4419f1cb664030cee67a52d1, 221) 932fa33490a2dedd10a9038d8cbaeb28.dir
2021-08-10 10:20:19,906 DEBUG: {'logs': 'modified'}
2021-08-10 10:20:19,909 DEBUG: Computed stage: 'train' md5: '0edb024834435569fba03080acd9a006'
2021-08-10 10:20:19,911 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:19,914 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:19,920 DEBUG: Checking out 'predictions.json' with cache 'object md5: 1ff05c79f3f0bcec32e269f997c1cdfb'.
2021-08-10 10:20:19,920 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 10:20:19,921 DEBUG: state save (121870708, 1628554819645959168, 310000) 1ff05c79f3f0bcec32e269f997c1cdfb
2021-08-10 10:20:19,936 DEBUG: Checking out 'model.pt' with cache 'object md5: dafd1a34fa7b2c4c6abe5a8cc8d3693e'.
2021-08-10 10:20:19,938 DEBUG: Created 'reflink': .dvc/cache/.cache_type_test_file -> .F5ReMpb79oagqh3PYdgcpi
2021-08-10 10:20:19,938 DEBUG: Removing '.F5ReMpb79oagqh3PYdgcpi'
2021-08-10 10:20:19,938 DEBUG: Removing '.dvc/cache/.cache_type_test_file'
2021-08-10 10:20:19,939 DEBUG: Removing 'model.pt'
2021-08-10 10:20:19,940 DEBUG: Created 'reflink': .dvc/cache/da/fd1a34fa7b2c4c6abe5a8cc8d3693e -> model.pt
2021-08-10 10:20:19,940 DEBUG: state save (121870718, 1627872723336442368, 410191) dafd1a34fa7b2c4c6abe5a8cc8d3693e
2021-08-10 10:20:19,949 DEBUG: state save (121870459, 1628554819648622336, 129) 5b5c195c3c119abb6587197d7c821388
2021-08-10 10:20:19,950 DEBUG: state save (121870458, 1628554819648974080, 92) edd15f1ee541e324e567d5ec7bff0fb6
2021-08-10 10:20:19,953 DEBUG: state save (121870714, 1628554819904773376, 135) 932fa33490a2dedd10a9038d8cbaeb28.dir
2021-08-10 10:20:19,953 DEBUG: Checking out 'logs' with cache 'object md5: 932fa33490a2dedd10a9038d8cbaeb28.dir'.
2021-08-10 10:20:19,953 DEBUG: Linking directory 'logs'.
2021-08-10 10:20:19,954 DEBUG: Created 'reflink': .dvc/cache/ed/d15f1ee541e324e567d5ec7bff0fb6 -> logs/acc.tsv
2021-08-10 10:20:19,955 DEBUG: Created 'reflink': .dvc/cache/5b/5c195c3c119abb6587197d7c821388 -> logs/loss.tsv
2021-08-10 10:20:19,956 DEBUG: state save (120697374, 546ae19a4419f1cb664030cee67a52d1, 221) 932fa33490a2dedd10a9038d8cbaeb28.dir
2021-08-10 10:20:19,962 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 10:20:19,962 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 10:20:19,963 DEBUG: Skipping copying for 'logs/loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 10:20:19,963 DEBUG: Skipping copying for 'logs/acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 10:20:19,963 DEBUG: Running checkpoint callback for stage 'stage: 'train''
Updating lock file 'dvc.lock'
2021-08-10 10:20:20,331 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-b7ef0'
Checkpoint experiment iteration '040fad5'.
2021-08-10 10:20:20,492 DEBUG: Removing signal file for 'checkpoint' task
file:///Users/mattseddon/PP/vscode-dvc/demo/logs.html
2021-08-10 10:20:20,608 DEBUG: Removing signal file for 'live' task
2021-08-10 10:20:33,648 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:33,656 DEBUG: state save (121870718, 1628554832895071488, 410191) a663a400660b676706f733471fe30c3a
2021-08-10 10:20:33,656 DEBUG: state save (121870718, 1628554832895071488, 410191) a663a400660b676706f733471fe30c3a
2021-08-10 10:20:33,657 DEBUG: {'model.pt': 'modified'}
2021-08-10 10:20:33,660 DEBUG: state save (121870708, 1628554833581150464, 310000) f02454d71235f329965d641cf38dced5
2021-08-10 10:20:33,660 DEBUG: state save (121870708, 1628554833581150464, 310000) f02454d71235f329965d641cf38dced5
2021-08-10 10:20:33,660 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 10:20:33,664 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 10:20:33,668 DEBUG: state save (121870720, 1628554833583878912, 165) 36aa797bc56b77935d3a11508e2ac764
2021-08-10 10:20:33,671 DEBUG: state save (121870719, 1628554833584843520, 116) 3ce0af943056234ccc16caa24fa55a2c
2021-08-10 10:20:33,674 DEBUG: Downloading 'memory://.dnzj4ev5irz7du9pawjvt8.tmp' to '.dvc/tmp/staging/e5/f4fb4c21072504159b8d1c57164ad1.dir'
2021-08-10 10:20:33,675 DEBUG: state save (121870743, 1628554833674944256, 135) e5f4fb4c21072504159b8d1c57164ad1.dir
2021-08-10 10:20:33,676 DEBUG: state save (120697374, 562c9e908c16e27b6e090a929ce5e987, 281) e5f4fb4c21072504159b8d1c57164ad1.dir
2021-08-10 10:20:33,676 DEBUG: {'logs': 'modified'}
2021-08-10 10:20:33,678 DEBUG: Computed stage: 'train' md5: '5001035be996f8d433f5fb96254ce5cf'
2021-08-10 10:20:33,680 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:33,684 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:33,688 DEBUG: Checking out 'predictions.json' with cache 'object md5: f02454d71235f329965d641cf38dced5'.
2021-08-10 10:20:33,688 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 10:20:33,688 DEBUG: state save (121870708, 1628554833581150464, 310000) f02454d71235f329965d641cf38dced5
2021-08-10 10:20:33,701 DEBUG: Checking out 'model.pt' with cache 'object md5: a663a400660b676706f733471fe30c3a'.
2021-08-10 10:20:33,702 DEBUG: Removing 'model.pt'
2021-08-10 10:20:33,703 DEBUG: Created 'reflink': .dvc/cache/a6/63a400660b676706f733471fe30c3a -> model.pt
2021-08-10 10:20:33,703 DEBUG: state save (121870745, 1627872736702185728, 410191) a663a400660b676706f733471fe30c3a
2021-08-10 10:20:33,712 DEBUG: state save (121870719, 1628554833584843520, 116) 3ce0af943056234ccc16caa24fa55a2c
2021-08-10 10:20:33,712 DEBUG: state save (121870720, 1628554833583878912, 165) 36aa797bc56b77935d3a11508e2ac764
2021-08-10 10:20:33,717 DEBUG: state save (121870743, 1628554833674944256, 135) e5f4fb4c21072504159b8d1c57164ad1.dir
2021-08-10 10:20:33,717 DEBUG: Checking out 'logs' with cache 'object md5: e5f4fb4c21072504159b8d1c57164ad1.dir'.
2021-08-10 10:20:33,717 DEBUG: Linking directory 'logs'.
2021-08-10 10:20:33,718 DEBUG: Created 'reflink': .dvc/cache/3c/e0af943056234ccc16caa24fa55a2c -> logs/acc.tsv
2021-08-10 10:20:33,719 DEBUG: Created 'reflink': .dvc/cache/36/aa797bc56b77935d3a11508e2ac764 -> logs/loss.tsv
2021-08-10 10:20:33,720 DEBUG: state save (120697374, 562c9e908c16e27b6e090a929ce5e987, 281) e5f4fb4c21072504159b8d1c57164ad1.dir
2021-08-10 10:20:33,756 DEBUG: Skipping copying for 'model.pt', since it is not a symlink or a hardlink.
2021-08-10 10:20:33,756 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 10:20:33,756 DEBUG: Skipping copying for 'logs/loss.tsv', since it is not a symlink or a hardlink.
2021-08-10 10:20:33,757 DEBUG: Skipping copying for 'logs/acc.tsv', since it is not a symlink or a hardlink.
2021-08-10 10:20:33,757 DEBUG: Running checkpoint callback for stage 'stage: 'train''
Updating lock file 'dvc.lock'
2021-08-10 10:20:34,096 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-b7ef0'
Checkpoint experiment iteration 'd8ee0bb'.
2021-08-10 10:20:34,252 DEBUG: Removing signal file for 'checkpoint' task
file:///Users/mattseddon/PP/vscode-dvc/demo/logs.html
2021-08-10 10:20:34,280 DEBUG: Removing signal file for 'live' task
file:///Users/mattseddon/PP/vscode-dvc/demo/logs.html
2021-08-10 10:20:34,463 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:34,468 DEBUG: {}
2021-08-10 10:20:34,468 DEBUG: Output 'model.pt' didn't change. Skipping saving.
2021-08-10 10:20:34,469 DEBUG: Output 'predictions.json' doesn't use cache. Skipping saving.
2021-08-10 10:20:34,473 DEBUG: Adding 'logs' to '.gitignore'.
2021-08-10 10:20:34,474 DEBUG: {}
2021-08-10 10:20:34,474 DEBUG: Output 'logs' didn't change. Skipping saving.
2021-08-10 10:20:34,475 DEBUG: Computed stage: 'train' md5: '5001035be996f8d433f5fb96254ce5cf'
2021-08-10 10:20:34,477 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:34,480 DEBUG: state save (121426853, 1628060074138815488, 3227) 9e42a611238f6e367b44cc3a82369b3a
2021-08-10 10:20:34,486 DEBUG: Checking out 'predictions.json' with cache 'object md5: f02454d71235f329965d641cf38dced5'.
2021-08-10 10:20:34,487 DEBUG: Skipping copying for 'predictions.json', since it is not a symlink or a hardlink.
2021-08-10 10:20:34,487 DEBUG: state save (121870708, 1628554833581150464, 310000) f02454d71235f329965d641cf38dced5
2021-08-10 10:20:34,493 DEBUG: Checking out 'model.pt' with cache 'object md5: a663a400660b676706f733471fe30c3a'.
2021-08-10 10:20:34,494 DEBUG: Removing 'model.pt'
2021-08-10 10:20:34,495 DEBUG: Created 'reflink': .dvc/cache/a6/63a400660b676706f733471fe30c3a -> model.pt
2021-08-10 10:20:34,495 DEBUG: state save (121870759, 1627872736702185728, 410191) a663a400660b676706f733471fe30c3a
2021-08-10 10:20:34,502 DEBUG: Checking out 'logs' with cache 'object md5: e5f4fb4c21072504159b8d1c57164ad1.dir'.
2021-08-10 10:20:34,503 DEBUG: Linking directory 'logs'.
2021-08-10 10:20:34,504 DEBUG: state save (121870746, 1628554833584843520, 116) 3ce0af943056234ccc16caa24fa55a2c
2021-08-10 10:20:34,504 DEBUG: state save (121870746, 1628554833584843520, 116) 3ce0af943056234ccc16caa24fa55a2c
2021-08-10 10:20:34,505 DEBUG: Removing 'logs/acc.tsv'
2021-08-10 10:20:34,506 DEBUG: Created 'reflink': .dvc/cache/3c/e0af943056234ccc16caa24fa55a2c -> logs/acc.tsv
2021-08-10 10:20:34,507 DEBUG: state save (121870747, 1628554833583878912, 165) 36aa797bc56b77935d3a11508e2ac764
2021-08-10 10:20:34,507 DEBUG: state save (121870747, 1628554833583878912, 165) 36aa797bc56b77935d3a11508e2ac764
2021-08-10 10:20:34,508 DEBUG: Removing 'logs/loss.tsv'
2021-08-10 10:20:34,508 DEBUG: Created 'reflink': .dvc/cache/36/aa797bc56b77935d3a11508e2ac764 -> logs/loss.tsv
2021-08-10 10:20:34,509 DEBUG: state save (120697374, 562c9e908c16e27b6e090a929ce5e987, 281) e5f4fb4c21072504159b8d1c57164ad1.dir
2021-08-10 10:20:34,515 DEBUG: stage: 'train' was reproduced
To track the changes with git, run:
git add logs.json dvc.lock predictions.json dvc.yaml params.yaml data/MNIST .gitignore train.py
2021-08-10 10:20:34,864 DEBUG: Commit to current experiment branch 'refs/exps/7a/957c08afdf1a3cf8276271ebe99f5f35ef321b/exp-b7ef0'
2021-08-10 10:20:35,022 DEBUG: Removing '/Users/mattseddon/PP/vscode-dvc/demo/.dvc/tmp/exps/run/workspace.run'
2021-08-10 10:20:35,028 DEBUG: Restore HEAD to 'master'
Reproduced experiment(s): exp-b7ef0
Experiment results have been applied to your workspace.
To promote an experiment to a Git branch run:
dvc exp branch <exp>
2021-08-10 10:20:35,071 DEBUG: Analytics is disabled.
We can see Existing checkpoint experiment 'exp-b7ef0' will be resumed is in the log. This doesn’t seem to be happening on the Windows machine at all.
I can also confirm that I have only seen this behaviour on this project on the Windows box (I have checkout out get-started-experiments and have no problem running multiple experiments underneath a single commit).
Expected
We can resume or run a second experiment underneath any commit across all platforms.
Environment information
Output of dvc doctor:
$ dvc doctor
DVC version: 2.5.4+63b710
---------------------------------
Platform: Python 3.9.6 on Windows-10-10.0.19042-SP0
Supports:
http (requests = 2.26.0),
https (requests = 2.26.0)
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: https
Workspace directory: NTFS on C:\
Repo: dvc (subdir), git
Additional Information (if any):
Please let me know if you need any more details. As you can see I did have a dig around in pygit2 but I suspect that if there is an actual bug in pygit2 then it exists in the C portion of the code (i.e in .env\Lib\site-packages\pygit2\_libgit2.pyd).
It should also be noted that this will only become a pressing issue when we release OR if we get another candidate come through with a Windows machine and even then we can provision a VM.
Thanks,
Issue Analytics
- State:
- Created 2 years ago
- Comments:6 (2 by maintainers)
Top Results From Across the Web
Remote Development Tips and Tricks - Visual Studio Code
This section includes some tips and tricks for getting the Remote - SSH extension up and running in different environments.
Read more >Source Control with Git in Visual Studio Code
First run Source Control experience. If you select Clone Repository, you will be asked for the URL of the remote repository (for example...
Read more >Remote Development FAQ - Visual Studio Code
Remote Development FAQ. This article covers frequently asked questions for each of the Visual Studio Code Remote Development extensions.
Read more >Visual Studio Code Server
The Visual Studio Code Server is a service you can run on a remote development machine, like your desktop PC or a virtual...
Read more >Troubleshoot Terminal launch failures - Visual Studio Code
Try running your designated integrated terminal shell outside VS Code from an external terminal or command prompt. Some terminal launch failures may be...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Ok, so it’s related to this issue: https://github.com/libgit2/libgit2/pull/4260
Basically, during execution on windows, we are cd’d into the demo directory (which is a subdir of the git repo root). libgit2 normally tries to delete the dir on checkout (along with any other git repo subdirs that it needs to re-build on checkout), but there is an open file handle for the current working dir, so the rmdir call fails.
On windows we need to be using the
GIT_CHECKOUT_SKIP_LOCKED_DIRECTORIESflag for libgit/pygit checkout which accounts for this behavior. This will require an upstream pygit patch since that constant is not imported right now, but we can also just use it directly in DVC temporarily.I can confirm that example-get-started is also fine.
I was running in
git bashthrough VS Code. I have just tried to use vanillagit bashand the result is the same.