Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

exp run: missing dependencies of frozen stages raise an error

See original GitHub issue

Bug Report

Description

When calling dvc exp run with a frozen stage, an error is raise if dependencies of that frozen stage or outputs of stages before that stage are missing in the project directory.

Reproduce

stages:
  dep:
    cmd: mkdir dep && echo 'dep' > dep/dep.txt
    deps:
      - start.txt
    outs:
      - dep

  frozen:
    cmd: echo 'frozen' > frozen.txt
    deps:
      - dep
    outs:
      - frozen.txt

  next:
    cmd: echo 'next' > next.txt
    deps:
      - frozen.txt
    outs:
      - next.txt
  1. Save the dvc.yaml above
  2. dvc config --local cache.type reflink,hardlink,symlink
  3. echo start > start.txt
  4. dvc repro
  5. rm $(readlink dep/dep.txt) start.txt frozen.txt
  6. rm -rf dep
  7. echo new > frozen.txt
  8. dvc freeze frozen
  9. dvc exp run dvc.yaml
    • The same error is raised by --temp and --queue then --run-all
$ dvc exp run --temp -v dvc.yaml
2021-11-06 04:23:39,265 DEBUG: Detaching HEAD at 'HEAD'               
2021-11-06 04:23:39,283 DEBUG: Stashing workspace
2021-11-06 04:23:39,394 DEBUG: Stashing changes in 'refs/stash'
2021-11-06 04:23:39,417 DEBUG: Applying stash commit 'cd53502fcbe907f71d45df87420d7466bfaa2ecf'
2021-11-06 04:23:40,119 DEBUG: Stashing changes in 'refs/exps/stash'
2021-11-06 04:23:40,163 DEBUG: Stashed experiment 'f03880d' with baseline '6ffd8ba' for future execution.
2021-11-06 04:23:40,171 DEBUG: Restoring stashed workspace
2021-11-06 04:23:40,171 DEBUG: Popping from stash 'refs/stash'
2021-11-06 04:23:40,172 DEBUG: Applying stash commit 'cd53502fcbe907f71d45df87420d7466bfaa2ecf'
2021-11-06 04:23:40,180 DEBUG: Dropping 'refs/stash@{0}'
2021-11-06 04:23:40,185 DEBUG: Restore HEAD to 'master'
2021-11-06 04:23:40,186 DEBUG: Reproducing experiment revs 'f03880d'
2021-11-06 04:23:40,512 DEBUG: Writing experiments local config '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/.dvc/config.local'                                                                                                                         
2021-11-06 04:23:40,512 DEBUG: Init temp dir executor in dir '<ExpTemporaryDirectory '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'>'
2021-11-06 04:23:40,602 DEBUG: Running repro in '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'
2021-11-06 04:23:40,603 DEBUG: Removing '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/.dvc/tmp/repro.dat'
2021-11-06 04:23:41,127 DEBUG: Cache type 'reflink' is not supported: reflink is not supported                                                                                                                                                         
2021-11-06 04:23:41,127 DEBUG: Cache type 'hardlink' is not supported: failed to link                                                                                                                                                                  
2021-11-06 04:23:41,128 DEBUG: Created 'symlink': ../../../../../cache/4d/70e93c98a3a5ff1e7e5160d909a9b9 -> next.txt                                                                                                                                   
2021-11-06 04:23:41,129 DEBUG: state save (58985507, 1636170526467139584, 5) 4d70e93c98a3a5ff1e7e5160d909a9b9                                                                                                                                          
2021-11-06 04:23:41,144 DEBUG: state save (58985507, 1636170526467139584, 5) 4d70e93c98a3a5ff1e7e5160d909a9b9                                                                                                                                          
2021-11-06 04:23:41,145 DEBUG: Created 'symlink': ../../../../../cache/9c/d599a3523898e6a12e13ec787da50a -> frozen.txt                                                                                                                                 
2021-11-06 04:23:41,146 DEBUG: state save (58985508, 1636171699415656448, 4) 9cd599a3523898e6a12e13ec787da50a                                                                                                                                          
2021-11-06 04:23:41,154 DEBUG: state save (58985508, 1636171699415656448, 4) 9cd599a3523898e6a12e13ec787da50a                                                                                                                                          
2021-11-06 04:23:41,156 DEBUG: Created 'symlink': ../../../../../cache/2a/4732b1de5db823e94d662d207b8fb2 -> dep/dep.txt                                                                                                                                
2021-11-06 04:23:41,157 ERROR: unexpected error - [Errno 2] No such file or directory: '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/dep/dep.txt'                                                                                         
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/experiments/executor/base.py", line 446, in _repro_dvc
    yield dvc
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/experiments/executor/base.py", line 343, in reproduce
    dvc_checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/repo/checkout.py", line 98, in checkout
    result = stage.checkout(
  File "/usr/local/lib/python3.8/site-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/stage/decorators.py", line 36, in rwlocked
    return call()
  File "/usr/local/lib/python3.8/site-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/stage/__init__.py", line 572, in checkout
    key, outs = self._checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/stage/__init__.py", line 584, in _checkout
    result = out.checkout(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/dvc/output.py", line 746, in checkout
    modified = checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/objects/checkout.py", line 275, in checkout
    _checkout(
  File "/usr/local/lib/python3.8/site-packages/dvc/objects/checkout.py", line 221, in _checkout
    _checkout_file(
  File "/usr/local/lib/python3.8/site-packages/dvc/objects/checkout.py", line 151, in _checkout_file
    state.save(path_info, fs, change.new.obj.hash_info)
  File "/usr/local/lib/python3.8/site-packages/dvc/state.py", line 82, in save
    mtime, size = get_mtime_and_size(path_info, fs, self.dvcignore)
  File "/usr/local/lib/python3.8/site-packages/dvc/utils/fs.py", line 63, in get_mtime_and_size
    base_stat = fs.info(path)
  File "/usr/local/lib/python3.8/site-packages/dvc/fs/local.py", line 160, in info
    return self.fs.info(path_info)
  File "/usr/local/lib/python3.8/site-packages/fsspec/implementations/local.py", line 83, in info
    out = os.stat(path, follow_symlinks=True)
FileNotFoundError: [Errno 2] No such file or directory: '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk/dep/dep.txt'
------------------------------------------------------------
2021-11-06 04:23:41,159 DEBUG: Removing '/home/sjawhar/example/.dvc/tmp/exps/run/f03880d31cbf3d857c2aba7c6057a50fa276ed84.run'
2021-11-06 04:23:41,164 ERROR: Failed to reproduce experiment 'f03880d'
2021-11-06 04:23:41,164 DEBUG: Removing tmpdir '<ExpTemporaryDirectory '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'>'
2021-11-06 04:23:41,165 DEBUG: Removing '/home/sjawhar/example/.dvc/tmp/exps/tmpl3byffdk'
2021-11-06 04:23:41,178 DEBUG: Dropping 'refs/exps/stash@{0}'         
2021-11-06 04:23:41,181 DEBUG: Analytics is enabled.
2021-11-06 04:23:41,226 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp4qhfkjh0']'
2021-11-06 04:23:41,227 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp4qhfkjh0']'

Expected

An experiment to be run successfully, starting from stage next

Environment information

I’m using the symlink cache type

Output of dvc doctor:

DVC version: 2.8.1 (pip)
---------------------------------
Platform: Python 3.8.8 on Linux-5.13.0-7614-generic-x86_64-with-glibc2.2.5
Supports:
        hdfs (fsspec = 2021.10.0, pyarrow = 4.0.1),
        webhdfs (fsspec = 2021.10.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2021.10.0, boto3 = 1.17.106),
        ssh (sshfs = 2021.11.0)
Cache types: symlink
Cache directory: ext4 on /dev/mapper/data-root
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/mapper/data-root
Repo: dvc, git

Additional Information (if any): From some basic debugging, I think I’ve tracked this down to how experiments collect stages. Specifically, they call StageLoad.collect(with_deps=True), whereas dvc repro doesn’t. So the experiment seems to be pulling extra stages.

https://github.com/iterative/dvc/blob/be6128470f6e7d590f889b20ea24bdde115f059e/dvc/repo/experiments/__init__.py#L288-L305

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:7 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
paredcommented, Nov 15, 2021

Seems like I am able to reproduce on mac:

#!/bin/bash

set -ex

pushd $TMPDIR
rm -rf test_wspace
mkdir test_wspace
pushd test_wspace

mkdir test_repo
pushd test_repo

git init 
dvc init

dvc config --local cache.type symlink

git add -A
git commit -am "init"

echo start >> start.txt

echo -e "stages:
  dep:
    cmd: mkdir dep && echo 'dep' > dep/dep.txt
    deps:
      - start.txt
    outs:
      - dep

  frozen:
    cmd: echo 'frozen' > frozen.txt
    deps:
      - dep
    outs:
      - frozen.txt

  next:
    cmd: echo 'next' > next.txt
    deps:
      - frozen.txt
    outs:
      - next.txt" >> dvc.yaml

dvc repro next
git add -A
git commit -am "ble"

rm -f $(readlink dep/dep.txt) start.txt frozen.txt
rm -rf dep

echo new > frozen.txt
dvc freeze frozen
dvc exp run dvc.yaml

version:

DVC version: 2.8.4.dev3+g87fadd32 (osxpkg)
---------------------------------
Platform: Python 3.8.12 on macOS-11.6-x86_64-i386-64bit
Supports:
        azure (adlfs = 2021.9.1, knack = 0.8.2, azure-identity = 1.7.0),
        gdrive (pydrive2 = 1.10.0),
        gs (gcsfs = 2021.11.0),
        hdfs (fsspec = 2021.11.0, pyarrow = 5.0.0),
        webhdfs (fsspec = 2021.11.0),
        http (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.7.4.post0, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2021.11.0, boto3 = 1.17.106),
        ssh (sshfs = 2021.9.0),
        oss (ossfs = 2021.8.0),
        webdav (webdav4 = 0.9.3),
        webdavs (webdav4 = 0.9.3)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git
1reaction
sjawharcommented, Nov 16, 2021

I’m running on Linux (Pop!OS 21.04), so this isn’t a Mac thing. Also, just for clarity, the rm $(readlink) step is just to simulate someone trying to reproduce a step without pulling cache for upstream, frozen stages (e.g. because they are very large)

Read more comments on GitHub >

github_iconTop Results From Across the Web

ROCm fails to install from APT repository in 22.04 · Issue #1713
Ubuntu 22.04's feature freeze has already passed and the version in the APT repository is not installable due to missing dependencies (in ...
Read more >
Troubleshooting | Data Version Control - DVC
The most common cause is changes pushed to Git without the corresponding data being uploaded to the DVC remote. Make sure to dvc...
Read more >
Error Codes | Yarn - Package Manager
The author of packageA can fix this problem by adding a peer dependency on packagePeer . If relevant, they can use optional peer...
Read more >
npm ci can only install packages with an ... - Stack Overflow
After a lot of research, I was able to figure out that this happens when you are not using npm install for installing...
Read more >
Troubleshooting common AWS CDK issues
When deploying my AWS CDK stack, I receive a NoSuchBucket error ... To run a locally installed AWS CDK Toolkit, use the command...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Possible Issue with Python 3.9.8

Next page

exp apply: safeguard against unclean workspaces