exp run: python: can't open file 'src/train.py'

Bug Report

Description

I added a couple of experiments with different parameters to queue. I wanted to run them sequentially with dvc exp run --run-all but I got some unexpected results:

1. I realize dvc checkout is runned automatically for each experiment. I think this is bad since I have to wait all my images and annotations were “checkout” once per experiment. I did not know why this was happening until I read this line in the source code: https://github.com/iterative/dvc/blob/master/dvc/command/experiments.py#L1293 That implies --temp means that “each exp is runned in a separate temporary directory instead of your workspace”. I guess this is related with parallelizing different runs… but I consider it should be possible to avoid it since it does not scale with large datasets.

2. The error I am pointing in the issue name python: can't open file 'src/train.py' is appearing because of --temp parameters that forces to commit in both dvc and git the src/train.py so file is present later in the tmp copy (this is only my current belief, I could not confirm this yet) Everything works fine with dvc repro.

Reproduce

1. dvc init
2. import and add dataset
3. create src/train.py script
4. dvc exp run --queue
5. dvc exp run --queue -S optimizer.lr=0.1
6. dvc exp run --run-all

Expected

I expected that my experiment runned in sequential order without duplicating my data in another tmp directory.

Environment information

Output of dvc doctor:

$ dvc doctor

DVC version: 2.5.4 (conda)
---------------------------------
Platform: Python 3.8.10 on Linux-4.15.0-96-generic-x86_64-with-glibc2.10
Supports:
        gdrive (pydrive2 = 1.9.1),
        http (requests = 2.26.0),
        https (requests = 2.26.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb1
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

Additional Information (if any):

$ dvc exp show 
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓               
┃ Experiment   ┃ Created      ┃ test_split.split ┃ optimizer.lr ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ workspace    │ -            │ -                │ -            │
│ main         │ Aug 19, 2021 │ -                │ -            │
│ ├── *098b5bc │ 01:07 PM     │ 0.1              │ 0.1          │
│ └── *6980253 │ 01:06 PM     │ -                │ -            │
└──────────────┴──────────────┴──────────────────┴──────────────┘

[daavoo]

This might be related: https://github.com/iterative/dvc/issues/6490

[pmrowla]

This is expected behavior since src/train.py is not tracked by git - from the exp run docs (https://dvc.org/doc/command-reference/exp/run#queueing-and-parallel-execution):

⚠️ Note that only tracked files and directories will be included in --queue/temp experiments. To include untracked files, stage them with git add first (before dvc exp run).

As you noted, the --queue/--run-all functionality is limited right now, and only supports running experiments in their own separate tempdir workspaces. This this behavior is also documented:

Use dvc exp run --run-all to process the queue. This is done outside your workspace (in temporary dirs in .dvc/tmp/exps) to preserve any changes between/after queueing runs.

To run an experiment in your main repo working directory, currently you cannot use the --queue/--run-all functionality. But if you just do

dvc exp run -S optimizer.lr=0.1

it will run in your workspace the same way as dvc repro (and it will work properly even with an untracked src/train.py file).