Stating dependencies between scripts/modules

Hello!

I hope the explanation below is clear. Please let me know otherwise.

Say I have a utils.py with some awesome helpful classes that I reuse frequently in a certain dvc tracked repo.

Say I have:

• script1.py that uses utils.py and that takes data0.csv and processes it to data1.csv
• script2.py that uses utils.py and that takes data1.csv and processes it to data2.csv
• script3.py that uses utils.py and that takes data2.csv and processes it to data3.csv
• etc …

(In the example above all scripts are part of the same pipeline but it they could be from different pipelines.)

The point that perhaps could be improved is that, as far as I know, for each data*.csv I have to add to its dependencies the correspondent script and utils.py. And of course that this can cascade if utils.py depends on utils1.py which depends on utils2.py, etc… If that is the case, then I have to remember to, every time utils.py is a dependency, to include the others utils*.py as dependencies as well.

Is there a way in the dvc to say that a scriptB.py depends on scriptA.py so that every time scriptB.py is a dependency, then scriptA.py is also an implicit dependency? Like a variant or an alternative to dvc run where the “output” is not a data file but a .py?

Thanks is advance!

[efiop]

Hi @anotherbugmaster !

Looks like I’ve indeed missed something in the original request, but for the issue you’ve described we have another ticket https://github.com/iterative/dvc/issues/1577 . Please take a look and let us know if that is what you mean 😃

Thanks for the feedback!

[efiop]

Hi @andrethrill !

Amazing idea! We could totally support that, by implementing something like --no-remove-outs(we will figure out a better name, suggestions are welcome 🙂 ) https://github.com/iterative/dvc/issues/1214 and using it with the already existing -O|--outs-no-cache option. So your command would look like dvc run -d scriptA.py -O scriptB.py --no-remove-outs.

Thanks, Ruslan