Failed to build DAG DAG("No name")

Hello! Just started to check out ploomber. Was going through the first pipeline example in the docs.

After doing a ploomber new, I then did ploomber build with errors below:

(ploomber_dev) λ ploomber build Rendering DAG: 100%|████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 14.19it/s] Executing: 100%|██████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:08<00:00, 1.60s/cell] Building task “raw”: 100%|██████████████████████████████████████████████████████████████████████████████| 3/3 [00:08<00:00, 2.69s/it] Traceback (most recent call last): File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\dag\DAG.py”, line 399, in _build task_reports = self._executor(dag=self, File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\executors\serial.py”, line 134, in call raise DAGBuildError('DAG build failed, the following ’ ploomber.exceptions.DAGBuildError: DAG build failed, the following tasks crashed (corresponding downstream tasks aborted execution):

• NotebookRunner: raw -> MetaProduct({‘nb’: File(D:\gitprojects\ploomber_dev\tutorial\output\data.ipynb), ‘data’: File(D:\gitprojects\ploomber_dev\tutorial\output\data.csv)}) Traceback (most recent call last): File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\tasks\Task.py”, line 456, in _build res = self._run() File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\tasks\Task.py”, line 545, in _run self.run() File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\tasks\notebook.py”, line 437, in run tmp.unlink() File “D:\Python38\lib\pathlib.py”, line 1324, in unlink self._accessor.unlink(self) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: ‘C:\Users\some_user\AppData\Local\Temp\tmp1jax1dtj.ipynb’

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\executors\serial.py”, line 186, in catch_exceptions fn() File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\executors\serial.py”, line 165, in call return self.fn(**self.kwargs) File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\executors\serial.py”, line 170, in catch_warnings result = fn() File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\executors\serial.py”, line 165, in call return self.fn(**self.kwargs) File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\executors\serial.py”, line 232, in build_in_subprocess report, meta = task._build(**build_kwargs) File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\tasks\Task.py”, line 469, in _build raise TaskBuildError(msg) from e ploomber.exceptions.TaskBuildError: Error building task “raw”

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “D:\Python38\lib\runpy.py”, line 194, in _run_module_as_main return _run_code(code, main_globals, None, File “D:\Python38\lib\runpy.py”, line 87, in run_code exec(code, run_globals) File "D:\Python38\envs\ploomber_dev\Scripts\ploomber.exe_main.py", line 7, in <module> File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\cli\cli.py”, line 173, in cmd_router fn() File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\cli\build.py”, line 28, in main report = dag.build(force=args.force) File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\dag\DAG.py”, line 376, in build report = self._build(force, show_progress) File “d:\python38\envs\ploomber_dev\lib\site-packages\ploomber\dag\DAG.py”, line 470, in _build raise DAGBuildError( ploomber.exceptions.DAGBuildError: Failed to build DAG DAG(“No name”)

In my outputs folder, it contains: data.csv and data.ipynb

Windows 10 machine, Python 3.8, ploomber version 0.8.2

Rest of my pip freeze output: alabaster==0.7.12
ansiwrap==0.8.4
appdirs==1.4.4
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
autopep8==1.5.4
Babel==2.8.0
backcall==0.2.0
black==20.8b1
bleach==3.2.1
certifi==2020.11.8
cffi==1.14.3
chardet==3.0.4
click==7.1.2
colorama==0.4.4
decorator==4.4.2
defusedxml==0.6.0
docutils==0.16
entrypoints==0.3
humanize==3.1.0
idna==2.10
imagesize==1.2.0
ipykernel==5.3.4
ipython==7.19.0
ipython-genutils==0.2.0
jedi==0.17.2
Jinja2==2.11.2
jsonschema==3.2.0
jupyter-client==6.1.7
jupyter-core==4.6.3
jupyterlab-pygments==0.1.2
jupytext==1.6.0
markdown-it-py==0.5.6
MarkupSafe==1.1.1
mistune==0.8.4
mypy-extensions==0.4.3
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
nest-asyncio==1.4.2
networkx==2.5
notebook==6.1.5
numpy==1.19.4
numpydoc==1.1.0
packaging==20.4
pandas==1.1.4
pandocfilters==1.4.3
papermill==2.2.2
parso==0.8.0
pathspec==0.8.1
pickleshare==0.7.5
ploomber==0.8.2
prometheus-client==0.8.0
prompt-toolkit==3.0.8
pyarrow==2.0.0
pycodestyle==2.6.0
pycparser==2.20
pyflakes==2.2.0
Pygments==2.7.2
pygraphviz==1.6.0 pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2020.4
pywin32==228
pywinpty==0.5.7
PyYAML==5.3.1
pyzmq==19.0.2
regex==2020.10.28
requests==2.24.0
Send2Trash==1.5.0
six==1.15.0
snowballstemmer==2.0.0
Sphinx==3.3.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.20
sqlparse==0.4.1
tabulate==0.8.7
tenacity==6.2.0
terminado==0.9.1
testpath==0.4.4
textwrap3==0.9.2
toml==0.10.2
tornado==6.1
tqdm==4.51.0
traitlets==5.0.5
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.25.11
wcwidth==0.2.5
webencodings==0.5.1

[edublancas]

Fantastic! I’ll write a few tests, then merge to master, in the meantime, you can install from the ci-win branch. I’ll make this part of the next stable release, which should come in 1-2 weeks. I’m keeping this issue open until I merge the fix to the master branch.

Regarding the .source files, that’s expected. Those are files that ploomber uses to keep track of source code changes (they’re just JSON files), if you run ploomber build again, it won’t execute anything if you haven’t changed the task’s source code.

Thanks again for reporting this. Feel free to open more issues if you encounter any problems!

[edublancas]

Closing this since changes have been merged to master. I also configured CI to run on Windows and fixed a few bugs I found along the way.