Exit with non-zero status if notebooks encounter an unexpected exception.
See original GitHub issueSummary of the problem
Our project is switching to jupyter-book for documentation. Our documentation is built as part of our CI process using github actions. One challenge we’re encountering is ensuring that changes to our project code don’t break examples in our documentation.
Since the notebooks used for documentation are executed as part of our CI process now, it seems natural that any unexpected errors in our notebook should trigger a failure on our CI system. However, this is not the case.
When the notebooks are executed jupyter-book will report a successful build of documentation even if unexpected exceptions were raised. (in sphinx.py, the build process returns app.statuscode
which doesn’t take into account whether any exceptions were raised in the notebook).
The solution we’d like
During the build process, myst-nb keeps track of which notebooks are successfully built, and which ones failed, as described here: https://myst-nb.readthedocs.io/en/latest/use/execute.html#execution-statistics
We propose allowing a new flag (perhaps -f
or --fail-on-exception
) that would cause the build process to return a non-zero value (perhaps the number of failed notebooks) in the event of an unexpected exception.
Notebooks meant to demonstrate exceptions could still allow them to be raised without issue by tagging their cells with raises-exception
. The machinery for this is already in place and works.
This system would allow our documentation to “test itself” during the build process. At least on github actions, the log files which further describe the exceptions encountered can be saved as build artifacts to provide more information about what’s going wrong in the notebooks.
Alternatives we’ve considered.
We had a test system in place that tests notebooks and looks for errors. This works but is really inefficient. Our notebooks are evaluated twice during the build process and it basically doubles the job time on the CI system.
We’d considered other third-party notebook testing packages like testbook, but we didn’t have a great experience with them.
Having the jupyter-book build process serve as its own testing process under CI seems like a reasonable solution.
Additional context
I have a branch that demonstrates the functionality here: https://github.com/robfalck/jupyter-book/tree/fail_on_exception
As a demonstration, I created a new JupyterBook and made four copies of notebooks.ipynb. All of these notebooks included a new cell with
assert 1 == 2
Two of these notebooks tag that cell with raises-exception
, and two do not.
Executing with the command
jupyter-book build -f .
Results in the following warnings during the build:
WARNING: Execution Failed with traceback saved in /Users/rfalck/Projects/test_error_book/_build/html/reports/notebooks2.log
WARNING: Execution Failed with traceback saved in /Users/rfalck/Projects/test_error_book/_build/html/reports/notebooks2.log
and the build ends with
The following notebooks encountered unexpected exceptions.
Add the tag 'raises-exception' to the offending cells to ignore these exceptions.
notebooks2
subfolder/notebooks2
===============================================================================
Building your book, returns a non-zero exit code (2). Look above for the cause.
===============================================================================
This demonstrates that the change allows exceptions when the raises-exception
tag is present.
The docs will also completely build when the -f
switch is not present, so it doesn’t break current behavior.
I’m not sure what the correct process is here for submitting pull requests. I’d be happy to submit one with my branch, or make any changes that are wanted before doing so.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:2
- Comments:15 (6 by maintainers)
@chrisjsewell : Hey, that worked! There may be some hope yet for this path. I’d really like to get down to zero warnings, because the warnings are useful for finding some real problems like invalid links.
@chrisjsewell : This looks like it will be a complete success. I am down to just 55 errors that I know how to fix, so hopefully by the end of today, I should have this working. We’ll have to use the bleeding-edge jupyter-notebook repo until the latest makes it to pypi, but I think this means we won’t need the feature requested here, as we will ideally be building with zero warnings.
Thank you so much for the help you’ve given me.