question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Exit with non-zero status if notebooks encounter an unexpected exception.

See original GitHub issue

Summary of the problem

Our project is switching to jupyter-book for documentation. Our documentation is built as part of our CI process using github actions. One challenge we’re encountering is ensuring that changes to our project code don’t break examples in our documentation.

Since the notebooks used for documentation are executed as part of our CI process now, it seems natural that any unexpected errors in our notebook should trigger a failure on our CI system. However, this is not the case.

When the notebooks are executed jupyter-book will report a successful build of documentation even if unexpected exceptions were raised. (in sphinx.py, the build process returns app.statuscode which doesn’t take into account whether any exceptions were raised in the notebook).

The solution we’d like

During the build process, myst-nb keeps track of which notebooks are successfully built, and which ones failed, as described here: https://myst-nb.readthedocs.io/en/latest/use/execute.html#execution-statistics

We propose allowing a new flag (perhaps -f or --fail-on-exception) that would cause the build process to return a non-zero value (perhaps the number of failed notebooks) in the event of an unexpected exception.

Notebooks meant to demonstrate exceptions could still allow them to be raised without issue by tagging their cells with raises-exception. The machinery for this is already in place and works.

This system would allow our documentation to “test itself” during the build process. At least on github actions, the log files which further describe the exceptions encountered can be saved as build artifacts to provide more information about what’s going wrong in the notebooks.

Alternatives we’ve considered.

We had a test system in place that tests notebooks and looks for errors. This works but is really inefficient. Our notebooks are evaluated twice during the build process and it basically doubles the job time on the CI system.

We’d considered other third-party notebook testing packages like testbook, but we didn’t have a great experience with them.

Having the jupyter-book build process serve as its own testing process under CI seems like a reasonable solution.

Additional context

I have a branch that demonstrates the functionality here: https://github.com/robfalck/jupyter-book/tree/fail_on_exception

As a demonstration, I created a new JupyterBook and made four copies of notebooks.ipynb. All of these notebooks included a new cell with

assert 1 == 2

Two of these notebooks tag that cell with raises-exception, and two do not.

Executing with the command

jupyter-book build -f .

Results in the following warnings during the build:

WARNING: Execution Failed with traceback saved in /Users/rfalck/Projects/test_error_book/_build/html/reports/notebooks2.log
WARNING: Execution Failed with traceback saved in /Users/rfalck/Projects/test_error_book/_build/html/reports/notebooks2.log

and the build ends with


The following notebooks encountered unexpected exceptions.
Add the tag 'raises-exception' to the offending cells to ignore these exceptions.

notebooks2
subfolder/notebooks2

===============================================================================

Building your book, returns a non-zero exit code (2). Look above for the cause.

===============================================================================

This demonstrates that the change allows exceptions when the raises-exception tag is present. The docs will also completely build when the -f switch is not present, so it doesn’t break current behavior.

I’m not sure what the correct process is here for submitting pull requests. I’d be happy to submit one with my branch, or make any changes that are wanted before doing so.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:2
  • Comments:15 (6 by maintainers)

github_iconTop GitHub Comments

2reactions
Kenneth-T-Moorecommented, May 10, 2021

@chrisjsewell : Hey, that worked! There may be some hope yet for this path. I’d really like to get down to zero warnings, because the warnings are useful for finding some real problems like invalid links.

1reaction
Kenneth-T-Moorecommented, May 11, 2021

@chrisjsewell : This looks like it will be a complete success. I am down to just 55 errors that I know how to fix, so hopefully by the end of today, I should have this working. We’ll have to use the bleeding-edge jupyter-notebook repo until the latest makes it to pypi, but I think this means we won’t need the feature requested here, as we will ideally be building with zero warnings.

Thank you so much for the help you’ve given me.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Can't access User Library in R - "Non-Zero Exit Status" warning
A non-zero exit status means in this case that the system failed to install the package. There seem to be a number of...
Read more >
The following SSH command responded with a non-zero exit ...
I'm still getting the error when trying to run "vagrant up" command in Mac OS. Error: ==> default: Mounting NFS shared folders... The...
Read more >
How to fix Python KeyError Exceptions in simple steps?
Know about Python KeyError Exception. And learn how to handle exceptions in Python. A detailed guide to Errors and Exceptions in Python.
Read more >
Error with workbench SceneGraphChart - Ansys Learning Forum
An unexpected error has occurred: The following required addins could not be loaded: Ans.SceneGraphChart.SceneGraphAddin. The software will exit.
Read more >
Exception and Error Handling in Python - DataCamp
Struggling with error types? Learn how to catch and handle exceptions in Python with our step-by-step tutorial. Raise exceptions in Python and catch...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found