How to clean/tests notebooks?

This was initially a question by @lwasser on Twitter: Is anyone using a tool that “cleans” or tests @ProjectJupyter notebooks for things like extra imports, PEP 8, etc etc?

It would be interesting to describe what we can do with jupytext at the command line, and also, what can be done with pre-commit hooks.

Regarding the command line, Jupytext can pipe the Python representation of a notebook to another program, and either test it (e.g. flake8) or pipe it (e.g. black)

jupytext notebook.ipynb --check flake8 # test the notebook with flake8
jupytext notebook.ipynb --pipe black  # apply black to the notebook

That works wells for the programs that read the standard input. For the others, one can use {} for a placeholder for a temporary file name. A good reference on what is know to work is this test file:

https://github.com/mwouts/jupytext/blob/master/tests/test_black.py

I also think that one can use the pre-commit package to do this on every commit. I remember chatting about this with @phaustin at #142, where he mentioned that he was using Jupytext (through a custom Python script), together with black / reorder_python_imports and flake8.

Maybe we could write a short blog post (or a bash notebook with examples…) to cover this?

[mwouts]

Hello @lwasser , thanks for your input.

• Regarding Markdown cells being mapped to long python comments that are not pep8 compliant, actually I do have the same issue… typically when we write a Markdown cell in Jupyter, we don’t insert line breaks. The Markdown renderer will wrap the text automatically, but in the internal representation there is no wrapping. Until now I addressed that issue by manually wrapping the comments in the py representation… I can’t find a tool that would be able to wrap single line comments in Python, do you think it would be worth adding an option in Jupytext to do that?
• I have noted your comment on the output of the flake8 command being printed in binary mode (i.e. without the line breaks), I can reproduce that and I will soon contribute a fix for this.

[mwouts]

Now we can do this with the pre-commit hook. The documentation is not very verbose at the moment (we have plans to improve this), but we do have (tested) examples at e.g. https://github.com/mwouts/jupytext/blob/main/tests/test_pre_commit_5_reformat_markdown.py