Notebook preamble injection

Is your feature request related to a problem? Please describe.

To keep my notebook pages mutually consistent, I have a fairly lengthy preamble cell that handles things like common imports, setting matplotlib styling and figure output format, defining helper functions, setting random seeds, and so on. Some of this code is necessary for ensuring the correctness and reproducibility of the notebook contents (eg it won’t work if you don’t import numpy as np), and some of it is purely cosmetic. (More on this below).

At this point, my preamble getting to be pretty long (>50 lines), and is definitely not the sort of thing that should be copy-pasted into each notebook page.

My solution for now is to factor all of the common preamble code into a dedicated setup.ipynb, and use

%run /path/to/setup.ipynb

magic to inject its contents to each page.

This generally works great for developing notebooks. However, this will obviously cause a huge problem if a reader tries to download the .ipynb file for a page (or launch on binderhub, colab, etc), where the preamble file may not exist.

Describe the solution you’d like

Would it be possible for jb to modify notebooks at compile-time? I’m specifically thinking of the following modifications:

1. Add a cell tag which would be stripped out by jb. A %run ../setup.ipynb cell could then be used for development, but stripped out at compile time and not appear on the rendered site.
2. Allow the author to specify a preamble notebook in _config.yml which is injected ahead of the first cell in the compiled notebook.

I imagine fancier schemes could also work (eg detecting and stripping specific magics, or monkeypatching them somehow), but the scheme I’m proposing seems a bit simpler and less likely to go horribly wrong. As far as I can tell, it would be sufficient to work around the download/execute problem I described above.

Describe alternatives you’ve considered

I guess I could go back to copy-pasting.

Additional context

As mentioned above, some of the preamble contents are necessary (and informative to a user), and some are purely cosmetic and could be hidden (eg collapsed) by default. It’d be great if it’s possible to do this kind of compile-time injection in a way that lets the author specify this, eg by cell tags for hidden/collapsed.

[bmcfee]

I guess in things like colab people often pip install at the top of the notebook, though that’s obviously sub-optimal too

Right. Even then, that would only solve things confined to python. Another use-case (though one that my %run hack does not seem to allow for, but maybe there’s a clever way to do it) is to have a shared preamble cell that sets up common tex macros for mathjax. I get that you’re supposed to do this in _config.yml, but that’s not too helpful for development because nothing renders inside jupyter as you’re writing.

[choldgraf]

Just a note that @matthew-brett had some links to other implementation of this feature in R, as well as some feedback, here: https://github.com/executablebooks/meta/discussions/172

Given how common this is in other tools like R, as well as the fact that many people have asked for it, I wonder if we could do something like:

• Support a notebook_preamble key that could:
  • map to a single value, which would be interpreted as a multi-line code string to be prepended to each notebook as a code cell before execution.
  • map to a dictionary of kernel: code pairs that would choose the code to add to each notebook based on its kernel.
• This code would be prepended to the notebooks before execution, and not included in the output pages (perhaps with a flag to disable this)
• The documentation would need to make clear that preambles will not be included in the downloaded notebooks, Binders, etc. It would be a “use at your own risk” feature