Dynamic configuration / environment variables / etc with book builds

Describe the problem/need and solution

Currently we use a static configuration file (_config.yml) for all of the book’s configuration. However, there are some cases where you want to dynamically choose configuration at build time. For example, “set a configuration value based on an environment variable.”

This isn’t currently possible with static configuration, but it is possible in Sphinx. We could find some way to allow a user to dynamically update their configuration (or run arbitrary Python code) at build time.

Guide for implementation

Current build process

Here’s where we invoke Sphinx:

https://github.com/executablebooks/jupyter-book/blob/aedee257645ee41906c4d64f66f71b7f0dc7acfa/jupyter_book/cli/main.py#L307-L321

In that case, we explicitly set noconfig=True, which means that Sphinx does not expect any conf.py file to exist.

We then generate a dictionary of Sphinx config, and pass it to the Sphinx build command as “overrides”:

https://github.com/executablebooks/jupyter-book/blob/aedee257645ee41906c4d64f66f71b7f0dc7acfa/jupyter_book/sphinx.py#L114-L129

We also already have the ability to generate a conf.py file from a _config.yml file:

https://github.com/executablebooks/jupyter-book/blob/aedee257645ee41906c4d64f66f71b7f0dc7acfa/jupyter_book/cli/main.py#L458

Three ideas for implementation

There a few ways we could add this functionality:

1. Support conf.py. We could allow users to add a conf.py (maybe we’d call it _config.py?) that we’d point to during the Sphinx build. This would behave no differently from how Sphinx currently handles it.
2. Generate a conf.py at build time, and add a extraConfig block. Instead of using configuration over-rides, we could generate a temporary conf.py file that was created via the function above. We could then support a configuration block that would contain arbitrary Python code to be run, and that could over-ride / set configuration values (by being added to the end of the conf.py file. This is similar to how JupyterHub uses extraConfig.
3. Pre-process config.yml with jinja. We could also add a pre-processing step before we parse the config.yml file. This would let users to something like ansible style variable injection.

Suggestion

After some discussion below, it seems like path 3 above has the most support for adding this functionality. Especially if we followed patterns that were already common in other frameworks, it would be a way to provide some dynamic configuration without supporting the total flexibility of a conf.py file.

Tasks and updates

No response

[fperez]

Agreed @psychemedia - the jupyterlab-myst extension is already great progress in this direction, big kudos to @agoose77 for it 😃 And I agree fully with the idea that Jupyter isn’t just a data science tool! While its use for data science is very important, to me a key element of the project is helping people tell the stories that come out of these data. Computing is obviously at the center of it all (we’re not a word-processing project 😃, but in many contexts, what matters is the content that comes out of the process of computing/data analysis…

This is an example of students doing data analysis work to ultimately share a complete story about air quality in the Bay Area. The repo is executable on binder and it has a JBook-powered website that has the main “narrative” with their conclusions, supported by the detailed analysis notebooks and code. We want many more such experiences to be very smooth, in particular with

• minimal gaps between the live, interactive experience of using the system and the rendered result (in this sense, juptyerlab-myst is a big step forward).
• a very smooth workflow between using the live system and sharing version of that for others (live website, binder, etc). There, I know that @rowanc1 has already been making progress with some of the faster/smoother JS tools his team has.

[psychemedia]

I don’t recall any particular issues adding a simple custom admonition for Jupyter Book, it was just a very simple py package/extension. The issues I have a more to do with being able to provide a consistent user experience in terms of notebook content stye across Jupyter Book and and JupyterLab notebook/RetroLab UIs. I’m all for opening up ways of allowing have-a-go end-user developers to tune things, but not at the risk of complicating the new-user experience. When I look in a repo I want to crib from, the more config and settings files there are, the less likely I am to be able to make sense of them or know where to look to see how a particular effect was achieved.

I think I’m probably more with @chrisjsewell on this in terms of seeing Jupyter Book as a distribution optimised for use with MyST source documents and a set of extensions that work together to render a particular flavour of interactive book using a relatively streamlined setup procedure.

Providing tools that can “export to a sphinx config” seems sensible. That could be used to bootstrap someone’s own Sphinx build environment, would support innovation in that more general context, and that innovation might then feed back feature suggestions to Jupyter Book based on proven ideas in the wider context.