Logging etiquette

The current default logging behaviour by pyam is rude, if you actually want to configure logging.

Usually, one can expect a zen-like experience by just invoking

import logging
logging.basicConfig(level="ERROR")

But pyam’s initialization in https://github.com/IAMconsortium/pyam/blob/f071e790de7fdc3ab6a1135e4b31be7f395cff15/pyam/__init__.py#L14 and https://github.com/IAMconsortium/pyam/blob/f071e790de7fdc3ab6a1135e4b31be7f395cff15/pyam/__init__.py#L21-L26 does not respect these settings.

With those, one has to additionally issue

pyam.logger.setLevel(level="ERROR")

Annoyingly,

logging.getLogger().setLevel("ERROR")

will still show INFO messages coming from pyam, since they are already written out into stderr by the pyam logger.

That’s why the Configuring logging for a library section in the logging tutorial has this note:

It is strongly advised that you do not add any handlers other than NullHandler to your library’s loggers.

I see two alternatives for improvement:

Best practice

Move the initialization into a separate routine setup_logger and tell everyone to use

import pyam
pyam.setup_logging()

(setup_logging should take keyword arguments like level)

If they forget about it, it’s not that problematic either as long as we get in tandem rid of the NullHandler, because since python 3.2 the default behaviour of the logging library for a LogRecord which bubbles through the logging tree without encountering a Handler is to send the message to stderr, if it is at least a warning (without timestamp and loglevel), as described in What happens if no configuration is provided.

Looks like so:

Least change (but still slightly rude)

Get the root logger in __init__.py and check whether the user has set up any handlers before, as like:

rootLogger = logging.getLogger()
if not rootLogger.hasHandlers():
    # add handlers to rootLogger

That way, the user experience for the average pyam user is unchanged, but one can configure logging, by using

logging.basicConfig(level="ERROR")

as long as one does it before import pyam.

And it is possible to make it shut up with

logging.getLogger().setLevel("ERROR")

even after the import is through.

Thoughts? @danielhuppmann @gidden @znicholls . I do strongly favor the Best practice option but would prepare a PR for either!

[gidden]

Hi folks, thought it could be useful to chime in here. First, love the discussion =)

I think it’s important to highlight that there are largely three ‘user types’ for pyam:

1. analysis users (work entirely in notebook environments, coming to pyam from the IPCC notebooks among other places)
2. power users (using pyam in scripting environments)
3. developers (us, actively updating pyam code)

Notebooks are a first-class environment for pyam where we want to support fairly verbose logging statements to help guide these users, hence the original (rude) design of high-jacking the logging.

However, it is fundamentally important to also provide first-class support for the other two user categories, and @coroa has provided a nice example of where we have failed to do so.

Therefore, I would support a solution that raises our log level sufficiently if in a notebook environment, which I think @coroa has also provided. What do others think?

[danielhuppmann]

Thanks a lot, I agree that 2a seems like a really nice solution and striking a great balance between our different user groups!