Should be able to turn off logging

🚀 Feature request

When doing a simple pipeline, I want to supress:

Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 230/230 [00:00<00:00, 136kB/s]
convert squad examples to features: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 241.08it/s]
add example index and unique id: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7037.42it/s]

Motivation

Your contribution

My code is pretty straightforward:

    args = parse_args()
    f = open(args.text_path, "r")
    context = f.read()
    # print(context)

    tokenizer = AutoTokenizer.from_pretrained(args.model)
    model = AutoModelForQuestionAnswering.from_pretrained(args.model)

    qa = pipeline('question-answering',
                  model='distilbert-base-uncased-distilled-squad', tokenizer='bert-base-cased')
    response = qa(context=context,
                  question=args.question)

    print(response['answer'])

[stas00]

Here you go:

# To control logging level for various modules used in the application:
import logging
import re
def set_global_logging_level(level=logging.ERROR, prefices=[""]):
    """
    Override logging levels of different modules based on their name as a prefix.
    It needs to be invoked after the modules have been loaded so that their loggers have been initialized.

    Args:
        - level: desired level. e.g. logging.INFO. Optional. Default is logging.ERROR
        - prefices: list of one or more str prefices to match (e.g. ["transformers", "torch"]). Optional.
          Default is `[""]` to match all active loggers.
          The match is a case-sensitive `module_name.startswith(prefix)`
    """
    prefix_re = re.compile(fr'^(?:{ "|".join(prefices) })')
    for name in logging.root.manager.loggerDict:
        if re.match(prefix_re, name):
            logging.getLogger(name).setLevel(level)

Usage:

1. override all module-specific loggers to a desired level (except whatever got logged during modules importing)

import everything, you, need
import logging
set_global_logging_level(logging.ERROR)

1. In case of transformers you most likely need to call it as:

import transformers, torch, ...
import logging
set_global_logging_level(logging.ERROR, ["transformers", "nlp", "torch", "tensorflow", "tensorboard", "wandb"])

add/remove modules as needed.

To disable logging globally - place at the beginning of the script

import logging
logging.disable(logging.INFO) # disable INFO and DEBUG logging everywhere
# or 
# logging.disable(logging.WARNING) # disable WARNING, INFO and DEBUG logging everywhere

If desired, set_global_logging_level could be expanded to be a scope manager too.

[shamoons]

Wish this would stay open.