calling handler.close() from different thread causes deadlock when using AzureLogHandler
See original GitHub issueDescribe your environment. Python 3.8.13 on Linux, opencensus-ext-azure == 1.1.4, azureml-sdk[automl] ~= 1.42.0, azureml-mlflow==1.42.0, mlflow-skinny==1.26.0
Steps to reproduce. I can’t give an exact test code. The problem is related to #1112. We add an AzureLogHandler on the main thread. Then later we run automl Experiment in multiple threads.
The code in the thread at some point runs import mlflow
, and the program stucks.
The import causes to run _configure_mlflow_loggers(root_module_name=__name__)
, which then runs the logging.config.dictConfig
This runs _clearExistingHandlers in logging.config, and inside of this, the shutdown function runs. The shutdown runs the following calls for all defined logging handlers:
h.acquire()
h.flush()
h.close()
With previous versions, for example opencensus-ext-azure == 1.1.3 the program stuck at the h.flush() call. With the 1.1.4 version the h.flush() finishes, but then the run still stops when the self._worker.stop(timeout) is called inside the h.close().
What is the expected behavior? The import finishes successfully and the program continues
What is the actual behavior? The program stops in deadlock
Issue Analytics
- State:
- Created a year ago
- Comments:5 (2 by maintainers)
Yes, but more like adding default timeout instead.
@csadam Ideally, we would use the sentinel pattern as described here to avoid breaking changes, and then remove the custom atexit hook in place of logging shutdown.