Doing import ray breaks my logging [Bug]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

I have implemented a separate class where i am using ray via modin to read a huge csv file.

When the class is called, it breaks the custom logging yielding

--- Logging error ---
Traceback (most recent call last):
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 434, in format
    return self._format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 430, in _format
    return self._fmt % record.__dict__
KeyError: 'project'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 666, in format
    s = self.formatMessage(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 635, in formatMessage
    return self._style.format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 436, in format
    raise ValueError('Formatting field not found in record: %s' % e)
ValueError: Formatting field not found in record: 'project'

I became suspicious about it and started commenting out the import section in my module and rerunning. Commenting out the import ray brought logging back.

Versions / Dependencies

python=3.9.7 ray==1.6.0 modin==0.11.3

Reproduction script

import os
import logging
import logging.config


def initialise_logger(config):
    """
    Args:
        section (str): the log_configuration section for the service. Defaults to "log_configuration".

    """

    logging.config.fileConfig(fname=config, disable_existing_loggers=False)

    old_factory = logging.getLogRecordFactory()

    def my_record_factory(*args, **kwargs):
        """
        Standardized method to expose LogRecordFactory so new parameters
        that are semi static can be injected into CBD log records.

        Returns:
            [LogRecord]: Modified log record
        """
        record = old_factory(*args, **kwargs)

        # project and any other keys that may be used in index creation in elasticsearch
        # must be lowercase ( in scenario where kafka logging is used )
        record.project = "alex"
        record.microservice = "alex"
        record.hostname = "alex"
        record.local_ip = "alex"

        # all Mesos containers in production have this environment variable.
        # if it is not present, then we are not in production
        record.mesos_task_id = "alex"

        return record

    logging.setLogRecordFactory(my_record_factory)


def main():
    text = """[loggers]
    keys=root,test_logger,Test_Class
    
    [handlers]
    keys=consoleHandler

    [formatters]
    keys=consoleFormatter
    
    [logger_root]
    level=INFO
    handlers=consoleHandler
    
    [logger_test_logger]
    level=INFO
    handlers=consoleHandler
    qualname=test_logger
    propagate=0
    
    [logger_Test_Class]
    level=INFO
    handlers=consoleHandler
    qualname=Test_Class
    propagate=0
    
    [handler_consoleHandler]
    class=StreamHandler
    level=INFO
    formatter=consoleFormatter
    args=(sys.stdout,)
    
    [formatter_consoleFormatter]
    format = %(asctime)s.%(msecs)03d %(project)s %(microservice)s %(levelname)s %(hostname)s %(local_ip)s %(name)s %(filename)s %(lineno)d %(funcName)s %(module)s %(processName)s %(process)d %(threadName)s %(thread)d  %(message)s
    datefmt= %Y-%m-%d %H:%M:%S   
    """

    with open(os.path.join(os.getcwd(), "config.ini"), "w") as conffile:
        conffile.write(text)

    initialise_logger(config="config.ini")

    logging.info("RAY NOT IMPORTED")

    import ray

    logging.info("RAY IMPORTED")


if __name__ == '__main__':
    main()

which yields

2022-02-11 10:36:07.564 alex alex INFO alex alex root test.py 85 main test MainProcess 2168 MainThread 140060783249216  RAY NOT IMPORTED
--- Logging error ---
Traceback (most recent call last):
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 434, in format
    return self._format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 430, in _format
    return self._fmt % record.__dict__
KeyError: 'project'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 666, in format
    s = self.formatMessage(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 635, in formatMessage
    return self._style.format(record)
  File "/home/vagrant/miniconda3/envs/cbd3/lib/python3.9/logging/__init__.py", line 436, in format
    raise ValueError('Formatting field not found in record: %s' % e)
ValueError: Formatting field not found in record: 'project'
Call stack:
  File "/vagrant/dist-repos/cbd/stab/test_cbd3/test.py", line 93, in <module>
    main()
  File "/vagrant/dist-repos/cbd/stab/test_cbd3/test.py", line 89, in main
    logging.info("RAY IMPORTED")
Message: 'RAY IMPORTED'
Arguments: ()

Process finished with exit code 0

Anything else

Every time

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[richardliaw]

This seems like a bad bug. It’s appearing in cli_logger.py. Can @cadedaniel look into this? The goal should be not to change the logger unless the CLI is used.

[rkooo567]

There’s an attempt to fix the PR ^, but I didn’t get enough bandwidth to get this in by 2.2. Will fix by 2.3