[Core] [Bug] local_mode=True causes unexpected print statements of format :task_name:<function_name>:

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core, Monitoring & Debugging

What happened + What you expected to happen

In local_mode, working with modin.pandas.DataFrames will result in unexpected statements to stdout/stderr. These statements look like this:

:task_name:deploy_ray_func
:task_name:apply_func

The expected behavior is, that they shouldn’t show up.

This only happens in local_mode and only since ray version 1.7.0. In version 1.6.0, this undesired behavior wasn’t there. And it still exists in 1.9.1, the latest version of ray.

Versions / Dependencies

Linux 5.11.0-43-generic python 3.8.10 pip packages:

• modin==0.12.0
• ray==1.7.0 or higher

Reproduction script

import logging import modin.pandas as pd import ray

ray.init(local_mode=True, logging_level=logging.CRITICAL)

df = pd.DataFrame({“a”: [1, 2, 3, 4, 5], “b”: [1, 2, 1, 1, 2]}) df[df.duplicated([“b”])]

Anything else

This happens everywhere, where modin dataframes are changed but only in local_mode.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[mattip]

The print statements were added in #18105. There is a comment there by @ericl that the output does not show up on the console. But even when using local_mode=False I get output to the terminal. I wonder what changed?

>>> import ray
>>> ray.init(local_mode=False)
2022-05-31 10:21:40,211	INFO services.py:1477 -- View the Ray dashboard at http://127.0.0.1:8265
RayContext(dashboard_url='127.0.0.1:8265', python_version='3.9.9', ray_version='3.0.0.dev0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '10.0.0.19', 'raylet_ip_address': '10.0.0.19', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-05-31_10-21-38_546970_376304/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-05-31_10-21-38_546970_376304/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-05-31_10-21-38_546970_376304', 'metrics_export_port': 53623, 'gcs_address': '10.0.0.19:63800', 'address': '10.0.0.19:63800', 'node_id': 'ef165a80fc1c98e739e8294112b4db385bb0a314ccd6a810f8e9cae2'})
>>> import modin.pandas as pd
>>> df = pd.DataFrame({"a": [1, 2, 3, 4, 5], "b": [1, 2, 1, 1, 2]})
UserWarning: Distributing <class 'dict'> object. This may take some time.
>>> df[df.duplicated(["b"])]
UserWarning: User-defined function verification is still under development in Modin. The function provided is not verified.
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func
   a  b
2  3  1
3  4  1
4  5  2
>>> (pid=376452) :task_name:deploy_ray_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:deploy_ray_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func
(pid=376452) :task_name:apply_func

[ShadenSmith]

Observed the same issue. Initializing ray with local_mode=True fills our logs with thousands of those printed statements regardless of logging level.

It looks like the logging could be coming from here in _raylet.pyx ?

https://github.com/ray-project/ray/blob/149d06442bd0e53010ead72e4a7c0620ee1c0966/python/ray/_raylet.pyx#L542 https://github.com/ray-project/ray/blob/149d06442bd0e53010ead72e4a7c0620ee1c0966/python/ray/_raylet.pyx#L573 https://github.com/ray-project/ray/blob/149d06442bd0e53010ead72e4a7c0620ee1c0966/python/ray/_raylet.pyx#L712