question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Release][Windows] Dashboard agent broken in 1.2.0 release candidate

See original GitHub issue

What is the problem?

In the 1.2.0 release candidate, the dashboard agent seems to be completely broken and spams the console from the background. This infinite loop of errors makes Ray pretty much unusable.

(p37) C:\Users\Administrator>python
Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> ray.init()
2021-02-10 03:05:34,529 INFO services.py:1174 -- View the Ray dashboard at http://127.0.0.1:8265
{'node_ip_address': '172.31.11.8', 'raylet_ip_address': '172.31.11.8', 'redis_address': '172.31.11.8:6379', 'object_store_address': 'tcp://127.0.0.1:58119', 'raylet_socket_name': 'tcp://127.0.0.1:56755', 'webui_url': '127.0.0.1:8265', 'session_dir': 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\2\\ray\\session_2021-02-10_03-05-33_888831_4556', 'metrics_export_port': 54734, 'node_id': 'dd5091720488d7393b326ddfac13adab6bfb97e048484f5ed87d08d8'}
>>> 2021-02-10 03:05:45,544     WARNING worker.py:1107 -- The agent on node EC2AMAZ-6T8IHN5 failed with the following error:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 312, in <module>
    loop.run_until_complete(agent.run())
  File "C:\ProgramData\Anaconda3\envs\p37\lib\asyncio\base_events.py", line 587, in run_until_complete
    return future.result()
  File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 188, in run
    await asyncio.gather(check_parent_task,
UnboundLocalError: local variable 'check_parent_task' referenced before assignment

(pid=None) Traceback (most recent call last):
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 323, in <module>
(pid=None)     raise e
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 312, in <module>
(pid=None)     loop.run_until_complete(agent.run())
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\asyncio\base_events.py", line 587, in run_until_complete
(pid=None)     return future.result()
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 188, in run
(pid=None)     await asyncio.gather(check_parent_task,
(pid=None) UnboundLocalError: local variable 'check_parent_task' referenced before assignment
(pid=None) --- Logging error ---
(pid=None) Traceback (most recent call last):
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\logging\handlers.py", line 69, in emit
(pid=None)     if self.shouldRollover(record):
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\logging\handlers.py", line 183, in shouldRollover
(pid=None)     self.stream = self._open()
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\logging\__init__.py", line 1116, in _open
(pid=None)     return open(self.baseFilename, self.mode, encoding=self.encoding)
(pid=None) NameError: name 'open' is not defined
(pid=None) Call stack:
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\aiohttp\client.py", line 337, in __del__
(pid=None)     self._loop.call_exception_handler(context)
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\asyncio\base_events.py", line 1645, in call_exception_handler
(pid=None)     self.default_exception_handler(context)
(pid=None)   File "C:\ProgramData\Anaconda3\envs\p37\lib\asyncio\base_events.py", line 1619, in default_exception_handler
(pid=None)     logger.error('\n'.join(log_lines), exc_info=exc_info)
(pid=None) Message: 'Unclosed client session\nclient_session: <aiohttp.client.ClientSession object at 0x0000020B6436E488>'
(pid=None) Arguments: ()
2021-02-10 03:05:47,482 WARNING worker.py:1107 -- The agent on node EC2AMAZ-6T8IHN5 failed with the following error:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 312, in <module>
    loop.run_until_complete(agent.run())
  File "C:\ProgramData\Anaconda3\envs\p37\lib\asyncio\base_events.py", line 587, in run_until_complete
    return future.result()
  File "C:\ProgramData\Anaconda3\envs\p37\lib\site-packages\ray\new_dashboard/agent.py", line 188, in run
    await asyncio.gather(check_parent_task,
UnboundLocalError: local variable 'check_parent_task' referenced before assignment

Ray version and other system information (Python version, TensorFlow version, OS):

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

Use python 3.7 (for example using conda on the aws windows 2016 deep learning ami). Install the 1.2.0 release candidate with

pip install --index-url https://test.pypi.org/simple/ ray

run

import ray

ray.init()

If the code snippet cannot be run by itself, the issue will be closed with “needs-repro-script”.

  • I have verified my script runs in a clean environment and reproduces the issue.
  • I have verified the issue also occurs with the latest wheels.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:13 (10 by maintainers)

github_iconTop GitHub Comments

1reaction
pcmoritzcommented, Oct 23, 2021
1reaction
wuisawesomecommented, Jul 29, 2021

Unfortunately it just doesn’t work on windows. We explicitely disabled it because it to contain the damage, since those errors made the rest of ray unusuable because it was spamming its errors.

Contributions are certainly welcome from people with windows expertise 😉

Read more comments on GitHub >

github_iconTop Results From Across the Web

InLong v1.2.0
Use the links below to download the Apache InLong from one of our mirrors. 1.2.0 release​. Name, Version, Date, Downloads. InLong Source Code ......
Read more >
BP Rewrites 1.2.0 maintenance release - BuddyPress.org
Immediately available is BP Rewrites 1.2.0. This maintenance release fixes two bugs. For details on the changes, please read the 1.2.0 ...
Read more >
Hotfixes - AppDynamics Documentation
This page lists hotfixes in the 4.5.x release. When artifacts are updated, the version number and availability date are listed below.
Read more >
SolarWinds RMM: Mac Agent 3.0 Release Candidate (RC ...
With this update, we're bringing the functionality of the standalone Mac-MSP interface into the RMM dashboard, as we continue to ensure we offer ......
Read more >
1.0.0-rc.6 release notes - Announcements - Drone
When will 1.0.0-final be released? · Agent Configuration. If you are using Drone with agents you need to pass DRONE_AGENTS_ENABLED=true to the server....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found