Log Monitor TypeError

No reproducible script (never seen this before), but this happened in the middle of my training:

2020-12-02 07:26:37,751	WARNING worker.py:1011 -- The log monitor on node ip-172-31-18-179 failed with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 354, in <module>
    log_monitor.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 275, in run
    self.open_closed_files()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 164, in open_closed_files
    self.close_all_files()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 102, in close_all_files
    os.kill(file_info.worker_pid, 0)
TypeError: an integer is required (got type str)

cc @rkooo567

[FarzanT]

@rkooo567 Great, so the only thing required is to add "autoscaler" here?

if (file_info.worker_pid != "raylet"
        and file_info.worker_pid != "gcs_server" and file_info.worker_pid != "autoscaler"):
    os.kill(file_info.worker_pid, 0)

I can submit a pull request, I would like to practice!

[wuisawesome]

Does anyone have a repro? If this was happening in december, then “autoscaler” was probably not the string breaking this.

Alternatively we could just do

try:
    os.kill(int(file_info.worker_pid), 0)
catch ...