[tune] AttributeError: 'NoneType' object has no attribute 'get_global_worker'

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Ray installed from (source or binary): binary
• Ray version: 0.8.0.dev1
• Python version: 3.7
• Exact command to reproduce:

import ray
from ray import tune

ray.init()


def train_func(config, reporter):  # add a reporter arg
    accuracy = config["momentum"] * 2 + 0.3
    reporter(mean_accuracy=accuracy)  # report metrics


all_trials = tune.run(
    train_func,
    name="quick-start",
    stop={"mean_accuracy": 99},
    config={"momentum": tune.grid_search([0.1, 0.2])})

Describe the problem

When I run the script above (based on the examples from here), I get the following error at the end of each run:

Exception ignored in: <function ActorHandle.__del__ at 0x7f3aeaffc730>
Traceback (most recent call last):
  File "/home/user/miniconda3/lib/python3.7/site-packages/ray/actor.py", line 613, in __del__
AttributeError: 'NoneType' object has no attribute 'get_global_worker'

However, this is not happening for 0.7.0.dev2, the process exists normally in that case.

This might be related to #4467 .

[ShamCondor]

Hm，I’m also removed tensorflow, so this issue go away now

[vakker]

@richardliaw Doing a bit of debugging it seems to be related to the Tensorboard logging. I used the Dockerfile:

FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04

RUN apt-get update && apt-get install -y --no-install-recommends wget

RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && \
  bash ./miniconda.sh -b -p /opt/conda
ENV PATH "/opt/conda/bin:$PATH"

RUN conda install -y -c conda-forge tensorflow==1.13.1 pandas
RUN pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev2-cp37-cp37m-manylinux1_x86_64.whl

WORKDIR /
COPY test.py .

with test.py (same as above):

import ray
from ray import tune

ray.init()


def train_func(config, reporter):  # add a reporter arg
    accuracy = config["momentum"] * 2 + 0.3
    reporter(mean_accuracy=accuracy)  # report metrics


all_trials = tune.run(
    train_func,
    name="quick-start",
    stop={"mean_accuracy": 99},
    config={"momentum": tune.grid_search([0.1, 0.2])})

Running python /test.py does give the error at the end, but running export RLLIB_TEST_NO_TF_IMPORT=1; python /test.py (that just disables the TB logging according to here) does not give the error.

Thanks