Flow run with Ray and agent throws: `ImportError: cannot import name 'SubprocessFlowRunner' from partially initialized module 'prefect.flow_runners'`

Opened from the Prefect Public Slack Community

christian.vogel: Hi Prefect Community, I am currently receiving an error when using a local (on my pc) prefect agent which pulls from a work queue in the the prefect cloud. The error only occurs when I try to use the RayTaskRunner:

(begin_task_run pid=141324) ImportError: cannot import name 'SubprocessFlowRunner' from partially initialized module 'prefect.flow_runners' (most likely due to a circular import) (/home/christian/Documents/ray_and_prefect/env/lib/python3.9/site-packages/prefect/flow_runners/__init__.py)

I am using the following dependencies: prefect==2.0b7 prefect-ray==0.1.0 ray==1.13.0 Apparently I am doing something wrong with my dependencies or when I am importing them. Do you have any idea?

christian.vogel: This is my flow and deployment:

from prefect import task, flow
from prefect_ray import RayTaskRunner
from prefect.deployments import DeploymentSpec


@task
def say_hello(name):
    print(f"hello {name}")


@flow(name="temp-flow-example", task_runner=RayTaskRunner(address="auto"))
def greetings():
    say_hello("Ford")


DeploymentSpec(
    name="temp-flow-example",
    flow=greetings
)

christian.vogel: What confuses me a bit, is the fact, that the flow only fails when pulled from the cloud and seems to work when i run it directly locally via python and a main method.

christian.vogel: Could it be related to the fact that I am running my agent in a virtual environment? Probably in the background the following is executed:

from prefect.flow_runners import SubprocessFlowRunner

which them leads to some import issues because of the already imported:

from prefect_ray import RayTaskRunner

Is that possible?

anna: do you run your agent in the same virtual environment as you run your local process? perhaps you can explicitly define your flow runner with the virtual environment you use for your agent?

flow_runner=SubprocessFlowRunner(condaenv="yourcondavenv"),

anna: I was trying to replicate and sth also didn’t work for me, even though I got a different error message using the same versions as you did

anna: my error was:

Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-07_14-08-35_008202_6668/logs/dashboard.log'

anna: wow it took me forever to set this up but this helped https://docs.ray.io/en/master/ray-overview/installation.html#m1-mac-apple-silicon-support

looks like you need to be super careful about your environment with Ray and grpcio package - do you use Conda? on which machine do you run it?

christian.vogel: Tried specifying the SubprocessFlowRunner with the virtual env path in my case. but did not help

christian.vogel: I am running a venv on a linux ubuntu machine

anna: yup you’re right, I was able to reproduce your error, thanks for reporting that

<@ULVA73B9P> open “Flow run with Ray and agent throws: ImportError: cannot import name 'SubprocessFlowRunner' from partially initialized module 'prefect.flow_runners'”

Original thread can be found here.

[ahuang11]

We can close this issue since I think there already exists an issue here: https://github.com/PrefectHQ/prefect-ray/issues/33

I will close the prefect-ray issue once ray v2.0.1 is released on pypi

[jmrobbins13]

The ImportError was fixed by moving to the ray nightlies (ray-3.0.0.dev0 right now). This issue could probably be closed if it can be fixed from the Ray side, right?