question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[autoscaler] Ray submit with docker doesn't rsync the script to the container

See original GitHub issue

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian GNU/Linux 9.8 (stretch) on GCP
  • Ray installed from (source or binary): source (5bfcfa8)
  • Ray version: master - 5bfcfa8 (0.7.0.dev0)
  • Python version: 3.7
  • Exact command to reproduce:
ray up -y minimal.yaml  # with a docker image of the master build of Ray
ray submit --docker minimal.yaml hello_ray.py

Describe the problem

If I run submit without --docker then the worker can execute the specified script from ~, but when --docker is set, it throws:

python: can't open file '/root/hello_ray.py': [Errno 2] No such file or directory

When the docker run is called in the initialization phase, there doesn’t seem to be any attached volumes, so I guess a shared location between the host and the container should resolve this, but there could be better ways as well.

Source code / logs

I can include full logs if necessary.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:6 (4 by maintainers)

github_iconTop GitHub Comments

4reactions
1Reiniercommented, Oct 22, 2019

So what ended up being the fix for this?

0reactions
preeth1commented, Mar 4, 2020

I am also looking for the fix to this.

Read more comments on GitHub >

github_iconTop Results From Across the Web

[Tune] Error when using docker containers and Sync - Ray
When using ray up to create an AWS cluster I'm having some trouble getting my log files to sync from the workers node(s)...
Read more >
Unable to Connect to Ray Cluster from machines other than ...
I see that /tmp/cluster-default.state exists on my ray nodes(inside docker container) as well as these machines from which I am connecting to ...
Read more >
Image Layer Details - anyscale/ray-ml:1.13.0-py38 - Docker Hub
1. ADD file ... in / ; 2. CMD ["bash"] ; 3. ENV NVARCH=x86_64 ; 4. ENV NVIDIA_REQUIRE_CUDA=cuda>=11.2 brand=tesla,driver>=418,driver<419 brand= ...
Read more >
Run Reinforcement Learning on Ray Cluster (Microsoft Azure)
When you run training on the cluster, the workload will be submitted in multiple machines and you cannot then use and configure real...
Read more >
ray Changelog - pyup.io
This includes jobs submitted via the job submission API and jobs launched from python scripts via ray.init(). * New Dashboard UI now shows...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found