[autoscaler] Ray submit with docker doesn't rsync the script to the container
See original GitHub issueSystem information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian GNU/Linux 9.8 (stretch) on GCP
- Ray installed from (source or binary): source (5bfcfa8)
- Ray version: master - 5bfcfa8 (0.7.0.dev0)
- Python version: 3.7
- Exact command to reproduce:
ray up -y minimal.yaml # with a docker image of the master build of Ray
ray submit --docker minimal.yaml hello_ray.py
Describe the problem
If I run submit
without --docker
then the worker can execute the specified script from ~
, but when --docker
is set, it throws:
python: can't open file '/root/hello_ray.py': [Errno 2] No such file or directory
When the docker run
is called in the initialization phase, there doesn’t seem to be any attached volumes, so I guess a shared location between the host and the container should resolve this, but there could be better ways as well.
Source code / logs
I can include full logs if necessary.
Issue Analytics
- State:
- Created 5 years ago
- Comments:6 (4 by maintainers)
Top Results From Across the Web
[Tune] Error when using docker containers and Sync - Ray
When using ray up to create an AWS cluster I'm having some trouble getting my log files to sync from the workers node(s)...
Read more >Unable to Connect to Ray Cluster from machines other than ...
I see that /tmp/cluster-default.state exists on my ray nodes(inside docker container) as well as these machines from which I am connecting to ...
Read more >Image Layer Details - anyscale/ray-ml:1.13.0-py38 - Docker Hub
1. ADD file ... in / ; 2. CMD ["bash"] ; 3. ENV NVARCH=x86_64 ; 4. ENV NVIDIA_REQUIRE_CUDA=cuda>=11.2 brand=tesla,driver>=418,driver<419 brand= ...
Read more >Run Reinforcement Learning on Ray Cluster (Microsoft Azure)
When you run training on the cluster, the workload will be submitted in multiple machines and you cannot then use and configure real...
Read more >ray Changelog - pyup.io
This includes jobs submitted via the job submission API and jobs launched from python scripts via ray.init(). * New Dashboard UI now shows...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
So what ended up being the fix for this?
I am also looking for the fix to this.