Rllib is configured to use just one CPU thread.

Observation When I install Flow as described here, I get a conda environment that cannot build Ray through the recommended python setup.py develop:

[ 65%] Linking CXX static library libraylet_client.a [ 65%] Built target raylet_client [ 65%] Built target ray_objlib Makefile:138: recipe for target ‘all’ failed make: *** [all] Error 2 Traceback (most recent call last): File “setup.py”, line 180, in <module> license=“Apache 2.0”) File “/root/anaconda2/envs/flow/lib/python3.5/site-packages/setuptools/init.py”, line 129, in setup return distutils.core.setup(**attrs) File “/root/anaconda2/envs/flow/lib/python3.5/distutils/core.py”, line 148, in setup dist.run_commands() File “/root/anaconda2/envs/flow/lib/python3.5/distutils/dist.py”, line 955, in run_commands self.run_command(cmd) File “/root/anaconda2/envs/flow/lib/python3.5/distutils/dist.py”, line 974, in run_command cmd_obj.run() File “/root/anaconda2/envs/flow/lib/python3.5/site-packages/setuptools/command/develop.py”, line 36, in run self.install_for_development() File “/root/anaconda2/envs/flow/lib/python3.5/site-packages/setuptools/command/develop.py”, line 136, in install_for_development self.run_command(‘build_ext’) File “/root/anaconda2/envs/flow/lib/python3.5/distutils/cmd.py”, line 313, in run_command self.distribution.run_command(command) File “/root/anaconda2/envs/flow/lib/python3.5/distutils/dist.py”, line 974, in run_command cmd_obj.run() File “setup.py”, line 79, in run subprocess.check_call([“…/build.sh”, “-p”, sys.executable]) File “/root/anaconda2/envs/flow/lib/python3.5/subprocess.py”, line 581, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command ‘[’…/build.sh’, ‘-p’, ‘/root/anaconda2/envs/flow/bin/python’]’ returned non-zero exit status 2

I have reproduced this error on two different computers (Ubuntu 16.04) and several lxd containers. When Ray fails to build through python setup.py develop, I am forced to install it from binaries pip install -U ray.

This works, but makes Ray and specifically Tune unable to use more than one CPU thread! To be exact, I run the same Rllib example from the same pip installation outside Flow environment, and it uses the specified number of CPUs in ray.init(). In contrast, the installation inside Flow environment runs the same example using only one CPU.

Question

1. What could be the reason behind python setup.py develop failing in Flow environment?
2. Are you able to reproduce this build error when following the installation guide?

Thank you, Farshad

[AboudyKreidieh]

Update: the new setup instructions (including changes to environment.yml are up). hope that helps!

[AboudyKreidieh]

Hi, we are transitioning to using pip to install ray, which I recommend you go for (specifically ray==0.6.1). A new update is in the works to support this throughout the repo, including updating the setup instructions.