New trials do not run after old trials terminates.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• Ray installed from (source or binary): binary
• Ray version: 24bfe8ab76f743983aa494528c4ec8fe1e9d1a3e
• Python version: 3.6.5
• Exact command to reproduce: Try https://github.com/ray-project/ray/files/2593051/test_ray.txt.

For fast reproduction, you can change training_iteration from 500 to 20.

Either single-node or distributed mode has this issue. It seems that the requested resources of terminated trials are not released.

[richardliaw]

And ray.global_state.available_resources() which was helpful in finding these nested actors.

On Sun, Nov 25, 2018 at 12:18 AM Eric Liang [e-mail] wrote:

If you’re using the autoscaler resource stats are printed periodically to the monitor logs.

There’s also ray.global_state.cluster_resources()

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/3392#issuecomment-441423361, or mute the thread https://github.com/notifications/unsubscribe-auth/AEUc5fQN7b3lSsiT_ahG3r6jhWM2vkfzks5uylJvgaJpZM4YxaIB .

[ericl]

If you’re using the autoscaler resource stats are printed periodically to the monitor logs.

There’s also ray.global_state.cluster_resources()