Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Actor cannot be restored after killing a node.

See original GitHub issue

How to reprod:

Start a Ray cluster with one node(called nodeA).
Run an actor(called actorA, with checkpointable) in the cluster.
Connect a new node(called nodeB) to the cluster.
Kill nodeA.

Then the actorA cannot be restored on the nodeB since there are no state info of actorA in the actor_registry_ of nodeB’s NodeManager.

A feasible scenario is to load all actor info from GCS into actor_registry_ when a node connecting to the cluster.

Issue Analytics

State:
Created 4 years ago
Reactions:3
Comments:12 (12 by maintainers)

Top GitHub Comments

1reaction

stephanie-wangcommented, Jun 11, 2019

One way to fix this would be to have nodeB look up actorA in the GCS if it cannot find an entry in its local actor_registry_. We do this already for a similar scenario, where nodeB wants to submit a task to actorA but doesn’t have its location yet (code).

0reactions

jovany-wangcommented, Mar 11, 2020

This is already fixed.

Top Results From Across the Web

Crash with message node --- has been marked dead because ...

Invalid return value : likely worker died or was killed while executing the task; check previous logs or dmesg for errors. Source code...

Ray Core API — Ray 2.2.0 - the Ray documentation

If you want to kill actors immediately, you can also call ray.kill(actor) . Tip. Avoid repeatedly passing in large arguments to remote task...

Respawning Actor on Player Death - Unreal Engine Forums

After the player dies, I can't seem to figure out how to respawn the actor back into the game. For example after collecting...

Cannot kill Node JS Process - Stack Overflow

I run the command: ps aux | grep node when I try to kill that process, it says – No such process. I...

Errors | Node.js v19.3.0 Documentation

emit('error', new Error('This will crash')); });. Errors generated in this way cannot be intercepted using try…catch as they are thrown after the calling...