Duplicated IDs are generated

What is the problem?

The string generated by GenerateUniqueBytes is not really unique. https://github.com/ray-project/ray/blob/0178d6318ead643fff6d9ed873c88f71fdac52e7/src/ray/common/id.cc#L40-L53

Ray version and other system information (Python version, TensorFlow version, OS): 1.0.1

Reproduction (REQUIRED)

Apply below patch and run bazel run //:id_test,

diff --git a/src/ray/common/id_test.cc b/src/ray/common/id_test.cc
index 926e6fbd11..f1274d92a5 100644
--- a/src/ray/common/id_test.cc
+++ b/src/ray/common/id_test.cc
@@ -51,6 +51,15 @@ TEST(ActorIDTest, TestActorID) {
     const ActorID actor_id = ActorID::Of(kDefaultJobId, kDefaultDriverTaskId, 1);
     ASSERT_EQ(kDefaultJobId, actor_id.JobId());
   }
+
+  {
+    // test no duplicated ID
+    std::unordered_set<ActorID> ids;
+    for (size_t i = 0; i < 1000000; i++) {
+        auto id = ActorID::Of(kDefaultJobId, kDefaultDriverTaskId, i);
+        RAY_CHECK(ids.insert(id).second) << "Duplicated ID generated: " << id;
+    }
+  }
 }

 TEST(TaskIDTest, TestTaskID) {

You will see the error:

[2020-11-20 05:37:07,442 C 104416 104416] id_test.cc:60:  Check failed: ids.insert(id).second Duplicated ID generated: 584a7e60c7000000

If we cannot run your script, we cannot fix your issue.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[rkooo567]

I see.

There are 2 comments here;

• When looking at code, it doesn’t look like you need to use an actor pool that recreates actors again and again? Why not just reusing it or use a task (it doesn’t seem to be stateful). Just a suggestion for a workaround.
• The error is about object ID. The object ID is not created randomly, but we create by incrementing the index for each Ray task (.remote method). This means each single remote task can create 32 bytes number of object ids without conflicts, so I don’t think this actor issue is related. But I’ve definitely seen the related issue; https://github.com/ray-project/ray/issues/9551

[Dadle]

We have been suffering from this issue for months. it is costly and we are working on external patch to automate restart to avoid this issue. The user experience is that Ray is unstable unfortunately.

We run evolutionary algorithm doing fitness calculations as ray actors. These actors are distributed across 100+ 2CPU EC2 isntance with 200+actors multiple times per second. We run this workload for days at a time. So for us to run 50 000 000+ actors on one cluster for one workload is very common.

Is there a known workaround to not create a new actor ID for each new actor?