Raylet Memory usage growing

System information

• OS Platform and Distribution: Ubuntu 16.04
• Ray installed from: binary
• Ray version: 0.6.5
• Python version:3.6.8
• Exact command to reproduce

import numpy as np
import ray
import time
ray.init(redis_max_memory=100000000)


@ray.remote
class Runner():
    def __init__(self, dataList):
        self.run(dataList)

    def run(self,dataList):
        while True:
            dataList.put.remote(np.ones(10))

@ray.remote
class Optimizer():
    def __init__(self, dataList):
        self.optimize(dataList)

    def optimize(self,dataList):
        while True:
            dataList.pop.remote()

@ray.remote
class DataServer():
    def __init__(self):
        self.dataList= []

    def put(self,data):
        self.dataList.append(data)

    def pop(self):
        if len(self.dataList) !=0:
            return self.dataList.pop()
    def get_size(self):
        return len(self.dataList)


dataServer = DataServer.remote()
runner = Runner.remote(dataServer)
optimizer1 = Optimizer.remote(dataServer)
optimizer2 = Optimizer.remote(dataServer)

while True:
    time.sleep(1)
    print(ray.get(dataServer.get_size.remote()))

Describe the problem

The memory usage is constantly increasing even though the list is cleared repeatedly until the program crashes. Does anybody know why this is happening ?

Source code / logs

Traceback (most recent call last): File “/home/test.py”, line 48, in <module>

File “/home/anaconda3/envs/py3/lib/python3.6/site-packages/ray/worker.py”, line 2310, in get value = worker.get_object([object_ids])[0] File “/home/anaconda3/envs/py3/lib/python3.6/site-packages/ray/worker.py”, line 531, in get_object self.current_task_id, File “python/ray/_raylet.pyx”, line 254, in ray._raylet.RayletClient.fetch_or_reconstruct File “python/ray/_raylet.pyx”, line 59, in ray._raylet.check_status Exception: [RayletClient] Connection closed unexpectedly.

Process finished with exit code 1

[ericl]

Thanks @KemalAcar07 , I can confirm the result. I see the raylet process also increasing in memory, though the other processes are stable in memory usage.

At first glance, this may be related to this: https://github.com/ray-project/ray/issues/4359, but will require more investigation. cc @robertnishihara

[simon-mo]

Should be fixed by direct task call and direct actor call re-architecture. please reopen if this still fails on latest master/release.