GPU actors slower than Serial execution

Hi, I want to use Ray to parallelize a GPU job, by using 3 GPU actors, as follows.

However, the result does not look good. It is even much slower than a serial single-GPU execution.

import psutil
import ray
import time
import numpy
import random

import face_alignment
from skimage import io


num_cpus = psutil.cpu_count(logical=False)
ray.init(num_cpus=num_cpus, num_gpus=3)

@ray.remote(num_gpus=1)
class GPUActor(object):
    def __init__(self):
        self.fa = face_alignment.FaceAlignment(
                face_alignment.LandmarksType._2D,
                flip_input=False, device='cuda')

        str = "This function is allowed to use GPUs {}.".format(ray.get_gpu_ids())
        print(str)
        self.preds = 0

    def proc(self, img):
        self.preds += self.fa.get_landmarks(img)[0].mean()

    def get_preds(self):
        return self.preds


gpuactors = [GPUActor.remote() for _ in range(3)]

begin = time.time()
for _ in range(20 * 3):
    image = io.imread('aflw-test.jpg')
    img_id = ray.put(numpy.array(image))
    gpuactors[_ % 3].proc.remote(img_id)

res = ray.get([actor.get_preds.remote() for actor in gpuactors ])
print(res)
end = time.time()
print('time', end- begin)

[Arjunsankarlal]

I actually found the answer in my case, and that was actually a pretty simple fix! 😅 After I set torch.set_num_threads(num_cpus)in the init function I was actually able to get the speed boost I was expecting.

Thanks for the wonderful library!

[stale[bot]]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you’d still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray’s public slack channel.

Thanks again for opening the issue!