[Core] support out of order execution for threaded/async actors

The following will timeout

import ray
ray.init()

@ray.remote
class A:
    def echo(self, inp):
        print(inp)
        return inp

@ray.remote
def never_return():
    import time
    time.sleep(10000)

inp_ref_1 = never_return.remote()
inp_ref_2 = ray.put(2)

a = A.options(max_concurrency=2).remote()

out_ref_1 = a.echo.remote(inp_ref_1)
out_ref_2 = a.echo.remote(inp_ref_2)

assert ray.get(out_ref_2, timeout=5) == 2 # currently timeout!

I went all the way back to Ray 0.8.0 and it is the same behavior, so this is not a regression. For concurrent actors and async actors, however, I think the snippet should work and we should allow later tasks to execute first when their dependencies ready. This is a semantics change.

Context: user bug report for Ray Serve https://discuss.ray.io/t/concurrent-queries-blocking-following-queries/3949/2

[ericl]

Sounds good. I looked at this a bit more and I think we can potentially just enable it by default for concurrent/async actors, since the ordering was never really guaranteed for these in the first place.

Unfortunately the implementation may be quite tricky upon a deeper look. We might have to both change the client side dependency resolution logic, as well as the server queueing logic.

[scv119]

Maybe I can take this. it’s a forcing function for me to get a bit more familiar with this part of the code.