`DirectExchange` type always lookup for queue in `delivery` method

I have profiled the apply_async method, and I’ve figured out that a lot of time is being spend on _lookup method in exchange module if redis is the backend for celery(thus calling SMEMBERS function). I didn’t find a clean way to add another exchange type, so in the end I’ve monkey patched the DirectExchange type to only execute _put method. I would appreciate if someone could point out how can I define custom exchange type.

Check out the snakeviz(profile) graph: You can see^ that 22.7s are being spent on the _lookup method, even more time than for the actual _put method.

So is this _lookup really required, and could it be skipped without monkey patching?

[thedrow]

To be honest, I haven’t looked into the implementation of the Redis broker at all.

The only place we call SMEMBERS is in get_table(): https://github.com/celery/kombu/blob/3a0454c21d5eee7b641af31ac6e046296aa38d57/kombu/transport/redis.py#L824-L830

get_table() is called in _lookup() as you said: https://github.com/celery/kombu/blob/3a0454c21d5eee7b641af31ac6e046296aa38d57/kombu/transport/virtual/base.py#L700-L726

Since we’re talking about the direct exchange we can use SISMEMBER instead to check if a queue with the same name exists in the set.

We could introduce a _lookup_direct method to the virtual channel which will do so when needed.

Care to provide the PR?

[matteius]

@stevanmilic I think there is a caveat to what you are saying. get_table calls smemebers O(n) which has to retrieve all the memebers for the set from redis which could be many, and then to parse that in Python. Instead Redis sismember just checks if the one member is a member of that set and returns a boolean which is O(1) – my optimization in the other PR calls just sismember by not calling get_table – so technically get_table is the performance because it is O(n) and now the reids _lookup implemetnation calls scard and sismemember which are both O(1).

See: https://redis.io/commands/smembers https://redis.io/commands/scard https://redis.io/commands/sismember

Thats why if you can profile this again, I think it will perform better for even users that don’t check of the exchange still exists.