Problems with concurrent subscriptions (pubsub)

I’ve implemented a websocket server that uses one pubsub connection which is then used by every individual connection handler to subscribe to a pubsub-channel only for this connection.

During load tests with multiple connections sometimes it happens, that the subscribe() method returns an existing channel of a previous subscribe-call. If I print the read-only channels dict, the requested channel is there, but the returned channel is not the expected one.

Here is a basic example with some logging, just to show you what happens. First imagine a server implementation like this:

#...
redis_pubsub = asyncio.get_event_loop().run_until_complete(aioredis.create_redis(('localhost', 6379), encoding='utf-8'))
#...

@asyncio.coroutine
def handler(websocket, connection_id):
    connection_channel_key = 'connection:{}:sub'.format(connection_id)
    channel, = yield from redis_pubsub.subscribe(connection_channel_key)

    logger.debug('subscribing to: "%s", got: "%s", all channels: "%s"' % (connection_id, channel, redis_pubsub.channels))

    # ... loop with websocket and pubsub processing
    # on close:
    yield from redis_pubsub.unsubscribe(connection_channel_key)

During load testing this happens:

... connection "1" and "2" subscribed

subscribing to: "connection:3:sub", got: "<Channel name:b'connection:3:sub', is_pattern:False, qsize:0>",
all channels: "{
b'connection:3:sub': <Channel name:b'connection:3:sub', is_pattern:False, qsize:0>, 
b'connection:2:sub': <Channel name:b'connection:2:sub', is_pattern:False, qsize:0>, 
b'connection:1:sub': <Channel name:b'connection:1:sub', is_pattern:False, qsize:0>
}"

... connection "1" and "2" unsubscribed...

subscribing to: "connection:4:sub", got: "<Channel name:b'connection:3:sub', is_pattern:False, qsize:0>",
all channels: "{
b'connection:3:sub': <Channel name:b'connection:3:sub', is_pattern:False, qsize:0>, 
b'connection:4:sub': <Channel name:b'connection:4:sub', is_pattern:False, qsize:0>
}"

The channel for connection 3 is returned again, but the newly subscribed channel for connection 4 is already present in the channels dict.

On the first look it seems obvious, that the library does not support concurrent subscribe calls. So only sequential (yielded) calls to subscribe seem to work. Is this right?

But even if it is not supported, it looks like there is also a bug in there. Why does a subsequent call to subscribe can return an already used channel? I would not think of a bug if both subscribe-calls had returned just the other channel.

[popravich]

@th3hamm0r , try version from master, it should be fixed now.

[th3hamm0r]

Nice, looks like you fixed it. I’ve removed the temporary locks from my code and tested it. Thanks a lot @popravich !