MULTI is dispatched to slave nodes using SLAVE readFrom

Hi there,

We are experiencing an issue on multi/exec transactions with master/slave configurations (readFrom is set as SLAVE). We are using spring-data-redis and selected lettuce as the redis framework.

Here is the code we wrote to catch the issue

RedisConnection connection= RedisConnectionUtils.getConnection(stringRedisTemplate.getConnectionFactory(), true);
      try {
                connection.watch(key.getBytes());
                connection.multi();                  
                connection.hashCommands().hMSet(key.getBytes(), hash);
                connection.exec();
        } catch (Exception e) {
        } finally {
            RedisConnectionUtils.releaseConnection(connection, stringRedisTemplate.getConnectionFactory());
        }

We set a master and slave to test it out hosts: - 127.0.0.1:6379 (master) - 127.0.0.1:6380

Then we hit the exception nested exception is io.lettuce.core.RedisCommandExecutionException: ERR EXEC without MULTI, And we found the the watch(), multi(), exec() are distributed between the master and slave.

MASTER [0 127.0.0.1:54159] “WATCH” “xxxx” [0 127.0.0.1:54159] “HMSET” “xxxx”

SLAVE [0 127.0.0.1:54157] “PING” “NODES” [0 127.0.0.1:54160] “MULTI”

This could explain why the “ERR EXEC without MULTI” happens. But is this a bug or there is a way to work around? We also tried with stringRedisTemplate.execute(new SessionCallback<Boolean>(){...} but there are no difference.

Right now What I can think about is to have another ConnectionFactory with readFrom -> MASTER, and move all transactional commands to its maintained connections. But this looks like a ugly fix… Please advice.

I also saw a comment in another ticket : “Lettuce connections require single-threaded/synchronized access when using transactions. If two or more threads call concurrently MULTI/EXEC methods this will destroy the connection state. I don’t think there is anything we could do here.” Wondering the “single-threaded/synchronized” requirement mentioned there is to the RedisConnection or to the shared native connection?

Thanks

[mp911de]

That’s fixed now, we no longer consider MULTI to be a read-only command.

[mp911de]

Does it mean if the readFrom set as MASTER, this might work

Yes, this will work

Or we still need to guarantee the single-thread/synchronized when handling the connection

Using transactions and/or blocking commands (BRPOP and such) requires a synchronized connection access (potentially connection pooling). If you don’t use transactions or blocking commands, then use a single connection.

If you don’t mind, could you please help explain how the LettuceConnection

That’s a specific implementation of Spring Data Redis. Spring Data takes care of the hairy bits for you meaning, that transactions, pipelining, and blocking commands get issued on dedicated connections.

ShareNativeConnection defaults to true to to maximize efficiency. Lettuce is thread-safe, so a single connection may be shared across multiple threads unless you change the connection state or block the connection. That’s what Spring Data is adapting to. The only thing you need pay attention to is disabling slave reads when you want to use transactions.

If you want to use Master/Slave with slave reads, then spin up another LettuceConnectionFactory (and share ClientResources amongst these to share thread pools) that isn’t used for transactions.

Does this make sense?