Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Warnings appeared after Redis sentinel restart

See original GitHub issue

Hi,

I’m facing the exact same issue as https://github.com/redisson/redisson/issues/4003, see the logs attached below.

A temporary solution was provided in the first issue. So, I tested with a single Netty thread, but the issue was still here.

Maybe the issue comes from Redis sentinels, but from what I tested with the Redis CLI, I didn’t find any relevant issue.

Any idea that could let me find the root cause of this issue?

Expected behavior

No warning should be produced.

Actual behavior

{"@timestamp":"2022-03-18T16:34:19.239Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:34:25.25Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:34:25.253Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:25.255Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:26.253Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:34:28.258Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:34:28.26Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:28.263Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:29.26Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:34:40.283Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:34:40.285Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:40.287Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:41.285Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:34:42.288Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:34:42.29Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:42.292Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:44.293Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:34:51.305Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:34:51.307Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:51.309Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:52.309Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:34:58.319Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:34:58.321Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:58.323Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:34:59.32Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:35:02.327Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:35:02.33Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:02.331Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:05.333Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:35:22.359Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:35:22.361Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:22.364Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:23.361Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:35:27.37Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:35:27.372Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:27.374Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:29.373Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}
{"@timestamp":"2022-03-18T16:35:33.381Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"slave: redis://10.42.0.180:6379 is up"}
{"@timestamp":"2022-03-18T16:35:33.387Z","@version":"1","logger_name":"org.redisson.connection.pool.PubSubConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"1 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:33.394Z","@version":"1","logger_name":"org.redisson.connection.pool.SlaveConnectionPool","thread_name":"redisson-netty-2-1","level":"INFO","level_value":20000,"message":"24 connections initialized for 10.42.0.180/10.42.0.180:6379"}
{"@timestamp":"2022-03-18T16:35:34.387Z","@version":"1","logger_name":"org.redisson.connection.SentinelConnectionManager","thread_name":"redisson-netty-2-1","level":"WARN","level_value":30000,"message":"slave: redis://10.42.0.180:6379 is down"}

From Redis:

➜ k exec -it redis-node-0 -c redis -- bash -c "redis-cli -h localhost -p 26379 sentinel get-master-addr-by-name redis-master | head -n 1"
redis-node-2.redis-headless.default.svc.cluster.local

➜ k exec -it redis-node-1 -c redis -- bash -c "redis-cli -h localhost -p 26379 sentinel get-master-addr-by-name redis-master | head -n 1"
redis-node-2.redis-headless.default.svc.cluster.local

➜ k exec -it redis-node-2 -c redis -- bash -c "redis-cli -h localhost -p 26379 sentinel get-master-addr-by-name redis-master | head -n 1"
redis-node-2.redis-headless.default.svc.cluster.local

➜ k exec -it redis-node-2 -c redis -- redis-cli -h redis-node-2.redis-headless.default.svc.cluster.local
redis-node-2.redis-headless.default.svc.cluster.local:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=redis-node-1.redis-headless.default.svc.cluster.local,port=6379,state=online,offset=13681281,lag=0
slave1:ip=redis-node-0.redis-headless.default.svc.cluster.local,port=6379,state=online,offset=13681281,lag=0
master_failover_state:no-failover
master_replid:d7de8e84b2be002d0d5e5b01f5429eb7a79d59fb
master_replid2:dd55756fabc189fcd8c06193b46ee1e832b52fe9
master_repl_offset:13681281
second_repl_offset:13415052
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:13387028
repl_backlog_histlen:294254

Steps to reproduce or test case

I’m currently using the bitnami chart to deploy Redis sentinels with the following configuration:

sentinel:
  enabled: true
  masterSet: redis-master
  downAfterMilliseconds: 16000
  failoverTimeout: 8000
  quorum: 2
  parallelSyncs: 1
  persistence:
    enabled: true

replica:
  replicaCount: 3

Redis version

Redis version: 6.2.6

Redisson version

Redisson library version: 3.16.8

Redisson configuration

Issue Analytics

State:
Created 2 years ago
Comments:23 (11 by maintainers)

Top GitHub Comments

1reaction

relgamescommented, May 26, 2022

Hi, this issue is not completed, multiple people including myself reported it.

We can’t disable lock check as it means the lock might be inconsistent, no?

DNS is fine. As others pointed out, it is most likely a concurrency issue, also it needs to use hostnames and not IPs.

0reactions

mrnikocommented, Jun 14, 2022

@jaudiger

Fixed in https://github.com/redisson/redisson/commit/2d3eaba83b3ec8340ecf99ccad048bb550430d1d

Top Results From Across the Web

How do you fix the warning "Redis must be restarted after THP ...

This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as ...

Make sure Redis client library reconnects after Redis failures ...

The webapp / ws servers need to reconnect to the Redis instance after the Redis sentinel says something is down and votes a...

Redis Sentinel Documentation

Redis Sentinel Documentation. Redis Sentinel is a system designed to help managing Redis instances. It performs the following four tasks: Monitoring.

Example sentinel.conf - Redis

Note that whatever is the ODOWN quorum, a Sentinel will require to # be elected by the majority of the known Sentinels in...

Rebooting nodes (cluster mode disabled only)

To reboot multiple nodes you must repeat the process for each node. Redis (Cluster Mode Enabled) parameter changes. If you make changes to...