question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Problem when a memcached server changes IP address

See original GitHub issue

We have to configure our client with failureMode=true, and we are getting errors like this when a memcached server gets down and comes back with a different IP:

net.rubyeye.xmemcached.exception.MemcachedException: Session(192.168.1.41:11211) has been closed
	at net.rubyeye.xmemcached.impl.MemcachedConnector.send(MemcachedConnector.java:512)
	at net.rubyeye.xmemcached.XMemcachedClient.sendCommand(XMemcachedClient.java:317)
	at net.rubyeye.xmemcached.XMemcachedClient.fetch0(XMemcachedClient.java:644)
	at net.rubyeye.xmemcached.XMemcachedClient.get0(XMemcachedClient.java:1085)
	at net.rubyeye.xmemcached.XMemcachedClient.get(XMemcachedClient.java:1043)
	at net.rubyeye.xmemcached.XMemcachedClient.get(XMemcachedClient.java:1054)
	at net.rubyeye.xmemcached.XMemcachedClient.get(XMemcachedClient.java:1076)

The old IP was 192.168.1.41 and the new one is 192.168.1.44. When the server recovers with the new IP we can see logs like:

com.google.code.yanf4j.core.impl.AbstractController:? Add a session: 192.168.1.44:11211

However, the client is still using sessions with the old IP that are closed.

I have been debugging a bit and found that in the class net.rubyeye.xmemcached.impl.MemcachedConnector there is an attribute called sessionMap that contains sessions with both the old and new IP, because the new sessions do not override the old ones, and then all those sessions are passed to the session locator in the method updateSessions(). I think the session locator should receive only the sessions with the new IP.

Please could you take a look?

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:15 (14 by maintainers)

github_iconTop GitHub Comments

1reaction
killme2008commented, Feb 22, 2018

I’ve fixed this issue, it happens in failure mode, the new release will be delivered ASAP.

1reaction
killme2008commented, Dec 22, 2017

I will look into it, thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshooting | Memorystore for Memcached - Google Cloud
Memorystore for Memcached requires that you use a private services access connection and an associated IP address range for that connection.
Read more >
Allowing remote connection of memcache server - Server Fault
By default memcached only serves the localhost as it otherwise has no protection from the wider network / internet and is insecure.
Read more >
Securing your Memcached Server - Leaseweb Knowledge Base
You can test your server's IP using the following commands (Replace <ipaddress> with your server's IP address) : Test for UDP:
Read more >
How To Secure Memcached by Reducing Exposure
When I change the IP from -l 127.0.0.1 to -l my server IP for remote access, it gives me: ExecStart=/usr/share/memcached/scripts/systemd- ...
Read more >
Memcached Under Attack - Radware
More ominously, by changing the source IP address of a packet, ... At the core of the Memcached problem is the number of...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found