Elasticsearch Master - Cannot resolved elasticsearch-master-headless

Chart version: latest

Kubernetes version: 1.12.7

Kubernetes provider: E.g. GKE (Google Kubernetes Engine) Bare Metal

Helm Version: 2.14

Values.yaml:

elasticsearch-master:
  enabled: true
  nodeSelector: {role: elasticsearch}
  roles:
    master: "true"
    ingest: "false"
    data: "false"

Describe the bug: After successfully deploying the 3 masters, I have removed one to test recovery, but the master node cannot recover.

The service is running and the other masters are running successfully but the deleted pod cannot resolve the headless DNS (or any for that matter):

Steps to reproduce:

1. Helm install chart
2. Wait for all masters to start (2/2)
3. Delete a pod
4. Pod will sit in 1/2 state, unable to resolve any hosts.

Expected behavior: The pod should recover successfully.

Provide logs and/or server output (if relevant):

"stacktrace": ["java.net.UnknownHostException: elasticsearch-master-headless",
"at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]",
"at java.net.InetAddress.getAllByName0(InetAddress.java:1505) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1364) ~[?:?]",
"at java.net.InetAddress.getAllByName(InetAddress.java:1298) ~[?:?]",
"at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:536) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:489) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:744) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:143) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:835) [?:?]"] }
{"type": "server", "timestamp": "2019-06-02T08:44:01,706+0000", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0",  "message": "failed to resolve host [elasticsearch-master-headless]" , 

[DandyDeveloper]

@Crazybus Sorry for not getting this to you sooner. I actually had a look at the specific node this was running against and it was infact an issue with the node itself.

Everything on the node was effective unable to resolve because it wasn’t appropriately provisioned to access the kube router.

After fixing this and killing the pod, it started to behave. The data node also came up successfully. Sorry to have wasted your time with this.

[daichi703n]

I faced same issue. In my case, firewalld blocks DNS request. Disabling firewalld (or permit 53/udp,tcp) fixes this issue.