7.10.1 Elasticsearch pods aren't readying up but curl -k http://localhost:9200/_cluster/health says green

I’m deploying elasticsearch 7.10.1 on GKE with xpack security and TLS on HTTPS enabled via helm3 and my elasticsearch pods aren’t readying up.

GKE tells me: Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" ) Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )

When I open a shell into elasticsearch-master-0 and curl itself using: curl --cert-type P12 --cert elastic-certificates.p12 http://elastic:<password>@localhost:9200/_cluster/health

The response is:

{"cluster_name":"elasticsearch","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"r
elocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"tas
k_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0} 

Lookin into the readiness probe section in the statefulset yaml I tried: curl -k http://elastic:<password>@localhost:9200/_cluster/health

and get:

{"cluster_name":"elasticsearch","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"r
elocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"tas
k_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

Any links to any solutions or suggestions?

[kudrew]

Relevant slack thread here: https://elasticstack.slack.com/archives/CNTP6NLTZ/p1609969609026200

[mix4242]

I had the same problem. Single node cluster, security + forced bootstrap checks. Probe failing but curl from inside pod returning green.

Problem was misconfiguration. As seen here: https://github.com/elastic/helm-charts/blob/c2689ff4241bc6a75f591dcc2c49a409eb8ef90e/elasticsearch/templates/statefulset.yaml#L250, both ELASTIC_USERNAME & ELASTIC_PASSWORD need to be set in order for the probe to correctly auth - I was only specifying password as I thought elastic username would be used by default. Adding ELASTIC_USERNAME env var fixed it!