Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

elasticsearch pod is in pending state due to nodegroup down and up

See original GitHub issue

Chart version: 7.17.3

Kubernetes version: 1.21

Kubernetes provider: EKS

Helm Version: 3

helm get release output

base) devops@devops-XPS-15-9570:~/Desktop/Repo$ helm list -n elastic --output yaml
- app_version: 7.17.3
  chart: elasticsearch-7.17.3
  name: elasticsearch
  namespace: elastic
  revision: "1"
  status: deployed
  updated: 2022-05-06 06:15:11.543820007 +0530 IST

Describe the bug: i have deployed elasticsearch with PVC in my EKS cluster and worked fine without issue and we tried to scale down node and scale up started facing issue and pod is in pending state.

Steps to reproduce:

1.helm repo add elastic https://helm.elastic.co 2.with Helm 3: helm install elasticsearch elastic/elasticsearch 3. git clone git@github.com:elastic/helm-charts.git 4. went to elastcsearch working directory and run below 5. kubectl create ns elastic 6. helm install elasticsearch . -n elastic

Expected behavior:

Cluster will run as expected without issue and tried to scale kubectl scale --replicas=0 sts/elasticsearch-master -n elastic kubectl scale --replicas=3 sts/elasticsearch-master -n elastic

now my pod is pending state and stuck

Provide logs and/or server output (if relevant):

Be careful to obfuscate every secrets (credentials, token, public IP, …) that could be visible in the output before copy-pasting

kubectl get po -n elastic
NAME                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0   0/1     Running   0          49m
elasticsearch-master-1   0/1     Pending   0          49m
elasticsearch-master-2   0/1     Pending   0          49m

logs of the running pod and stuck

kubectl logs pod/elasticsearch-master-0 -n elastic
{"type": "server", "timestamp": "2022-05-09T01:03:18,156Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "version[7.17.3], pid[7], build[default/docker/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Linux/5.4.181-99.354.amzn2.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18/18+36]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,159Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,160Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-8450783023809497710, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,531Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,531Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [legacy-geo]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [vector-tile]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded plugin [repository-s3]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,547Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ml] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ml", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,629Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.data", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,644Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdck)]], net usable_space [9.6gb], net total_space [9.7gb], types [ext4]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,645Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,725Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.master", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,729Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.remote_cluster_client", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,730Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ingest", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,846Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "node name [elasticsearch-master-0], node ID [VeyMV5ZNRMW3UyW8taNGkQ], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,848Z", "level": "CRITICAL", "component": "o.e.d.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]]", "key": "legacy role settings", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:33,947Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[controller/180] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV" }
{"type": "server", "timestamp": "2022-05-09T01:03:35,946Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,230Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,232Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-8450783023809497710/geoip-databases/VeyMV5ZNRMW3UyW8taNGkQ]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,421Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,455Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,556Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2022-05-09T01:03:39,663Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,243Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,243Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "starting ..." }
{"type": "server", "timestamp": "2022-05-09T01:03:41,260Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,261Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "deprecation component started" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,458Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {192.168.131.86:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,582Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,636Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "cluster UUID [miG-kHiRS02W-Tf8F-Nqjg]" }
(base) devops@devops-XPS-15-9570:~/Desktop/Repo$ kubectl logs pod/elasticsearch-master-0 -n elastic
{"type": "server", "timestamp": "2022-05-09T01:03:18,156Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "version[7.17.3], pid[7], build[default/docker/5ad023604c8d7416c9eb6c0eadb62b14e766caff/2022-04-19T08:11:19.070913226Z], OS[Linux/5.4.181-99.354.amzn2.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18/18+36]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,159Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:18,160Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-8450783023809497710, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,531Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,531Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,532Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,533Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [legacy-geo]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,534Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,535Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,536Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,537Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [vector-tile]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,538Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,539Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,540Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,541Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,542Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,543Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,544Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded plugin [repository-s3]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,547Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ml] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ml", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,629Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.data", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,644Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvdck)]], net usable_space [9.6gb], net total_space [9.7gb], types [ext4]" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,645Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "heap size [1gb], compressed ordinary object pointers [true]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,725Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.master", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,729Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.remote_cluster_client] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.remote_cluster_client", "category": "settings" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,730Z", "level": "CRITICAL", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "key": "node.ingest", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:23,846Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "node name [elasticsearch-master-0], node ID [VeyMV5ZNRMW3UyW8taNGkQ], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "deprecation.elasticsearch", "timestamp": "2022-05-09T01:03:23,848Z", "level": "CRITICAL", "component": "o.e.d.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "legacy role settings [node.data, node.remote_cluster_client, node.ingest, node.master, node.ml] are deprecated, use [node.roles=[transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]]", "key": "legacy role settings", "category": "settings" }
{"type": "server", "timestamp": "2022-05-09T01:03:33,947Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[controller/180] [Main.cc@122] controller (64 bit): Version 7.17.3 (Build b3cb07de236c11) Copyright (c) 2022 Elasticsearch BV" }
{"type": "server", "timestamp": "2022-05-09T01:03:35,946Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,230Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2022-05-09T01:03:37,232Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-8450783023809497710/geoip-databases/VeyMV5ZNRMW3UyW8taNGkQ]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,421Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,455Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2022-05-09T01:03:38,556Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2022-05-09T01:03:39,663Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,243Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,243Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "starting ..." }
{"type": "server", "timestamp": "2022-05-09T01:03:41,260Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,261Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "deprecation component started" }
{"type": "server", "timestamp": "2022-05-09T01:03:41,458Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {192.168.131.86:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,582Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
{"type": "server", "timestamp": "2022-05-09T01:03:42,636Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "cluster UUID [miG-kHiRS02W-Tf8F-Nqjg]" }
{"type": "server", "timestamp": "2022-05-09T01:03:52,646Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:02,648Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,649Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,650Z", "level": "WARN", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "timed out while waiting for initial discovery state - timeout: 30s" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,661Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {192.168.131.86:9200}, bound_addresses {0.0.0.0:9200}" }
{"type": "server", "timestamp": "2022-05-09T01:04:12,661Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "started" }
{"type": "server", "timestamp": "2022-05-09T01:04:14,174Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }
{"type": "server", "timestamp": "2022-05-09T01:04:22,650Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:24,129Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }
{"type": "server", "timestamp": "2022-05-09T01:04:32,651Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered or elected yet, an election requires 2 nodes with ids [VeyMV5ZNRMW3UyW8taNGkQ, r6usGUfQScG8GBc-39cang], have only discovered non-quorum [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{VeyMV5ZNRMW3UyW8taNGkQ}{TogPUQAJRqqipcnACoBtoQ}{192.168.131.86}{192.168.131.86:9300}{cdfhilmrstw}] from last-known cluster state; node term 1, last-accepted version 196 in term 1" }
{"type": "server", "timestamp": "2022-05-09T01:04:34,136Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.3.jar:7.17.3]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:1
  • Comments:6

github_iconTop GitHub Comments

1reaction
devopsdymyrcommented, May 9, 2022

describe pod

 kubectl describe po elasticsearch-master-0 -n elastic
Name:         elasticsearch-master-0
Namespace:    elastic
Priority:     0
Node:         ip-192-168-131-144.us-east-2.compute.internal/192.168.131.144
Start Time:   Mon, 09 May 2022 06:32:53 +0530
Labels:       app=elasticsearch-master
              chart=elasticsearch
              controller-revision-hash=elasticsearch-master-8495d9f968
              release=elasticsearch
              statefulset.kubernetes.io/pod-name=elasticsearch-master-0
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Running
IP:           192.168.131.86
IPs:
  IP:           192.168.131.86
Controlled By:  StatefulSet/elasticsearch-master
Init Containers:
  configure-sysctl:
    Container ID:  docker://5840e31c6617f887082b979869b6adedfcd1039f15e6568cd9843efff4d1e45b
    Image:         devopsdymyr/elasticsearch:7.17.3
    Image ID:      docker-pullable://devopsdymyr/elasticsearch@sha256:6417d6b350e1ec1ea4fe8f97469b4992c81204e453219bee41571fd773cde028
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 09 May 2022 06:33:10 +0530
      Finished:     Mon, 09 May 2022 06:33:10 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s794p (ro)
Containers:
  elasticsearch:
    Container ID:   docker://77270f3b393b241ac039608e6d95f97c77d701a21b2ae1a8440d2b8d7b1a4cfa
    Image:          devopsdymyr/elasticsearch:7.17.3
    Image ID:       docker-pullable://devopsdymyr/elasticsearch@sha256:6417d6b350e1ec1ea4fe8f97469b4992c81204e453219bee41571fd773cde028
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 09 May 2022 06:33:11 +0530
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Readiness:  exec [bash -c set -e
# If the node is starting up wait for the cluster to be ready (request params: "wait_for_status=green&timeout=1s" )
# Once it has started only check that the node itself is responding
START_FILE=/tmp/.es_start_file

# Disable nss cache to avoid filling dentry cache when calling curl
# This is required with Elasticsearch Docker using nss < 3.52
export NSS_SDB_USE_CACHE=no

http () {
  local path="${1}"
  local args="${2}"
  set -- -XGET -s

  if [ "$args" != "" ]; then
    set -- "$@" $args
  fi

  if [ -n "${ELASTIC_PASSWORD}" ]; then
    set -- "$@" -u "elastic:${ELASTIC_PASSWORD}"
  fi

  curl --output /dev/null -k "$@" "http://127.0.0.1:9200${path}"
}

if [ -f "${START_FILE}" ]; then
  echo 'Elasticsearch is already running, lets check the node is healthy'
  HTTP_CODE=$(http "/" "-w %{http_code}")
  RC=$?
  if [[ ${RC} -ne 0 ]]; then
    echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with RC ${RC}"
    exit ${RC}
  fi
  # ready if HTTP code 200, 503 is tolerable if ES version is 6.x
  if [[ ${HTTP_CODE} == "200" ]]; then
    exit 0
  elif [[ ${HTTP_CODE} == "503" && "7" == "6" ]]; then
    exit 0
  else
    echo "curl --output /dev/null -k -XGET -s -w '%{http_code}' \${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code ${HTTP_CODE}"
    exit 1
  fi

else
  echo 'Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )'
  if http "/_cluster/health?wait_for_status=green&timeout=1s" "--fail" ; then
    touch ${START_FILE}
    exit 0
  else
    echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
    exit 1
  fi
fi
] delay=10s timeout=5s period=10s #success=3 #failure=3
    Environment:
      node.name:                             elasticsearch-master-0 (v1:metadata.name)
      cluster.initial_master_nodes:          elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,
      discovery.seed_hosts:                  elasticsearch-master-headless
      cluster.name:                          elasticsearch
      network.host:                          0.0.0.0
      cluster.deprecation_indexing.enabled:  false
      node.data:                             true
      node.ingest:                           true
      node.master:                           true
      node.ml:                               true
      node.remote_cluster_client:            true
    Mounts:
      /usr/share/elasticsearch/data from elasticsearch-master (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s794p (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  elasticsearch-master:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-master-elasticsearch-master-0
    ReadOnly:   false
  kube-api-access-s794p:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              environment=demo-uch
Tolerations:                 demo-uch=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               51m                  default-scheduler        Successfully assigned elastic/elasticsearch-master-0 to ip-192-168-131-144.us-east-2.compute.internal
  Normal   SuccessfulAttachVolume  51m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-5aab34ac-e41b-4323-9f69-581f677f39f7"
  Normal   Pulled                  50m                  kubelet                  Container image "devopsdymyr/elasticsearch:7.17.3" already present on machine
  Normal   Created                 50m                  kubelet                  Created container configure-sysctl
  Normal   Started                 50m                  kubelet                  Started container configure-sysctl
  Normal   Pulled                  50m                  kubelet                  Container image "devopsdymyr/elasticsearch:7.17.3" already present on machine
  Normal   Created                 50m                  kubelet                  Created container elasticsearch
  Normal   Started                 50m                  kubelet                  Started container elasticsearch
  Warning  Unhealthy               44s (x300 over 50m)  kubelet                  Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )
Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" ) ```
**Any additional context:**

# PVC status
persistentvolumeclaim/elasticsearch-master-elasticsearch-master-0   Bound    pvc-5aab34ac-e41b-4323--581f677f39f7   10Gi       RWO            elastic-sc     3d1h
persistentvolumeclaim/elasticsearch-master-elasticsearch-master-1   Bound    pvc-51bf9a20-f4ca-4f96-cf660f85d107   10Gi       RWO            elastic-sc     3d1h
persistentvolumeclaim/elasticsearch-master-elasticsearch-master-2   Bound    pvc-1e5a1c52-a12d-474f-61f86b999e49   10Gi       RWO            elastic-sc     3d1h
0reactions
botelastic[bot]commented, Oct 15, 2022

This issue has been automatically closed because it has not had recent activity since being marked as stale.

Read more comments on GitHub >

github_iconTop Results From Across the Web

elasticsearch pod is in pending state due to nodegroup down ...
You cannot scale down master nodes directly. To scale down Elasticsearch master nodes, you need to exclude the nodes from voting first.
Read more >
Troubleshooting methods | Elastic Cloud on Kubernetes [2.5]
If an Elasticsearch node does not start up, it is probably because Kubernetes cannot schedule the associated Pod. First, check the StatefulSets to...
Read more >
Deploying E+K onto K8s cluster, multiple subnets and ...
The thread ask for assistance in deploying the Elastic+Kibana stack onto a K8S hosted at AWS as a EKS cluster. Still finding my...
Read more >
Common problems | Elastic Cloud on Kubernetes [2.5]
Scale down of Elasticsearch master-eligible Pods seems stuckedit ... To find out whether you are affected, check if the Pod in question is...
Read more >
Advanced Elasticsearch node scheduling | Elastic Cloud on ...
Topology spread constraints and availability zone awarenessedit. Starting with ECK 2.0 the operator can make Kubernetes Node labels available as Pod annotations ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Use Kubernetes/Helm recommended labels

Next page

[logstash] PodSecurityPolicy - no policy/v1, deprecation since k8s 1.22+