Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Crate on Kubernetes: master not discovered yet, this node has not previously joined a bootstrapped (v4+) cluster, and this node must discover master-eligible nodes [crate-0, crate-1] to bootstrap a cluster: have discovered []

See original GitHub issue

CrateDB version

4.2.4 (also tried on latest)

CrateDB setup information

Environment: Kubernetes Nodes: 2 CRATE_HEAP_SIZE: 16GB

Node startup args: - -Cnode.max_local_storage_nodes=2 - -Cnode.name=${POD_NAME} - -Ccluster.name=${CLUSTER_NAME} - -Ccluster.initial_master_nodes=crate-0,crate-1 - -Cdiscovery.seed_providers=srv - -Cdiscovery.srv.query=_crate-internal._tcp.crate-internal-service.${NAMESPACE}.svc.cluster.local - -Cgateway.recover_after_nodes=2 - -Cgateway.expected_nodes=${EXPECTED_NODES} - -Cpath.data=/data

Steps to Reproduce

Follow the Run CrateDB on Kubernetes steps. I have tried with both 2 and 3 node configurations and cannot get the cluster started.

StatefulSet yaml:

kind: StatefulSet
apiVersion: "apps/v1"
metadata:
  # This is the name used as a prefix for all pods in the set.
  name: crate
spec:
  serviceName: "crate-set"
  # Our cluster has two nodes.
  replicas: 2
  selector:
    matchLabels:
      # The pods in this cluster have the `app:crate` app label.
      app: crate
  template:
    metadata:
      labels:
        app: crate
    spec:
      serviceAccount: crate-user
      serviceAccountName: crate-user
      # InitContainers run before the main containers of a pod are
      # started, and they must terminate before the primary containers
      # are initialized. Here, we use one to set the correct memory
      # map limit.
      initContainers:
      - name: init-sysctl
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      # This final section is the core of the StatefulSet configuration.
      # It defines the container to run in each pod.
      containers:
      - name: crate
        # Use the CrateDB 4.2.4 Docker image.
        image: crate:4.2.4
        # Pass in configuration to CrateDB via command-line options.
        # We are setting the name of the node's explicitly, which is
        # needed to determine the initial master nodes. These are set to
        # the name of the pod.
        # We are using the SRV records provided by Kubernetes to discover
        # nodes within the cluster.
        args:
          - -Cnode.max_local_storage_nodes=2
          - -Cnode.name=${POD_NAME}
          - -Ccluster.name=${CLUSTER_NAME}
          - -Ccluster.initial_master_nodes=crate-0,crate-1
          - -Cdiscovery.seed_providers=srv
          - -Cdiscovery.srv.query=_crate-internal._tcp.crate-internal-service.${NAMESPACE}.svc.cluster.local
          - -Cgateway.recover_after_nodes=2
          - -Cgateway.expected_nodes=${EXPECTED_NODES}
          - -Cpath.data=/data
        volumeMounts:
          - mountPath: /data
            name: data
          - mountPath: /eval
            name: evaluation-data
        resources:
          requests:
            cpu: '10'
            memory: 100Gi
        ports:
          # Port 4300 for inter-node communication.
        - containerPort: 4300
          name: crate-internal
          # Port 4200 for HTTP clients.
        - containerPort: 4200
          name: crate-web
          # Port 5432 for PostgreSQL wire protocol clients.
        - containerPort: 5432
          name: postgres
        # Environment variables passed through to the container.
        env:
          # These variables are detected by CrateDB.
        - name: CRATE_HEAP_SIZE
          value: "16g"
          # The rest of these variables are used in the command-line
          # options.
        - name: EXPECTED_NODES
          value: "2"
        - name: CLUSTER_NAME
          value: "crate"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: crate-data
        - name: evaluation-data
          persistentVolumeClaim:
            claimName: evaluation-data

Service yaml:

kind: Service
apiVersion: v1
metadata:
  name: crate-internal-service
  labels:
    app: crate
spec:
  # A static IP address is assigned to this service. This IP address is
  # only reachable from within the Kubernetes cluster.
  type: ClusterIP
  ports:
    # Port 4200 for inter-node communication.
  - port: 4200
    name: crate-internal
  - port: 5432
    name: crate-postgres
  selector:
    # Apply this to all nodes with the `app:crate` label.
    app: crate

Expected Result

Cluster bootstraps successfully

Actual Result

Cluster does not bootstrap with the following errors:

[2022-01-06T16:57:36,298][WARN ][o.e.c.c.ClusterFormationFailureHelper] [crate-0] master not discovered yet, this node has not previously joined a bootstrapped (v4+) cluster, and this node must discover master-eligible nodes [crate-0, crate-1] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:4301, 127.0.0.1:4302, 127.0.0.1:4303, 127.0.0.1:4304, 127.0.0.1:4305, [::1]:4301, [::1]:4302, [::1]:4303, [::1]:4304, [::1]:4305, 172.21.251.246:4200] from hosts providers and [{crate-0}{Nu-cXh51Q8mG84e4OY3oYA}{Qe7brzuaRPO2krIITahI2A}{172.30.67.229}{172.30.67.229:4300}{http_address=172.30.67.229:4200}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

[2022-01-06T16:57:44,351][WARN ][o.e.t.n.Netty4Transport  ] [crate-0] exception caught on transport layer [NettyTcpChannel{localAddress=/172.30.67.229:39570, remoteAddress=crate-internal-service.poc-db-crate.svc.cluster.local/172.21.251.246:4200}], closing connection

io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)

Node 0 logs:

[2022-01-06T16:57:23,047][INFO ][o.e.e.NodeEnvironment    ] [crate-0] heap size [16gb], compressed ordinary object pointers [true]
[2022-01-06T16:57:23,055][INFO ][o.e.n.Node               ] [crate-0] node name [crate-0], node ID [Nu-cXh51Q8mG84e4OY3oYA]
[2022-01-06T16:57:23,071][INFO ][o.e.n.Node               ] [crate-0] version[4.2.4], pid[1], build[62b0092/2020-08-26T14:15:09Z], OS[Linux/3.10.0-1160.49.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.2/14.0.2+12]
[2022-01-06T16:57:23,303][INFO ][i.c.plugin               ] [crate-0] plugins loaded: [jmx-monitoring, lang-js, enterpriseFunctions] 
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-01-06T16:57:24,394][INFO ][o.e.p.PluginsService     ] [crate-0] no modules loaded
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [crate-azure-discovery]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [es-repository-hdfs]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [io.crate.plugin.BlobPlugin]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [io.crate.plugin.PluginLoaderPlugin]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [io.crate.plugin.SrvPlugin]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [io.crate.udc.plugin.UDCPlugin]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
[2022-01-06T16:57:24,396][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin]
[2022-01-06T16:57:24,397][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
[2022-01-06T16:57:24,397][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin]
[2022-01-06T16:57:24,397][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
[2022-01-06T16:57:24,397][INFO ][o.e.p.PluginsService     ] [crate-0] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
[2022-01-06T16:57:25,285][INFO ][o.e.d.DiscoveryModule    ] [crate-0] using discovery type [zen] and seed hosts providers [settings, srv]
[2022-01-06T16:57:25,942][INFO ][psql                     ] [crate-0] PSQL SSL support is disabled.
[2022-01-06T16:57:26,168][INFO ][i.c.p.PipelineRegistry   ] [crate-0] HTTP SSL support is disabled.
[2022-01-06T16:57:26,215][INFO ][o.e.n.Node               ] [crate-0] initialized
[2022-01-06T16:57:26,216][INFO ][o.e.n.Node               ] [crate-0] starting ...
[2022-01-06T16:57:26,253][INFO ][psql                     ] [crate-0] publish_address {172.30.67.229:5432}, bound_addresses {[::1]:5432}, {127.0.0.1:5432}, {172.30.67.229:5432}
[2022-01-06T16:57:26,268][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-0] publish_address {172.30.67.229:4200}, bound_addresses {[::1]:4200}, {127.0.0.1:4200}, {172.30.67.229:4200}
[2022-01-06T16:57:26,281][INFO ][o.e.t.TransportService   ] [crate-0] publish_address {172.30.67.229:4300}, bound_addresses {[::1]:4300}, {127.0.0.1:4300}, {172.30.67.229:4300}
[2022-01-06T16:57:26,285][INFO ][o.e.b.BootstrapChecks    ] [crate-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-01-06T16:57:26,351][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:27,298][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:28,297][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:29,299][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:30,299][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:31,299][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:32,300][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:33,300][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:34,301][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:35,302][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:36,298][WARN ][o.e.c.c.ClusterFormationFailureHelper] [crate-0] master not discovered yet, this node has not previously joined a bootstrapped (v4+) cluster, and this node must discover master-eligible nodes [crate-0, crate-1] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:4301, 127.0.0.1:4302, 127.0.0.1:4303, 127.0.0.1:4304, 127.0.0.1:4305, [::1]:4301, [::1]:4302, [::1]:4303, [::1]:4304, [::1]:4305, 172.21.251.246:4200] from hosts providers and [{crate-0}{Nu-cXh51Q8mG84e4OY3oYA}{Qe7brzuaRPO2krIITahI2A}{172.30.67.229}{172.30.67.229:4300}{http_address=172.30.67.229:4200}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-01-06T16:57:36,302][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:37,302][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:38,302][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:39,303][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:40,304][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:41,303][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:42,304][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:43,304][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:44,306][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-0] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:44,351][WARN ][o.e.t.n.Netty4Transport  ] [crate-0] exception caught on transport layer [NettyTcpChannel{localAddress=/172.30.67.229:39570, remoteAddress=crate-internal-service.poc-db-crate.svc.cluster.local/172.21.251.246:4200}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
	at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1052) ~[crate-app.jar:4.2.4]
	at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:42) ~[crate-app.jar:4.2.4]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	... 18 more

Node 1 logs:

[2022-01-06T16:57:32,559][INFO ][o.e.e.NodeEnvironment    ] [crate-1] using [1] data paths, mounts [[/data (fsf-dal1302g-fz.service.softlayer.com:/IBM02SEV305490_14389/data01)]], net usable_space [1023.9gb], net total_space [1tb], types [nfs4]
[2022-01-06T16:57:32,562][INFO ][o.e.e.NodeEnvironment    ] [crate-1] heap size [16gb], compressed ordinary object pointers [true]
[2022-01-06T16:57:32,581][INFO ][o.e.n.Node               ] [crate-1] node name [crate-1], node ID [IeXgYqRWQlOevLjvvysrVQ]
[2022-01-06T16:57:32,599][INFO ][o.e.n.Node               ] [crate-1] version[4.2.4], pid[1], build[62b0092/2020-08-26T14:15:09Z], OS[Linux/3.10.0-1160.25.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.2/14.0.2+12]
[2022-01-06T16:57:32,986][INFO ][i.c.plugin               ] [crate-1] plugins loaded: [jmx-monitoring, enterpriseFunctions, lang-js] 
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-01-06T16:57:33,985][INFO ][o.e.p.PluginsService     ] [crate-1] no modules loaded
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [crate-azure-discovery]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [es-repository-hdfs]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [io.crate.plugin.BlobPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [io.crate.plugin.PluginLoaderPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [io.crate.plugin.SrvPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [io.crate.udc.plugin.UDCPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.discovery.ec2.Ec2DiscoveryPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.plugin.analysis.AnalysisPhoneticPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.plugin.repository.url.URLRepositoryPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.repositories.azure.AzureRepositoryPlugin]
[2022-01-06T16:57:33,987][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.repositories.s3.S3RepositoryPlugin]
[2022-01-06T16:57:33,988][INFO ][o.e.p.PluginsService     ] [crate-1] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
[2022-01-06T16:57:34,920][INFO ][o.e.d.DiscoveryModule    ] [crate-1] using discovery type [zen] and seed hosts providers [settings, srv]
[2022-01-06T16:57:35,707][INFO ][psql                     ] [crate-1] PSQL SSL support is disabled.
[2022-01-06T16:57:35,970][INFO ][i.c.p.PipelineRegistry   ] [crate-1] HTTP SSL support is disabled.
[2022-01-06T16:57:36,028][INFO ][o.e.n.Node               ] [crate-1] initialized
[2022-01-06T16:57:36,028][INFO ][o.e.n.Node               ] [crate-1] starting ...
[2022-01-06T16:57:36,071][INFO ][psql                     ] [crate-1] publish_address {172.30.196.226:5432}, bound_addresses {[::1]:5432}, {127.0.0.1:5432}, {172.30.196.226:5432}
[2022-01-06T16:57:36,089][INFO ][o.e.h.n.Netty4HttpServerTransport] [crate-1] publish_address {172.30.196.226:4200}, bound_addresses {[::1]:4200}, {127.0.0.1:4200}, {172.30.196.226:4200}
[2022-01-06T16:57:36,104][INFO ][o.e.t.TransportService   ] [crate-1] publish_address {172.30.196.226:4300}, bound_addresses {[::1]:4300}, {127.0.0.1:4300}, {172.30.196.226:4300}
[2022-01-06T16:57:36,109][INFO ][o.e.b.BootstrapChecks    ] [crate-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-01-06T16:57:36,198][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:37,126][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:38,126][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:39,126][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:40,127][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:41,127][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:42,128][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:43,129][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:44,129][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:45,129][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:46,126][WARN ][o.e.c.c.ClusterFormationFailureHelper] [crate-1] master not discovered yet, this node has not previously joined a bootstrapped (v4+) cluster, and this node must discover master-eligible nodes [crate-0, crate-1] to bootstrap a cluster: have discovered []; discovery will continue using [127.0.0.1:4301, 127.0.0.1:4302, 127.0.0.1:4303, 127.0.0.1:4304, 127.0.0.1:4305, [::1]:4301, [::1]:4302, [::1]:4303, [::1]:4304, [::1]:4305, 172.21.251.246:4200] from hosts providers and [{crate-1}{IeXgYqRWQlOevLjvvysrVQ}{PoMSzXHHS9Sjc6SjlIBKZA}{172.30.196.226}{172.30.196.226:4300}{http_address=172.30.196.226:4200}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-01-06T16:57:46,129][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:47,130][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:48,130][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:49,131][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:50,133][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:51,132][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:52,133][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:53,134][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:54,133][INFO ][i.c.d.SrvUnicastHostsProvider] [crate-1] Using dynamic nodes [172.21.251.246:4200]
[2022-01-06T16:57:54,158][WARN ][o.e.t.n.Netty4Transport  ] [crate-1] exception caught on transport layer [NettyTcpChannel{localAddress=/172.30.196.226:60912, remoteAddress=crate-internal-service.poc-db-crate.svc.cluster.local/172.21.251.246:4200}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
	at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1052) ~[crate-app.jar:4.2.4]
	at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:42) ~[crate-app.jar:4.2.4]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
	... 18 more```

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
nwsmcommented, Jan 10, 2022

@proddata thank you so much, this was the issue! I must have gotten configs crossed when making the service.

@mbaron2 check proddata’s comment and check your service configuration

1reaction
proddatacommented, Jan 10, 2022

the internal service definition seems to be wrong and not follow the guide. CrateDBs inter-node communication is typically handled on port 4300. Your service configuration only seems to define the external ports 4200 (http) and 5432 (pg)

Read more comments on GitHub >

github_iconTop Results From Across the Web

Master not discovered yet, this node has not previously joined ...
I'm trying to upgrade from 6.7.1 to 7.0.0 but getting the below error. I changed discovery.zen.ping.unicast.hosts to discovery.seed_hosts ...
Read more >
master not discovered yet, this node has not previously joined ...
On a Kubernetes Cluster Version 1.21.3 i tried to deploy OpenSearch with Helm. Cloned https://github.com/opensearch-project/helm-charts; run in the folder ...
Read more >
Unable to bootstrap Elasticsearch v7.4.0 3 node cluster
... master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible ...
Read more >
Securing a Cluster | Kubernetes
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended ......
Read more >
Run Your First CrateDB Cluster on Kubernetes, Part One
This cluster must have at least one master node and may have any number of non-master (i.e., worker) nodes. Containers (that run the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Table isn't suddenly accepting new data because of bug

Next page

HBA: Enforcing transport ssl breaks other authentication methods