Error in workloop Quorum check failed HostUnreachable

I’m having an issue getting the sidecar to see my two MongoDB replicas.

Error in workloop { MongoError: Quorum check failed because not enough voting nodes responded; required 3 but only the following 1 voting nodes responded: mongodb-0:27017; the following nodes did not respond affirmatively: mongodb-0.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable, mongodb-1.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable, mongodb-2.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable at Function.MongoError.create (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/error.js:31:11) at /opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:497:72 at authenticateStragglers (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:443:16) at Connection.messageHandler (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:477:5) at Socket.<anonymous> (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/connection.js:331:22) at Socket.emit (events.js:159:13) at addChunk (_stream_readable.js:265:12) at readableAddChunk (_stream_readable.js:252:11) at Socket.Readable.push (_stream_readable.js:209:10) at TCP.onread (net.js:598:20) name: ‘MongoError’, message: ‘Quorum check failed because not enough voting nodes responded; required 3 but only the following 1 voting nodes responded: mongodb-0:27017; the following nodes did not respond affirmatively: mongodb-0.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable, mongodb-1.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable, mongodb-2.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable’, ok: 0, errmsg: ‘Quorum check failed because not enough voting nodes responded; required 3 but only the following 1 voting nodes responded: mongodb-0:27017; the following nodes did not respond affirmatively: mongodb-0.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable, mongodb-1.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable, mongodb-2.mongodb.stage-sprint.svc.cluster.local:27017 failed with HostUnreachable’, code: 74, codeName: ‘NodeNotFound’,

Here’s my Kubernetes yaml configuration that defines my resources.

# Based off of http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html 
# and https://github.com/cvallance/mongo-k8s-sidecar/tree/master/example/StatefulSet
#
# Learn more about StatefulSets, PersistentVolumes and PersistentVolumeClaims:
# https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/
#

# Based off of https://github.com/cvallance/mongo-k8s-sidecar/blob/master/example/StatefulSet/azure_ssd.yaml

# Define the Azure volume storage class for use by our customer MongoDB nodes
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: mongodb-storage
  namespace: stage-sprint
  labels:
    app: mongodb
    tier: storage
provisioner: kubernetes.io/azure-disk
parameters:
  skuName: Premium_LRS
  location: centralus
  storageAccount: <storage_acct>
---

# Create a headless service so each MongoDB pod will get a stable hostname (note: this does NOT load balance)
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: stage-sprint
  labels:
    app: mongodb
    tier: svc
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    app: mongodb
    tier: db
---

# Create the stateful set for the MongoDB nodes
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: stage-sprint
spec:
  serviceName: "mongodb"
  replicas: 3
  template:
    metadata:
      labels:
        app: mongodb
        tier: db
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:latest
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
            - "--bind_ip_all"
          ports:
            - containerPort: 27017
          #imagePullPolicy: Always
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "app=mongodb,tier=db"
            - name: KUBE_NAMESPACE
              value: "stage-sprint"
            - name: KUBERNETES_MONGO_SERVICE_NAME
              value: "mongodb"
      imagePullSecrets:
        - name: regsecret
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "mongodb-storage"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

[tverboon]

Fixed it:

---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-service
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongod
spec:
  serviceName: mongodb-service
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: development
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongod-container
          image: mongo:3.6
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--bind_ip"
            - 0.0.0.0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo"
            - name: KUBERNETES_MONGO_SERVICE_NAME
              value: "mongodb-service"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "fast"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi

Honestly, I am not sure why your version is not working. In my case I had an inconsistency in StatefulSet serviceName and the name of the headless service. I found out when I executed hostname -f in the container.

In your case it’s all serviceName: mongodb. Maybe you should get rid of the quotes in the statefulset:

kind: StatefulSet
metadata:
  name: mongodb
  namespace: stage-sprint
spec:
  serviceName: "mongodb"

[fackyhigh]

Same thing here. This error appears when serviceName in StatefulSet differs from the Headless Service name. And the Headless Service must be created before the StatefulSet.