Master without slaves

Hi:

Deploying mongo replset with side-car rise up to me the following problem:

Master are configured properly but sidecar cant add slave nodes because a NewReplicaSetConfigurationIncompatible error. Master work fine, I can insert and read data but of course, I cant read this data from the slave nodes because they dont appear in the replset configuration as slave.

The logs say that The hosts mongo-0:27017 and 10.36.0.4:27017 all map to this node in new configuration version 2 for replica set rs0 but the IP and the hostname mongo-0 are the same node.

Below the logs from sidecar container and mongo instances, also the IP of mongo instances (they dont run in the same node):

root@s-smartc2-zprei:/opt/kubernetes/mongodb# kubectl get po -o wide
NAME                   READY     STATUS    RESTARTS   AGE       IP          NODE
mongo-0                2/2       Running   0          47m       10.36.0.4   s-smartc3-zprei
mongo-1                2/2       Running   0          47m       10.44.0.2   s-smartc4-zprei
mongo-2                2/2       Running   0          46m       10.32.0.5   s-smartc2-zprei
root@s-smartc2-zprei:/opt/kubernetes/mongodb#

sidecar logs:

Error in workloop { MongoError: The hosts mongo-0:27017 and 10.36.0.4:27017 all map to this node in new configuration version 2 for replica set rs0
    at Function.MongoError.create (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/error.js:31:11)
    at /opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:489:72
    at authenticateStragglers (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:435:16)
    at Connection.messageHandler (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:469:5)
    at Socket.<anonymous> (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/connection.js:321:22)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:191:7)
    at readableAddChunk (_stream_readable.js:178:18)
    at Socket.Readable.push (_stream_readable.js:136:10)
    at TCP.onread (net.js:561:20)
  name: 'MongoError',
  message: 'The hosts mongo-0:27017 and 10.36.0.4:27017 all map to this node in new configuration version 2 for replica set rs0',
  ok: 0,
  errmsg: 'The hosts mongo-0:27017 and 10.36.0.4:27017 all map to this node in new configuration version 2 for replica set rs0',
  code: 103,
  codeName: 'NewReplicaSetConfigurationIncompatible' }

Status of 3 mongo instance:

root@s-smartc2-zprei:/opt/kubernetes/mongodb# for i in `seq 0 2`; do kubectl exec mongo-$i -- sh -c '/usr/bin/mongo --eval="printjson(rs.isMaster())"'; done
Defaulting container name to mongo.
Use 'kubectl describe pod/mongo-0' to see all of the containers in this pod.
MongoDB shell version v3.4.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.8
{
        "hosts" : [
                "mongo-0:27017"
        ],
        "setName" : "rs0",
        "setVersion" : 1,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "mongo-0:27017",
        "me" : "mongo-0:27017",
        "electionId" : ObjectId("59b283bd98a268a30af4eeb3"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1504872388, 1),
                        "t" : NumberLong(-1)
                },
                "lastWriteDate" : ISODate("2017-09-08T12:06:28Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 1000,
        "localTime" : ISODate("2017-09-08T12:06:36.437Z"),
        "maxWireVersion" : 5,
        "minWireVersion" : 0,
        "readOnly" : false,
        "ok" : 1
}
Defaulting container name to mongo.
Use 'kubectl describe pod/mongo-1' to see all of the containers in this pod.
MongoDB shell version v3.4.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.8
{
        "ismaster" : **false**,
        "secondary" : **false**,
        "info" : "**Does not have a valid replica set config**",
        "isreplicaset" : true,
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 1000,
        "localTime" : ISODate("2017-09-08T12:06:33.620Z"),
        "maxWireVersion" : 5,
        "minWireVersion" : 0,
        "readOnly" : false,
        "ok" : 1
}
Defaulting container name to mongo.
Use 'kubectl describe pod/mongo-2' to see all of the containers in this pod.
MongoDB shell version v3.4.8
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.8
{
        "ismaster" : **false**,
        "secondary" : **false**,
        "info" : "**Does not have a valid replica set config**",
        "isreplicaset" : true,
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 1000,
        "localTime" : ISODate("2017-09-08T12:06:37.636Z"),
        "maxWireVersion" : 5,
        "minWireVersion" : 0,
        "readOnly" : false,
        "ok" : 1
}
root@s-smartc2-zprei:/opt/kubernetes/mongodb#

Logs for Members that dont have master or slave state:

> rs.status()
{
        "info" : "run rs.initiate(...) if not yet done for the set",
        "ok" : 0,
        "errmsg" : "**no replset config has been received**",
        "code" : 94,
        "codeName" : "**NotYetInitialized**"
}
>

[antoninbeaufort]

This is a problem of permissions for the sidecar, all we have to do is create a file with this content :

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: default-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default

And do a kubectl apply -f filename.yaml

[majest]

The error occurs because side car tries to add the ip address and the domain name is already there. Adding KUBERNETES_MONGO_SERVICE_NAME env variable to the sidecar solved the problem for me. The value should be the name of the service.

- name: KUBERNETES_MONGO_SERVICE_NAME 
  value: "mongo"