Error on creating cluster on distributed nodes: ids are different

Am using Orient DB 2.1.0 and running in clustered environment. I tried with 2.0.13, same issue. Am trying to setup 3 Orient DB servers and create database in one of them

hazelcast.xml looks like this:

<hazelcast xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.h
azelcast.com/schema/config hazelcast-config-3.0.xsd">
  <group>
    <name>orient-db</name>
    <password>cluster-db</password>
  </group>
  <network>
    <port auto-increment="false">2434</port>
    <join>
      <multicast enabled="false">
        <multicast-group>235.1.1.1</multicast-group>
        <multicast-port>2434</multicast-port>
      </multicast>
      <tcp-ip enabled="true">
          <!-- Host names of 3 OrientDB servers and port number -->
          <member>orientdb1-prod-dev-brn:2434</member>
          <member>orientdb2-prod-dev-brn:2434</member>
          <member>orientdb3-prod-dev-brn:2434</member>       
      </tcp-ip>
    </join>
    <symmetric-encryption enabled="true">
      <algorithm>PBEWithMD5AndDES</algorithm>
      <!-- salt value to use when generating the secret key -->
      <salt>orientdb-cluster-salt</salt>
      <!-- pass phrase to use when generating the secret key -->
      <password>orientdb-cluster-pp</password>
      <!-- iteration count to use when generating the secret key -->
      <iteration-count>19</iteration-count>
    </symmetric-encryption>
  </network>
  <executor-service>
    <pool-size>16</pool-size>
  </executor-service>
</hazelcast>

Handler OHazelcastPlugin in orientdb-server-config.xml looks like this

        <!-- CLUSTER PLUGIN, TO TURN ON SET THE 'ENABLED' PARAMETER TO 'true' -->
        <handler class="com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin">
            <parameters>
                <parameter name="nodeName" value="orientdb3-prod-dev-brn" />
                <parameter name="enabled" value="true"/>
                <parameter name="configuration.db.default"
                           value="${ORIENTDB_HOME}/config/default-distributed-db-config.json"/>
                <parameter name="configuration.hazelcast" value="${ORIENTDB_HOME}/config/hazelcast.xml"/>
            </parameters>
        </handler>

default-distributed-db-config.json looks like this

{
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "asynchronous",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "servers": {
        "*": "master"
  },
  "clusters": {
    "internal": {},
    "index": {},
    "*": {
      "servers": [
        "<NEW_NODE>"
      ]
    }
  }
}

ERROR in logs

2015-08-10 21:03:58:208 INFO  [product-orientdb3-prod-dev-brn]--[product-orientdb2-prod-dev-brn] update configuration db=product [OHazelcastPlugin]
2015-08-10 21:04:06:407 INFO  [product-orientdb3-prod-dev-brn] class 'someEdgeFrom', creation of new local cluster 'someedgefrom_product-orientdb3-prod-dev-brn' (id=-1) [OHazelcastPlugin]
2015-08-10 21:04:09:992 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 0/10... [ODistributedStorage]
2015-08-10 21:04:16:949 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 1/10... [ODistributedStorage]
2015-08-10 21:04:20:619 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 2/10... [ODistributedStorage]
2015-08-10 21:04:22:369 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 3/10... [ODistributedStorage]
2015-08-10 21:04:24:767 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 4/10... [ODistributedStorage]
2015-08-10 21:04:26:191 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 5/10... [ODistributedStorage]
2015-08-10 21:04:27:593 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 6/10... [ODistributedStorage]
2015-08-10 21:04:28:956 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 7/10... [ODistributedStorage]
2015-08-10 21:04:30:331 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 8/10... [ODistributedStorage]
2015-08-10 21:04:31:699 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 9/10... [ODistributedStorage]
2015-08-10 21:04:32:799 SEVER [product-orientdb3-prod-dev-brn] error on creating cluster 'someedgefrom_product-orientdb3-prod-dev-brn' in class 'someEdgeFrom':  [OHazelcastPlugin]Exception in thread "main" com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'someedgefrom_product-orientdb3-prod-dev-brn' in class 'someEdgeFrom'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:224)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1055)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:937)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1350)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:184)
        at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:971)
        at com.orientechnologies.orient.server.OServer.activate(OServer.java:338)
        at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'someedgefrom_product-orientdb3-prod-dev-brn' in class 'someEdgeFrom'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1494)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1207)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$1.call(OHazelcastPlugin.java:1058)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$1.call(OHazelcastPlugin.java:1055)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:222)
        ... 7 more
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
        at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1268)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1253)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2124)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1105)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1487)
        ... 11 more

[aleksnikolic]

+1 The same issue as others.

I have upgraded my docker image to 2.1.5 version and running it in distributed mode directly from docker-compose file. I can put some startup script to create databases before running it in distributed mode but I don’t think that’s a good option for production.

Do you guys think you can fix this quickly?

[shubhamod1]

done that also.Initially speed is fine but later it depreciates and after some time the process stops.