system ID mismatch, node belongs to a different cluster - after reboot

Hi,

I ended up with a node out of the cluster after a minor postgresql upgrade and system reboot. I did this before in the past but never with patroni 1.2.5.

Error:

2017-10-10 11:20:44,915 CRITICAL: system ID mismatch, node sql2 belongs to a different cluster: 6283810682861444259 !=
2017-10-10 11:20:44,931 ERROR:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/patroni/__init__.py", line 134, in patroni_main
    patroni.run()
  File "/usr/local/lib/python3.5/dist-packages/patroni/__init__.py", line 110, in run
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.5/dist-packages/patroni/ha.py", line 946, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.5/dist-packages/patroni/ha.py", line 903, in _run_cycle
    sys.exit(1)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 329, in wrapper
    retval = func(self, *args, **kwargs) is not None
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 511, in delete_leader
    return self._client.delete(self.leader_path, prevValue=self._name)
  File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 584, in delete
    self.key_endpoint + key, self._MDELETE, params=kwds)
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 211, in api_execute
    return self._handle_server_response(response)
  File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 928, in _handle_server_response
    etcd.EtcdError.handle(r)
  File "/usr/local/lib/python3.5/dist-packages/etcd/__init__.py", line 304, in handle
    raise exc(msg, payload)
etcd.EtcdCompareFailed: Compare failed : [sql2 != sql1]

I found this: https://github.com/zalando/patroni/issues/438 Edited that file, indented return to proper position but still the same issue. Any ideas on how to debug this?

Thank you.

[bradnicholson]

For our use case, since we are doing the pg_upgrade on newly provisioned DB’s only, the simplest approach is to upgrade the data before Patroni sees it.

[ghost]

I doubt that you want to delete the key from Consul, as it reports that there is an existing lock owner (odoo-psql). The initialize key comes from PostgreSQL, and can be seen using the pg_controldata command. All instances in your Patroni cluster should have the same ID, which will be the case if the second node was not initialized as a separate database but rather created as a copy of the first.

# /usr/lib/postgresql/12/bin/pg_controldata -D /var/lib/postgresql/12/main | grep "Database system identifier"
Database system identifier:           6792036218013953695

The only reason to delete the key from Consul is if you want to replace the existing database cluster entirely. To do that, you can use the Consul UI, Key/Value tab.