Ganeti 2.12 - CA is unknown

Hello, I inherited a really old Ganetic cluster that came to the time of needing it’s certificates renewed.

I did the following: gnt-cluster renew-crypto --new-cluster-certificate --new-node-certificates

This ended up breaking my 2 node cluster (I know, kill me) to the point where I had to mark the 2nd node as offline.

Now, when I try to readd the node I get the following:

2021-12-14 16:35:18,327: Unhandled Ganeti error: Given cluster certificate does not match local key Failure: command execution error: Command 'ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oPort=22 -oStrictHostKeyChecking=ask -4 [e-mail] '/bin/sh -c '\''test -d /usr/local/lib/ganeti/2.12 && ln -s -f -T /usr/local/lib/ganeti/2.12 /etc/ganeti/lib && ln -s -f -T /usr/local/share/ganeti/2.12 /etc/ganeti/share && /usr/local/lib/ganeti/prepare-node-join'\'''' failed: exited with exit code 1

And if I try to sett offline to no, I get:

root@phx-gnt-1[A79]:~$ gnt-node modify --offline=no phx-gnt-2.x.com Tue Dec 14 16:15:57 2021 - INFO: Auto-promoting node to master candidate Failure: prerequisites not met for this operation: error type: wrong_state, error details: Node phx-gnt-2.x.com is being de-offlined but fails to report its version: Error 35: gnutls_handshake() failed: CA is unknown

I tried the following from the snippet in the below article:

https://groups.google.com/g/ganeti/c/CusPuOWuHYE/m/OKcv12GT5hUJ

But that left me with:

root@phx-gnt-1[A79]:anthony.cleaves$ python -c 'import pycurl; print pycurl.version' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (gnutls)

So I reverted that change.

If I run cluster verify I get the same Unknown CA message, so I can’t seem to repair my cluster until I get the 2nd node back online, but can’t seem to fix that with fixing the CA issue so it’s a case of chicken and egg.

I appreciate this is a really old cluster version, and we are looking to deprecate it but for that I need the cluster online.

I’d appreciate any feedback, thanks.

[saschalucas]

I really should have read what you wrote more accurately. So I think the fix was a combination of things.

👍 I’m glade to hear you manged to get it working again. Yes seems a combination …

WCONFD_ARGS=“–no-voting --yes-do-it” LUXID_ARGS=“–no-voting --yes-do-it”

Don’t forget to remove this ARGS and restart services on master after everything works again.

[saschalucas]

Hi,

I recently had the same issue with a two node Cluster using Ganeti-2.14. I observed, that cert renew is broken in two node clusters, after certs has expired.

Just a quick try… I’ve forced the current known master to always start by inserting --no-voting --yes-do-it in /etc/default/ganeti. In 2.14 this is WCONFD_ARGS and LUXID_ARGS. Maybe 2.12 has still the single MASTERD??? Bring up the master (/etc/init.d/ganeti restart). After that gnt-cluster renew-crypto worked. If everything is OK, remove the no-voting from defaults file and restart services on master.

HTH, Sascha.