Joining a cluster not possible in v1.24

Summary

Setting up a fresh cluster according to the docs results in a “Token invalid. (500)” error on the joining node. Building a microk8s cluster is therefore not possible.

What Should Happen Instead?

Joining a cluster using the microk8s join command generated by the microk8s add-node from another node should join them together to a cluster.

Reproduction Steps

Tested on:

• Two VMs from DigitalOcean
• Ubuntu 20.04.4 LTS
• Linux node 5.4.0-107-generic 121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I was able to reliably reproduce it also on servers from another hoster (Hetzner) and even on Debian 11.

Steps:

1. Node A: snap install microk8s --classic --channel=1.24/stable
2. Node B: echo "10.0.0.3 node-b" >> /etc/hosts
3. Node B: snap install microk8s --classic --channel=1.24/stable
4. Node A: microk8s add-node
5. Node B: microk8s join 10.0.0.2:25000/token/token

   Contacting cluster at 10.0.0.2
   Connection failed. Invalid token (500)
   

6. Node A: microk8s kubectl get no

   NAME      STATUS   ROLES    AGE     VERSION
   node-a    Ready    <none>   7m52s   v1.24.0-2+59bbb3530b6769
   

Things I tried:

• Pinging from node B to A works and vice-versa
• Using curl to contact API on node B from A works:

  root@node-b:~# curl --insecure -0 https://10.0.0.2:25000
  {"error":"Not found"}
  

• Downgrading to MicroK8s v1.23.6 on both nodes fixes the issue. Then both connect just fine.

Introspection Report

• Node A inspection-report-20220607_150244.tar.gz
• Node B inspection-report-20220607_150120.tar.gz

Can you suggest a fix?

I tested v1.23.6 on the same setup and they connected just fine. Some change between v1.23.6 & v1.24 affects the connectivity or validity of tokens.

Are you interested in contributing with a fix?

No

[KennethWussmann]

Hi everyone!

Glad to hear this gains traction and sorry for not getting back to @lferran after your initial response. While my command was intended like that and your suggestion didn’t made it work out of the box, but it made something click for me.

After re-testing the steps today (1.24/stable still), I found my issue.

I have to say I’m new to microk8s & k8s in general. The error message I got from node-b:

Connection failed. The hostname (node-b) of the joining node does not resolve to the IP "10.0.0.3". Refusing join (400).

sounded to me like node-b need to respond with it’s IPv4 when the hostname node-b is pinged. Actually, this error message seems to be generated from node-a’s perspective and only displayed on node-b, then it makes total sense. I was able to resolve the issues mentioned here by making sure every node has a /etc/hosts with entries for every node I expect to join the cluster, before joining them together. It’s indeed weird that this was not an issue on 1.23. That definitely contributed to me believing this is a bug in implementation.

Here are the entire steps of a clean working setup of 1.24/stable on Ubuntu 20.04.4 LTS:

Node A IPv4: 10.0.0.2 Node B IPv4: 10.0.0.3

1. All nodes: apt update
2. All nodes: apt install snapd
3. All nodes: snap install microk8s --classic --channel=1.24/stable
4. Node A: echo "10.0.0.2 node-a" >> /etc/hosts
5. Node A: echo "10.0.0.3 node-b" >> /etc/hosts
6. Node B: echo "10.0.0.2 node-a" >> /etc/hosts
7. Node B: echo "10.0.0.3 node-b" >> /etc/hosts
8. Node A: microk8s add-node
9. Node B: microk8s join 10.0.0.2:25000/token/token
10. Node A: microk8s kubectl get no:

NAME     STATUS   ROLES    AGE     VERSION
node-a   Ready    <none>   3m59s   v1.24.0-2+59bbb3530b6769
node-b   Ready    <none>   12s     v1.24.0-2+59bbb3530b6769

@neoaggelos I can’t really tell much about the inspection reports, sorry. I’ve not tinkered with them or something 🙈

I hope I was able to elaborate on this issue and happy to provide any more info if needed. To me it would be solved at this point, but I would appreciate a mention of this /etc/hosts requirement in the setup or troubleshooting documentation. Also because it appears this may affect others as well 🙏

[neoaggelos]

Hello, apologies for missing the issue.

@KennethWussmann I am looking at the inspection reports but I see no logs from the microk8s.daemon-cluster-agent service.

Note that if you attempt to microk8s join a node and it fails for whatever reason, the token will still be consumed (so you need to create a new one). Alternatively, you can create a token with a specified TTL like so:

microk8s add-node --token-ttl 3600

The token that will be printed by the command can then be used for the next 1 hour.

@TANISSIA2017 @ezequiels This also sounds interesting, but unfortunately I was unable to reproduce the issue with the 1.24 stable branch. Can you provide a few more details as to the situation you encounter? An inspection report would be very useful to have.

Thank you for reporting the issue, I would really like to get to the bottom of this.