Adding node to existing cluster with a shared subnet fails

A little overview of the structure of my cluster:

• Currently contains 3 nodes, each one of these nodes has an interface in both the subnets 192.168.2.0/24 and 172.16.1.0/28. The 172 subnet is a completely isolated 10G network existing between the existing 3 nodes (no connection to 4th node)
• These 3 nodes were joined together on the 172 subnet.
• In the output of microk8s status they are referred to by their IPs in the 172 subnet
• The 4th node does not have a 172 interface, only a 192 interface
• The 4th node is unable to connect using microk8s join $IP_AND_TOKEN_OF_MASTER_NODE_192_ADDRESS microk8s inspect tarball on failed 4th node I raised this issue in slack and @balchua says it appears that dqlite is trying to connect to the other nodes using their 172 interfaces which fails as that route doesn’t exist

[khumps]

To sum up my conversation with @ktsakalozos, this is expected behavior as dqlite does not support inter-subnet communication within it’s cluster. As a feature request based on that knowledge, microk8s add-node should not show any interfaces that it can not properly initiate a connection over (all interfaces work on a fresh install but as soon as a node is added, the interface is locked)

[balchua]

Sorry my response wasn’t very clear. What i mean is that if in the case that all nodes have access to each other’s subnets, then one can use any of those join tokens printed. But yes if a node like in your case do not belong to that subnet then it is a good option.