Cannot join over ipv6

It seems the microk8s join script doesn’t handle ipv6 addresses at all. My nodes use an ipv6 overlay network to communicate and cannot reach each other over ipv4. Maybe adding a -6 flag is a good solution, or potentially detecting the ipv6 format.

ubuntu@mm-cluster-stephen-do:~$ microk8s join [201:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]:25000/xxxxxxxxxxx
Contacting cluster at [201
Traceback (most recent call last):
  File "/snap/microk8s/1791/scripts/cluster/join.py", line 967, in <module>
    join_dqlite(connection_parts)
  File "/snap/microk8s/1791/scripts/cluster/join.py", line 871, in join_dqlite
    info = get_connection_info(master_ip, master_port, token, cluster_type="dqlite")
  File "/snap/microk8s/1791/scripts/cluster/join.py", line 70, in get_connection_info
    verify=False,
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/requests/api.py", line 107, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/requests/sessions.py", line 466, in request
    prep = self.prepare_request(req)
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/requests/sessions.py", line 400, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/requests/models.py", line 293, in prepare
    self.prepare_url(url, params)
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/requests/models.py", line 345, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/snap/microk8s/1791/usr/lib/python3/dist-packages/urllib3/util/url.py", line 184, in parse_url
    host, url = url.split(']', 1)
ValueError: not enough values to unpack (expected 2, got 1)

I tried to fix it with this patch:

diff --git a/scripts/cluster/join.py b/scripts/cluster/join.py
index 9ba826d..269bc9a 100755
--- a/scripts/cluster/join.py
+++ b/scripts/cluster/join.py
@@ -64,8 +64,11 @@ def get_connection_info(master_ip, master_port, token, callback_token=None, clus
 
         # TODO: enable ssl verification
         try:
+            format_string = "https://{}:{}/{}/join"
+            if master_ip.find(":"):
+                format_string = "https://[{}]:{}/{}/join"
             connection_info = requests.post(
-                "https://{}:{}/{}/join".format(master_ip, master_port, CLUSTER_API_V2),
+                format_string.format(master_ip, master_port, CLUSTER_API_V2),
                 json=req_data,
                 verify=False,
             )  # type: requests.models.Response
@@ -83,8 +86,11 @@ def get_connection_info(master_ip, master_port, token, callback_token=None, clus
 
         # TODO: enable ssl verification
         try:
+            format_string = "https://{}:{}/{}/join"
+            if master_ip.find(":"):
+                format_string = "https://[{}]:{}/{}/join"
             connection_info = requests.post(
-                "https://{}:{}/{}/join".format(master_ip, master_port, CLUSTER_API),
+                format_string.format(master_ip, master_port, CLUSTER_API),
                 json=req_data,
                 verify=False,
             )
@@ -864,8 +870,8 @@ def join_dqlite(connection_parts):
     """
     token = connection_parts[1]
     master_ep = connection_parts[0].split(":")
-    master_ip = master_ep[0]
-    master_port = master_ep[1]
+    master_ip = ":".join(master_ep[:-1])
+    master_port = master_ep[-1]
 
     print("Contacting cluster at {}".format(master_ip))
     info = get_connection_info(master_ip, master_port, token, cluster_type="dqlite")

But I just realized that it seems whatever is on port 25000 is only on the lan ipv4 address of my first node. Going off this documentation and port scanning both the v4 and v6 address, it seems that out of the box, port 25000 is the only one that is v4 only for some reason. Maybe it will work once that’s fixed.

[stephen304]

Success! At least it looks like I’ve managed to get the nodes to cluster over ipv6. I had to edit:

--- a/scripts/cluster/join.py
+++ b/scripts/cluster/join.pyq
@@ -820,7 +836,7 @@ def update_dqlite(cluster_cert, cluster_key, voters, host):
     if 'Address' in data:
         port = data['Address'].split(':')[1]
 
-    init_data = {'Cluster': voters, 'Address': "{}:{}".format(host, port)}
+    init_data = {'Cluster': voters, 'Address': "[{}]:{}".format(host, port)}
     with open("{}/init.yaml".format(cluster_dir), 'w') as f:
         yaml.dump(init_data, f)

~$ kubectl get nodes
NAME                    STATUS   ROLES    AGE   VERSION
stephen-1    Ready    <none>   44m   v1.19.3-34+b9e8e732a07cb6
stephen-2   Ready    <none>   40m   v1.19.3-34+a56971609ff35a

Checking the health of the cluster shows that only calico-node on the new node doesn’t start. The logs show this:

2020-11-23 22:50:07.407 [WARNING][8] startup.go 675: Unable to auto-detect IPv4 address by connecting to 200:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:ee79: dial udp4: address 200:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:ee79: no suitable address found
2020-11-23 22:50:07.408 [WARNING][8] startup.go 438: Couldn't autodetect an IPv4 address. If auto-detecting, choose a different autodetection method. Otherwise provide an explicit address.
2020-11-23 22:50:07.408 [INFO][8] startup.go 244: Clearing out-of-date IPv4 address from this node IP=""
2020-11-23 22:50:07.477 [WARNING][8] startup.go 1187: Terminating
Calico node failed to start

Editing the pod yml shows:

    - name: FELIX_IPV6SUPPORT
      value: "false"

It doesn’t seem to let me save any changes to that, but hopefully after that it works completely.

Edit: Also I believe this documentation may be useful to solve this: https://docs.projectcalico.org/networking/ipv6#enabling-ipv6-support-in-calico

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.