Network problems with Calico when using HA mode

I’m using the new HA mode in 1.19 with a 3-node cluster on CentOS 8, and I’m running into some network problems which I think are related to Calico, but I don’t have enough experience to diagnose this on my own. This is a greenfield HA deployment on top of which I have installed rook-ceph. Several pods are reporting errors contacting the apiserver.

All of these commands are run on the first node in the cluster, kube01

Nodes are all OK

[jonathan@kube01 ~]$ kubectl get nodes
NAME                         STATUS   ROLES    AGE    VERSION
kube01.jonathangazeley.com   Ready    <none>   6d8h   v1.19.0-34+09a4aa08bb9e93
kube03.jonathangazeley.com   Ready    <none>   6d8h   v1.19.0-34+09a4aa08bb9e93
kube02.jonathangazeley.com   Ready    <none>   6d8h   v1.19.0-34+09a4aa08bb9e93

The cluster has some success, I am able to schedule various pods. Should there be an apiserver here?

[jonathan@kube01 ~]$ kubectl get pods --all-namespaces -o wide 
NAMESPACE     NAME                                                              READY   STATUS              RESTARTS   AGE     IP             NODE                         NOMINATED NODE   READINESS GATES
rook-ceph     csi-rbdplugin-nw9p8                                               0/3     ContainerCreating   0          2d7h    192.168.0.42   kube02.jonathangazeley.com   <none>           <none>
rook-ceph     csi-rbdplugin-5znbs                                               0/3     ContainerCreating   0          2d7h    192.168.0.41   kube01.jonathangazeley.com   <none>           <none>
ingress       nginx-ingress-microk8s-controller-mh7lm                           1/1     Running             10         6d7h    192.168.0.43   kube03.jonathangazeley.com   <none>           <none>
rook-ceph     csi-rbdplugin-kksjh                                               0/3     ContainerCreating   0          2d7h    192.168.0.43   kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-crashcollector-kube03.jonathangazeley.com-fb6847qqwp6   1/1     Running             1          3d12h   10.1.252.72    kube03.jonathangazeley.com   <none>           <none>
kube-system   calico-node-bcw4c                                                 1/1     Running             7          6d8h    192.168.0.43   kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-discover-zdjth                                               1/1     Running             2          2d7h    10.1.252.75    kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-osd-0-5cd95b479-plcfb                                   1/1     Running             1          3d12h   10.1.252.78    kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-mon-a-776944fc54-5qrqz                                  1/1     Running             1          3d12h   10.1.252.79    kube03.jonathangazeley.com   <none>           <none>
kube-system   kubernetes-dashboard-7ffd448895-lbgfs                             1/1     Running             23         3d3h    10.1.252.84    kube03.jonathangazeley.com   <none>           <none>
kube-system   dashboard-metrics-scraper-6c4568dc68-5pdbr                        1/1     Running             1          3d3h    10.1.252.80    kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-operator-775d4b6c5f-rjpm8                               1/1     Running             0          2d1h    10.1.252.76    kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-mgr-a-7c966954fc-v5fwr                                  1/1     Running             0          2d1h    10.1.252.98    kube03.jonathangazeley.com   <none>           <none>
ingress       nginx-ingress-microk8s-controller-485st                           1/1     Running             4          6d7h    192.168.0.41   kube01.jonathangazeley.com   <none>           <none>
kube-system   calico-node-p4vgc                                                 1/1     Running             13         6d8h    192.168.0.41   kube01.jonathangazeley.com   <none>           <none>
rook-ceph     rook-discover-4m6zj                                               1/1     Running             0          2d7h    10.1.58.44     kube01.jonathangazeley.com   <none>           <none>
kube-system   calico-kube-controllers-555fc8cc5c-zdlqq                          1/1     Running             5          3d3h    10.1.252.81    kube03.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-crashcollector-kube01.jonathangazeley.com-77ccfb78pz4   1/1     Running             0          2d1h    10.1.58.50     kube01.jonathangazeley.com   <none>           <none>
kube-system   coredns-588fd544bf-cl85v                                          1/1     Running             0          2d      10.1.58.51     kube01.jonathangazeley.com   <none>           <none>
rook-ceph     csi-rbdplugin-provisioner-77459cc496-vwzjn                        6/6     Running             0          2d      10.1.58.55     kube01.jonathangazeley.com   <none>           <none>
kube-system   metrics-server-8bbfb4bdb-vwh4l                                    1/1     Running             2          2d      10.1.58.59     kube01.jonathangazeley.com   <none>           <none>
rook-ceph     csi-cephfsplugin-provisioner-7468b6bf56-wm7mg                     6/6     Running             1          2d      10.1.58.56     kube01.jonathangazeley.com   <none>           <none>
rook-ceph     csi-rbdplugin-provisioner-77459cc496-vfhpl                        6/6     Running             50         2d1h    10.1.252.107   kube03.jonathangazeley.com   <none>           <none>
rook-ceph     csi-cephfsplugin-provisioner-7468b6bf56-vjp2l                     6/6     Running             50         2d1h    10.1.252.96    kube03.jonathangazeley.com   <none>           <none>
kube-system   hostpath-provisioner-5c65fbdb4f-2dbcp                             1/1     Running             1          27h     10.1.58.10     kube01.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-osd-prepare-kube03.jonathangazeley.com-s5clx            0/1     Completed           3          4h33m   10.1.252.77    kube03.jonathangazeley.com   <none>           <none>
ingress       nginx-ingress-microk8s-controller-58mk8                           1/1     Running             4          6d7h    192.168.0.42   kube02.jonathangazeley.com   <none>           <none>
rook-ceph     rook-discover-p75pv                                               1/1     Running             6          2d7h    10.1.111.103   kube02.jonathangazeley.com   <none>           <none>
kube-system   calico-node-phmh8                                                 1/1     Running             17         6d8h    192.168.0.42   kube02.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-mon-h-5cdff9696f-td4zm                                  1/1     Running             0          6h17m   10.1.111.106   kube02.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-crashcollector-kube02.jonathangazeley.com-66f8b5bk49z   1/1     Running             0          6h17m   10.1.111.67    kube02.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-mon-g-577484d5db-pnm66                                  1/1     Running             0          8m5s    10.1.58.16     kube01.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-detect-version-v9c48                                    0/1     Terminating         26         153m    10.1.111.74    kube02.jonathangazeley.com   <none>           <none>
rook-ceph     rook-ceph-mon-f-544cdf8cbb-77x6j                                  0/1     Init:0/2            0          30s     <none>         kube01.jonathangazeley.com   <none>           <none>

Error messages from different nodes about apiserver:

[jonathan@kube01 ~]$ kubectl describe pods --all-namespaces | grep api
      Message:      Get https://10.152.183.1:443/api/v1/namespaces/rook-ceph/configmaps/local-device-kube03.jonathangazeley.com: dial tcp 10.152.183.1:443: connect: no route to host
      Message:      could not get the node for topology labels: could not find node "kube03.jonathangazeley.com" by name: Get https://10.152.183.1:443/api/v1/nodes/kube03.jonathangazeley.com: dial tcp 10.152.183.1:443: connect: no route to host
      Message:      Get https://10.152.183.1:443/api/v1/namespaces/rook-ceph/configmaps/local-device-kube02.jonathangazeley.com: dial tcp 10.152.183.1:443: connect: no route to host
      Message:      failed to save command output to ConfigMap. failed to determine if ConfigMap rook-ceph-detect-version is preexisting. Get https://10.152.183.1:443/api/v1/namespaces/rook-ceph/configmaps/rook-ceph-detect-version: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  103s  kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bba2d3d770de50aac8fbf3310267da64e11ca7aa62a40ee5d202028034d28c41": error getting ClusterInformation: Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  89s   kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d1198a671444b1ebe2b7ffff76278d00461c09bb108754ba05fa00079a3bbaae": Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/ippools: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  77s   kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "538d26aa8e0e9bbc121e74085f5f3b209d5d5706ebed568dd6c770197d0d6897": error getting ClusterInformation: Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  62s   kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "209cb8ef7358f038877568d21e99cfcc2c32542f914a7276c32b17a13573c6a1": error getting ClusterInformation: Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  50s   kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8a483b08f14afd7aafdc30fd55109c0ecc1ab4346a98267d1c022f11e6aa5f0a": error getting ClusterInformation: Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  37s   kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2ee92a4fb0ac594b68320d80c8488df51c91d37994550c2edb3fa446baadbd98": error getting ClusterInformation: Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  24s   kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6a00be29479ed0521d3ec3a83e29cceecde8d8d0d0ef75166c70280daa455b56": error getting ClusterInformation: Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.152.183.1:443: connect: no route to host
  Warning  FailedCreatePodSandBox  8s    kubelet, kube01.jonathangazeley.com  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2e251f1bd6f5e6ff89ed20f2adf76251a53923a86e7ec6a899069e9dff0d392c": Get https://[10.152.183.1]:443/apis/crd.projectcalico.org/v1/ippools: dial tcp 10.152.183.1:443: connect: no route to host

My firewall config:

[jonathan@kube01 ~]$ sudo firewall-cmd --list-all --zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 10255/tcp 25000/tcp 19001/tcp 8285/udp 8472/udp
  protocols: 
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	
[jonathan@kube01 ~]$ sudo firewall-cmd --list-all --zone=trusted
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: vxlan.calico
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

Finally, the microk8s inspection reports:

kube01-inspection-report-20200908_190417.tar.gz kube02-inspection-report-20200908_184446.tar.gz kube03-inspection-report-20200908_184559.tar.gz

[djjudas21]

@mistrymanan I found that as well as a list of static ports, you also need to add the calico virtual interface to the trusted zone. This is what I had to do on Fedora/CentOS. It took a bit of experimenting to compile this list, which includes Kubernetes ports but also some Calico ports.

sudo firewall-cmd --permanent --add-port=6443/tcp			# Kube API server
sudo firewall-cmd --permanent --add-port=2379-2380/tcp		# etcd
sudo firewall-cmd --permanent --add-port=10250/tcp			# kubelet
sudo firewall-cmd --permanent --add-port=10251/tcp			# kube-scheduler
sudo firewall-cmd --permanent --add-port=10252/tcp			# kube-controller-manager
sudo firewall-cmd --permanent --add-port=10255/tcp
sudo firewall-cmd --permanent --add-port=25000/tcp			# microk8s cluster
sudo firewall-cmd --permanent --add-port=19001/tcp
sudo firewall-cmd --permanent --add-port=4789/udp			# Calico with VXLAN 
sudo firewall-cmd --permanent --add-port=5473/tcp			# Calico with Typha
sudo firewall-cmd --permanent --add-port={8285,8472}/udp	# Flannel
sudo firewall-cmd --zone=trusted --add-interface=vxlan.calico --permanent
sudo firewall-cmd --permanent --add-port=30000-32767/tcp	# NodePorts on control plane IP
sudo firewall-cmd --reload

[djjudas21]

Hi @ktsakalozos

Thanks for this. I disabled my firewall and tried with latest/edge at revision 1677 and it seems to work for me now 🙂