Inaccessible pods on other nodes for high availability cluster

I made a 3 node cluster on ec2, and wanted to just launch a generic application to make sure everything is accessible. I created a microk8s environment on each machine, and got them to add to a HA cluster. When I tried to launch the microbot deployment on ubuntu tutorial, each machine could only access its own pod.

When running microk8s kubectl get pods -o wide, I get the following:

NAME                        READY   STATUS    RESTARTS   AGE    IP            NODE               NOMINATED NODE   READINESS GATES
microbot-5f5499d479-ngz56   1/1     Running   0          179m   10.1.94.76    ip-172-31-18-128   <none>           <none>
microbot-5f5499d479-nkctv   1/1     Running   1          175m   10.1.162.72   ip-172-31-21-37    <none>           <none>
microbot-5f5499d479-zkjcn   1/1     Running   1          175m   10.1.162.73   ip-172-31-21-37    <none>           <none>

This is with the deployment scaled to 3.

If I curl on the machine ending in 128, I get a 1/3 chance in hitting itself, and 2/3 with the one ending in 37. On my other machine, it always hangs because it is attempting to access one of the other two machines and does not seem to be able to.

ufw is disabled, and I’ve tried running

sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent

on each machine, to no avail. I can ping them on other services fine. I enabled ingress w/o a service, and each one shoots me a 404 error, so it can clearly route.

I’ve attached the inspection logs. inspection-report-20210112_132731.tar.gz

[rockaut]

I also had problems on my Ubuntu Cluster, for me it turned out to be problems with net_bridge. So I had to enable the modules and sysctl.

Added to /etc/modules-load.d/modules.conf:

overlay
br_netfilter
bridge

and to /etc/sysctl.conf:

net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1

Rebooted and all worked.

I previously had docker already installed and uninstalled and also played around with cni and podman - so it might got crushed somewhere with those fiddlings.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.