HTTP 500 Internal Server Error - RPC error - currencyservice

I am fairly new to Kubernetes, therefore having a hard time understanding what is exactly the error and how to resolve it. Thanks in advance!

Describe the bug

To Reproduce

Simply cloned the repository and ran: kubectl apply -f ./release/kubernetes-manifests.yaml Then tried to open the frontend page in browser, and the above error shows.

The scripts I’m using to construct my cluster’s nodes (master & worker): (after that I’m joining them)

Master:

#! /bin/bash

sudo apt-get update -y
sudo swapoff -a
sudo modprobe br_netfilter
sudo echo "net.bridge.bridge-nf-call-iptables = 1" | sudo tee -a /etc/sysctl.conf
sudo echo "1" | sudo tee -a /proc/sys/net/ipv4/ip_forward
sudo sysctl -p /etc/sysctl.conf
kube
sudo apt update -y && sudo apt upgrade -y

OS=xUbuntu_20.04
CRIO_VERSION=1.23
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$CRIO_VERSION/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION.list

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION/$OS/Release.key | sudo apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add -


sudo apt update -y
sudo apt install cri-o cri-o-runc -y


sudo systemctl enable crio.service
sudo systemctl start crio.service


sudo apt install cri-tools


sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl restart crio
sudo kubeadm init --cri-socket unix:///var/run/crio/crio.sock
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl get cs
kubectl cluster-info
kubectl get pods -n kube-system

mv $HOME/.kube $HOME/.kube.bak
mkdir $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
sudo systemctl restart crio
sudo service kubelet restart

Workers:

#! /bin/bash

sudo apt-get update -y
sudo swapoff -a
sudo modprobe br_netfilter
sudo echo "net.bridge.bridge-nf-call-iptables = 1" | sudo tee -a /etc/sysctl.conf
sudo echo "1" | sudo tee -a /proc/sys/net/ipv4/ip_forward
sudo sysctl -p /etc/sysctl.conf
kube
sudo apt update -y && sudo apt upgrade -y

OS=xUbuntu_20.04
CRIO_VERSION=1.23
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$CRIO_VERSION/$OS/ /"|sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION.list

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$CRIO_VERSION/$OS/Release.key | sudo apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | sudo apt-key add -


sudo apt update -y
sudo apt install cri-o cri-o-runc -y


sudo systemctl enable crio.service
sudo systemctl start crio.service


sudo apt install cri-tools


sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl restart crio

Logs

currencyservice pod logs:

Name:         currencyservice-694d677bfb-k9kcc
Namespace:    default
Priority:     0
Node:         ip-172-31-3-139/172.31.3.139
Start Time:   Sun, 21 Aug 2022 13:44:38 +0000
Labels:       app=currencyservice
              pod-template-hash=694d677bfb
Annotations:  cni.projectcalico.org/containerID: b62b65134833f0f32c530f0768124ad5075fecdff4bf05d114cdf4fe77560e2b
              cni.projectcalico.org/podIP: 172.16.110.156/32
              cni.projectcalico.org/podIPs: 172.16.110.156/32
Status:       Running
IP:           172.16.110.156
IPs:
  IP:           172.16.110.156
Controlled By:  ReplicaSet/currencyservice-694d677bfb
Containers:
  server:
    Container ID:   cri-o://53af05e04b7ccc0657d8dc461c9cc938dcfbcac1ebab59c5614ec22f36118217
    Image:          gcr.io/google-samples/microservices-demo/currencyservice:v0.3.9
    Image ID:       gcr.io/google-samples/microservices-demo/currencyservice@sha256:c52b683e393b43fc12ffa3d356b7f9b8aec68805e7cb1b09e689450e1a0d7159
    Port:           7000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 21 Aug 2022 13:44:39 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  128Mi
    Requests:
      cpu:      100m
      memory:   64Mi
    Liveness:   exec [/bin/grpc_health_probe -addr=:7000] delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/grpc_health_probe -addr=:7000] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      PORT:              7000
      DISABLE_TRACING:   1
      DISABLE_PROFILER:  1
      DISABLE_DEBUGGER:  1
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6fcvj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-6fcvj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  26s   default-scheduler  Successfully assigned default/currencyservice-694d677bfb-k9kcc to ip-172-31-3-139
  Normal   Pulled     25s   kubelet            Container image "gcr.io/google-samples/microservices-demo/currencyservice:v0.3.9" already present on machine
  Normal   Created    25s   kubelet            Created container server
  Normal   Started    25s   kubelet            Started container server
  Warning  Unhealthy  23s   kubelet            Readiness probe failed: timeout: failed to connect service ":7000" within 1s

Environment

• AWS EC2 instances, all with the same specs: Ubuntu 20.04, 4 CPU, 16GB memory, 100GB storage memory
• I built the cluster with kubeadm, using CRI-O engine.

Exposure

Deploying the application on my local machine with minikube seems to work just fine, which means the problem is probably in my k8s cluster.

[scomri]

In the end I managed to deploy the application successfully using Istio mesh. Thank you anyways for your comment!!

[NimJay]

Hi @scomri,

Thanks for creating such a clear (detailed) issue! 😃

1. If you suspect that the issue is your environment, I suggest trying to deploy an app simpler than Online Boutique. Try deploying this “Hello, world!” app. You will need to deploy this Deployment and this Service using kubectl apply -f <path-to-file>. The Service is of type LoadBalancer so it will get an external IP address assigned to it which you can check using kubectl get service.
2. How are all the Pods and Services doing? What’s the output of kubectl get pods and kubectl get services?
3. Is there a reason you’re not using a managed Kubernetes service like Google Kubernetes Engine? You’re less likely to run into environment-related issues if you use a managed service.

Hope my thoughts above help. 🤞 Good luck!