i/o timeout in coredns pod in Canonical Microk8s

Explanation of the problem

The logs from the microk8s environment show that there are several timeout errors occurring while trying to query DNS servers 8.8.8.8 and 8.8.4.4 for the domain api.jujucharms.com. The log entries show that the requests are being made via UDP, with the source IP being 10.1.21.36 and the destination IP being either 8.8.8.8 or 8.8.4.4, with the destination port being 53.

The logs contain both error and info messages, where the error messages indicate a timeout occurred during the query, and the info messages indicate a successful query was made. Each log entry includes information such as the source and destination IP and port, the type of query being made (A or AAAA), and the time taken for the query to complete.

[ERROR] plugin/errors: 2 2577415620770853004.8780143945028532492. HINFO: read udp 10.1.21.36:44631->8.8.4.4:53: i/o timeout
[INFO] 10.1.21.38:33385 - 46047 "AAAA IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000755655s
[ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. AAAA: read udp 10.1.21.36:57766->8.8.8.8:53: i/o timeout
[INFO] 10.1.21.38:36178 - 22767 "A IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000677982s
[ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. A: read udp 10.1.21.36:44211->8.8.8.8:53: i/o timeout

Troubleshooting with the Lightrun Developer Observability Platform

Getting a sense of what’s actually happening inside a live application is a frustrating experience, one that relies mostly on querying and observing whatever logs were written during development.
Lightrun is a Developer Observability Platform, allowing developers to add telemetry to live applications in real-time, on-demand, and right from the IDE.

• Instantly add logs to, set metrics in, and take snapshots of live applications
• Insights delivered straight to your IDE or CLI
• Works where you do: dev, QA, staging, CI/CD, and production

Start for free today

Problem solution for i/o timeout in coredns pod in Canonical Microk8s

The issue of i/o timeout in coredns pod in Canonical Microk8s can occur due to several reasons, including network configuration and DNS resolution issues. The log messages present in the issue description are indicating a failure in resolving the domain api.jujucharms.com using the IP addresses 8.8.8.8 and 8.8.4.4.

The first step to resolve this issue is to verify the network configuration of the coredns pod. The following code block can be used to check the network settings of the coredns pod:

kubectl describe pod coredns-<hash> -n kube-system

If the network configuration is correct, the next step is to verify that the coredns pod can communicate with the DNS servers 8.8.8.8 and 8.8.4.4. The following code block can be used to test the DNS resolution using nslookup:

kubectl exec -it coredns-<hash> -n kube-system -- nslookup api.jujucharms.com 8.8.8.8
kubectl exec -it coredns-<hash> -n kube-system -- nslookup api.jujucharms.com 8.8.4.4

If the issue persists, it may be necessary to check the logs of the coredns pod and other network components to determine the root cause of the i/o timeout. The following code block can be used to check the logs of the coredns pod:

kubectl logs coredns-<hash> -n kube-system

Other popular problems with Canonical Microk8s

Problem: Error response from daemon: rpc error: code = Unknown desc = failed to dial gRPC: cannot connect to the Docker daemon at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json?all=1: dial unix /var/run/docker.sock: connect: no such file or directory

This error occurs when trying to interact with the Docker daemon through the Canonical Microk8s environment. It is caused by the missing Docker socket, which is necessary for communication between the Docker CLI and the Docker daemon.

Solution:

To resolve this issue, make sure that Docker is installed and running on the host machine. You can check the status of Docker using the following command:

systemctl status docker

If Docker is not running, start the service using the following command:

systemctl start docker

Problem: Error: failed to start container "coredns": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/snap/microk8s/common/var/lib/coredns/etcd\\\" to rootfs \\\"/var/lib/docker/overlay2/<ID>/merged\\\" at \\\"/var/lib/docker/overlay2/<ID>/merged/var/lib/coredns/etcd\\\" caused \\\"no such file or directory\\\"\"": unknown

This error occurs when the Canonical Microk8s CoreDNS pod fails to start. It is usually caused by missing or inaccessible directories within the Docker container file system.

Solution:

To resolve this issue, make sure that the directories specified in the error message exist and are accessible to the Docker daemon. If the directories are missing, create them and set the appropriate permissions. Additionally, restart the Docker daemon to apply the changes:

systemctl restart docker

Problem: Error: unable to upgrade connection: pod does not have a host network

This error occurs when trying to upgrade the connection to a Canonical Microk8s cluster from a remote machine. It is caused by the absence of a host network, which is necessary for communication between the remote machine and the cluster nodes.

Solution:

To resolve this issue, enable the host network on the Canonical Microk8s cluster using the following command:

microk8s.config > config.yaml

Edit the config.yaml file to include the following line:

network_mode: host

Finally, apply the changes using the following command:

microk8s.kubectl apply -f config.yaml

A brief introduction to Canonical Microk8s

Canonical Microk8s is a lightweight, fast, and secure distribution of Kubernetes that runs natively on Linux. It provides a single-node deployment of a Kubernetes cluster and is designed to simplify the deployment and management of microservices-based applications. With Microk8s, developers can deploy and manage Kubernetes applications without having to set up a full-fledged cluster, making it an ideal solution for development, testing, and edge computing.

Canonical Microk8s is a snap package, which makes it simple to install and upgrade. It features a simple, yet powerful, command-line interface that allows users to manage their microservices and containers. Microk8s includes popular Kubernetes add-ons such as Istio, Knative, and Prometheus, providing users with a complete suite of tools for building, deploying, and managing their applications. The microservices deployed on Microk8s can be easily managed and updated through the Kubernetes API, making it an ideal solution for DevOps teams looking to streamline their operations.

Most popular use cases for Canonical Microk8s

1. Container Orchestration: Canonical Microk8s provides a simple, lightweight solution for container orchestration, enabling users to manage and deploy containerized applications with ease. It integrates with the Kubernetes API, allowing users to use familiar tools and interfaces to manage their applications.
2. Development and Testing Environment: Canonical Microk8s is an ideal solution for developers and testers, providing a robust environment for building and testing containerized applications. With its lightweight design, Canonical Microk8s enables users to spin up a cluster on their local machine, eliminating the need for large, complex infrastructure and allowing for faster, more efficient development and testing processes.
3. Easy Deployment: Canonical Microk8s makes deployment of containerized applications easy and efficient. Its single-node deployment model means that users can get started with a minimal amount of resources, and its simple, straightforward installation process makes it accessible to a wide range of users, regardless of their technical expertise. A sample code block of deployment of a sample app using Microk8s can be like:

# Install Microk8s
snap install microk8s --classic

# Enable required add-ons
microk8s.enable dns
microk8s.enable storage

# Create a deployment
cat <<EOF | microk8s.kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.17
        ports:
        - containerPort: 80
EOF

# Check deployment status
microk8s.kubectl get pods