i/o timeout in coredns pod in Canonical Microk8s
Explanation of the problem
The logs from the microk8s environment show that there are several timeout errors occurring while trying to query DNS servers 220.127.116.11 and 18.104.22.168 for the domain api.jujucharms.com. The log entries show that the requests are being made via UDP, with the source IP being 10.1.21.36 and the destination IP being either 22.214.171.124 or 126.96.36.199, with the destination port being 53.
The logs contain both error and info messages, where the error messages indicate a timeout occurred during the query, and the info messages indicate a successful query was made. Each log entry includes information such as the source and destination IP and port, the type of query being made (A or AAAA), and the time taken for the query to complete.
[ERROR] plugin/errors: 2 2577415620770853004.8780143945028532492. HINFO: read udp 10.1.21.36:44631->188.8.131.52:53: i/o timeout [INFO] 10.1.21.38:33385 - 46047 "AAAA IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000755655s [ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. AAAA: read udp 10.1.21.36:57766->184.108.40.206:53: i/o timeout [INFO] 10.1.21.38:36178 - 22767 "A IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000677982s [ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. A: read udp 10.1.21.36:44211->220.127.116.11:53: i/o timeout
The logs clearly show that there is an issue with the queries being made to the DNS servers for the domain api.jujucharms.com. This issue needs to be addressed in order to resolve the errors and ensure successful queries.
Troubleshooting with the Lightrun Developer Observability Platform
Getting a sense of what’s actually happening inside a live application is a frustrating experience, one that relies mostly on querying and observing whatever logs were written during development.
Lightrun is a Developer Observability Platform, allowing developers to add telemetry to live applications in real-time, on-demand, and right from the IDE.
- Instantly add logs to, set metrics in, and take snapshots of live applications
- Insights delivered straight to your IDE or CLI
- Works where you do: dev, QA, staging, CI/CD, and production
Start for free today
Problem solution for i/o timeout in coredns pod in Canonical Microk8s
The issue of i/o timeout in coredns pod in Canonical Microk8s can occur due to several reasons, including network configuration and DNS resolution issues. The log messages present in the issue description are indicating a failure in resolving the domain
api.jujucharms.com using the IP addresses
The first step to resolve this issue is to verify the network configuration of the coredns pod. The following code block can be used to check the network settings of the coredns pod:
kubectl describe pod coredns-<hash> -n kube-system
If the network configuration is correct, the next step is to verify that the coredns pod can communicate with the DNS servers
18.104.22.168. The following code block can be used to test the DNS resolution using
kubectl exec -it coredns-<hash> -n kube-system -- nslookup api.jujucharms.com 22.214.171.124 kubectl exec -it coredns-<hash> -n kube-system -- nslookup api.jujucharms.com 126.96.36.199
If the issue persists, it may be necessary to check the logs of the coredns pod and other network components to determine the root cause of the i/o timeout. The following code block can be used to check the logs of the coredns pod:
kubectl logs coredns-<hash> -n kube-system
Other popular problems with Canonical Microk8s
Error response from daemon: rpc error: code = Unknown desc = failed to dial gRPC: cannot connect to the Docker daemon at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json?all=1: dial unix /var/run/docker.sock: connect: no such file or directory
This error occurs when trying to interact with the Docker daemon through the Canonical Microk8s environment. It is caused by the missing Docker socket, which is necessary for communication between the Docker CLI and the Docker daemon.
To resolve this issue, make sure that Docker is installed and running on the host machine. You can check the status of Docker using the following command:
systemctl status docker
If Docker is not running, start the service using the following command:
systemctl start docker
Error: failed to start container "coredns": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/snap/microk8s/common/var/lib/coredns/etcd\\\" to rootfs \\\"/var/lib/docker/overlay2/<ID>/merged\\\" at \\\"/var/lib/docker/overlay2/<ID>/merged/var/lib/coredns/etcd\\\" caused \\\"no such file or directory\\\"\"": unknown
This error occurs when the Canonical Microk8s CoreDNS pod fails to start. It is usually caused by missing or inaccessible directories within the Docker container file system.
To resolve this issue, make sure that the directories specified in the error message exist and are accessible to the Docker daemon. If the directories are missing, create them and set the appropriate permissions. Additionally, restart the Docker daemon to apply the changes:
systemctl restart docker
Error: unable to upgrade connection: pod does not have a host network
This error occurs when trying to upgrade the connection to a Canonical Microk8s cluster from a remote machine. It is caused by the absence of a host network, which is necessary for communication between the remote machine and the cluster nodes.
To resolve this issue, enable the host network on the Canonical Microk8s cluster using the following command:
microk8s.config > config.yaml
Edit the config.yaml file to include the following line:
Finally, apply the changes using the following command:
microk8s.kubectl apply -f config.yaml
These are three of the most common problems encountered when using Canonical Microk8s. By following the solutions outlined above, you should be able to resolve these issues and ensure a smooth experience when working with your Canonical Microk8s cluster.
A brief introduction to Canonical Microk8s
Canonical Microk8s is a lightweight, fast, and secure distribution of Kubernetes that runs natively on Linux. It provides a single-node deployment of a Kubernetes cluster and is designed to simplify the deployment and management of microservices-based applications. With Microk8s, developers can deploy and manage Kubernetes applications without having to set up a full-fledged cluster, making it an ideal solution for development, testing, and edge computing.
Canonical Microk8s is a snap package, which makes it simple to install and upgrade. It features a simple, yet powerful, command-line interface that allows users to manage their microservices and containers. Microk8s includes popular Kubernetes add-ons such as Istio, Knative, and Prometheus, providing users with a complete suite of tools for building, deploying, and managing their applications. The microservices deployed on Microk8s can be easily managed and updated through the Kubernetes API, making it an ideal solution for DevOps teams looking to streamline their operations.
Most popular use cases for Canonical Microk8s
- Container Orchestration: Canonical Microk8s provides a simple, lightweight solution for container orchestration, enabling users to manage and deploy containerized applications with ease. It integrates with the Kubernetes API, allowing users to use familiar tools and interfaces to manage their applications.
- Development and Testing Environment: Canonical Microk8s is an ideal solution for developers and testers, providing a robust environment for building and testing containerized applications. With its lightweight design, Canonical Microk8s enables users to spin up a cluster on their local machine, eliminating the need for large, complex infrastructure and allowing for faster, more efficient development and testing processes.
- Easy Deployment: Canonical Microk8s makes deployment of containerized applications easy and efficient. Its single-node deployment model means that users can get started with a minimal amount of resources, and its simple, straightforward installation process makes it accessible to a wide range of users, regardless of their technical expertise. A sample code block of deployment of a sample app using Microk8s can be like:
# Install Microk8s snap install microk8s --classic # Enable required add-ons microk8s.enable dns microk8s.enable storage # Create a deployment cat <<EOF | microk8s.kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.17 ports: - containerPort: 80 EOF # Check deployment status microk8s.kubectl get pods
It’s Really not that Complicated.
You can actually understand what’s going on inside your live applications. It’s a registration form away.