Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

High CPU usage when standalone

See original GitHub issue

What did you do?

I started microK8s cluster

What did you expect to happen?

I expect CPU usage to be around 5%, but it oscillates between 40-80% CPU

top -o cpu

PID    COMMAND      %CPU TIME     #TH   #WQ  #PORT MEM    PURG   CMPRS  PGRP  PPID  STATE    BOOSTS           %CPU_ME %CPU_OTHRS UID  FAULTS     COW      MSGSENT
71452  hyperkit     72.0 12:29.26 12    1    39    4485M  0B     391M   60028 60028 sleeping *0[1]            0.00000 0.00000    0    1226131+   460      165

Which OS & Architecture?

MacOS 10.14.6 (18G4032)
x86

I don’t know how to obtain the tarball created by microk8s inspect from the VM

Issue Analytics

State:
Created 3 years ago
Reactions:7
Comments:25 (3 by maintainers)

Top GitHub Comments

9reactions

aleric323commented, May 30, 2021

Observing this same problem on Ubuntu 20.04 (running on a raspberry PI 4) 😦

6reactions

devZer0commented, Mar 23, 2021

yes, i also have high cpu load on idle microk8s.

i did nothing but “snap install microk8s --classic” and kube-apiserver is constantly >10% cpu load.

i see this in the logs every 2 seconds:

# tail -f syslog|grep nil
Mar 23 17:10:02 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:02.977871923Z" level=info msg="Exec process \"d678497a8047ebe4d608e6307b763326577d162dcfa247c4f9822f5b2af6248c\" exits with exit code 0 and error <nil>"
Mar 23 17:10:06 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:06.582351200Z" level=info msg="Exec process \"89b68a58097f7ca0152b20a694c768ebcfe7915c88ed872d8aa5a5d1bc189263\" exits with exit code 0 and error <nil>"
Mar 23 17:10:08 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:08.060196886Z" level=info msg="Exec process \"fea78c5c20ee48f68e61f9ba48243e3b2fe2ae9c72741397ff67e69fb6f9d990\" exits with exit code 0 and error <nil>"
Mar 23 17:10:12 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:12.977476476Z" level=info msg="Exec process \"1931b176410e67953ff600adab55f830d84dd481c9d8d7bb945547640acb3f4a\" exits with exit code 0 and error <nil>"
Mar 23 17:10:16 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:16.593262433Z" level=info msg="Exec process \"3f769b56e281b4d8cd5d542325cdb5c8e5da05db99c4a17bef8b7783b4501d69\" exits with exit code 0 and error <nil>"
Mar 23 17:10:18 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:18.053523348Z" level=info msg="Exec process \"5308879d4ecf563101738df82b56435b784a7fd6ab0348c3792cc33757112cfe\" exits with exit code 0 and error <nil>"
Mar 23 17:10:22 microk8s microk8s.daemon-containerd[4637]: time="2021-03-23T17:10:22.973033474Z" level=info msg="Exec process \"d93ea8a5dc9c7e5ec22bf2f79afac63f636a622951d71d6c1f83c5861f878aac\" exits with exit code 0 and error <nil>"

journalctrl telling

Mar 23 17:24:59 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:24:59.939428059Z" level=info msg="ExecSync for \"9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78\" with command [/bin/calico-node -felix-ready] and timeout 1 (s)"
Mar 23 17:25:00 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:00.064402715Z" level=info msg="Finish piping \"stderr\" of container exec \"764a58a0b354ce851f0a9e04485a3f2182c04daf947d9a3b141dc1aa63aacd30\""
Mar 23 17:25:00 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:00.064553108Z" level=info msg="Finish piping \"stdout\" of container exec \"764a58a0b354ce851f0a9e04485a3f2182c04daf947d9a3b141dc1aa63aacd30\""
Mar 23 17:25:00 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:00.066784612Z" level=info msg="Exec process \"764a58a0b354ce851f0a9e04485a3f2182c04daf947d9a3b141dc1aa63aacd30\" exits with exit code 0 and error <nil>"
Mar 23 17:25:00 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:00.068672127Z" level=info msg="ExecSync for \"9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78\" returns with exit code 0"
Mar 23 17:25:02 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:02.040082285Z" level=info msg="ExecSync for \"185837912bb662cf9274aa48c2cbecce5a9253a37feb90449bdcbb3b7df7a450\" with command [/usr/bin/check-status -r] and timeout 1 (s)"
Mar 23 17:25:02 microk8s systemd[1212]: run-containerd-runc-k8s.io-185837912bb662cf9274aa48c2cbecce5a9253a37feb90449bdcbb3b7df7a450-runc.ooflp5.mount: Succeeded.
Mar 23 17:25:02 microk8s systemd[1]: run-containerd-runc-k8s.io-185837912bb662cf9274aa48c2cbecce5a9253a37feb90449bdcbb3b7df7a450-runc.ooflp5.mount: Succeeded.
Mar 23 17:25:02 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:02.122983177Z" level=info msg="Exec process \"83056627d7576e69a1b89b877d8dcc0c3182bde26ac87234c1ab4e16f655d742\" exits with exit code 0 and error <nil>"
Mar 23 17:25:02 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:02.123138847Z" level=info msg="Finish piping \"stdout\" of container exec \"83056627d7576e69a1b89b877d8dcc0c3182bde26ac87234c1ab4e16f655d742\""
Mar 23 17:25:02 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:02.123229848Z" level=info msg="Finish piping \"stderr\" of container exec \"83056627d7576e69a1b89b877d8dcc0c3182bde26ac87234c1ab4e16f655d742\""
Mar 23 17:25:02 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:02.129618433Z" level=info msg="ExecSync for \"185837912bb662cf9274aa48c2cbecce5a9253a37feb90449bdcbb3b7df7a450\" returns with exit code 0"
Mar 23 17:25:05 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:05.797896549Z" level=info msg="ExecSync for \"9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78\" with command [/bin/calico-node -felix-live] and timeout 1 (s)"
Mar 23 17:25:05 microk8s systemd[1212]: run-containerd-runc-k8s.io-9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78-runc.jtzht5.mount: Succeeded.
Mar 23 17:25:05 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:05.911052091Z" level=info msg="Finish piping \"stderr\" of container exec \"30c693442011267aa02d6fc4625c4e1198a7c0dfcf21b9e4ff18988a4304571c\""
Mar 23 17:25:05 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:05.911178571Z" level=info msg="Finish piping \"stdout\" of container exec \"30c693442011267aa02d6fc4625c4e1198a7c0dfcf21b9e4ff18988a4304571c\""
Mar 23 17:25:05 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:05.914663745Z" level=info msg="Exec process \"30c693442011267aa02d6fc4625c4e1198a7c0dfcf21b9e4ff18988a4304571c\" exits with exit code 0 and error <nil>"
Mar 23 17:25:05 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:05.916413580Z" level=info msg="ExecSync for \"9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78\" returns with exit code 0"
Mar 23 17:25:09 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:09.939588119Z" level=info msg="ExecSync for \"9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78\" with command [/bin/calico-node -felix-ready] and timeout 1 (s)"
Mar 23 17:25:10 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:10.052671203Z" level=info msg="Finish piping \"stdout\" of container exec \"3207dc67aac9c6a670bbcbf9b27583be3747e9b9df11be6fe54271e41ec17da8\""
Mar 23 17:25:10 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:10.052813029Z" level=info msg="Finish piping \"stderr\" of container exec \"3207dc67aac9c6a670bbcbf9b27583be3747e9b9df11be6fe54271e41ec17da8\""
Mar 23 17:25:10 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:10.054210554Z" level=info msg="Exec process \"3207dc67aac9c6a670bbcbf9b27583be3747e9b9df11be6fe54271e41ec17da8\" exits with exit code 0 and error <nil>"
Mar 23 17:25:10 microk8s microk8s.daemon-containerd[21071]: time="2021-03-23T17:25:10.056127991Z" level=info msg="ExecSync for \"9f7726d2f2e77bedc3af2636cd5d141760de0a18bb2b5e7202649bb067238c78\" returns with exit code 0"
^C

there is tons of futex() syscalls

# strace -c -f -p $(pidof kube-apiserver)
strace: Process 4724 attached with 16 threads
^Cstrace: Process 4724 detached
strace: Process 4769 detached
strace: Process 4770 detached
strace: Process 4771 detached
strace: Process 4772 detached
strace: Process 4773 detached
strace: Process 4774 detached
strace: Process 4775 detached
strace: Process 4776 detached
strace: Process 4791 detached
strace: Process 4828 detached
strace: Process 4829 detached
strace: Process 4830 detached
strace: Process 4831 detached
strace: Process 4832 detached
strace: Process 9753 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 72.94    3.195455         642      4977       834 futex
 10.30    0.451047        4902        92           epoll_wait
  8.58    0.375965          91      4092         1 epoll_pwait
  5.53    0.242349          98      2448           nanosleep
  1.01    0.044349          60       737           write
  0.98    0.043014          41      1045       410 read
  0.28    0.012273        3068         4           io_getevents
  0.12    0.005421          32       167        74 epoll_ctl
  0.10    0.004333         120        36           tgkill
  0.05    0.001993          53        37           getpid
  0.03    0.001514          42        36         1 rt_sigreturn
  0.01    0.000524         131         4           io_submit
  0.01    0.000430          18        23           setsockopt
  0.01    0.000426          42        10         5 accept4
  0.01    0.000396          22        18           mprotect
  0.01    0.000376          53         7           close
  0.01    0.000331          30        11           sched_yield
  0.00    0.000193          32         6           getsockname
  0.00    0.000165          82         2           socket
  0.00    0.000152          76         2           connect
  0.00    0.000129           7        17           getrandom
  0.00    0.000097          97         1         1 restart_syscall
  0.00    0.000052          26         2           getpeername
  0.00    0.000007           7         1           getsockopt
  0.00    0.000006           6         1           shutdown
------ ----------- ----------- --------- --------- ----------------
100.00    4.380997                 13776      1326 total

disabling ha-cluster made no difference

OS: Ubuntu 20.04 Server LTS x86_64 with all updates MicroK8S: v1.20.4 classic Kernel: 5.4.0-67-generic CPU: proxmox/kvm hypervisor/ 4vCPU on Dual-CPU Xeon L5630 @ 2.13GHz System RAM: 8GB