You are here
KVM Host- Read your hypervisor’s Performance using Top data KVM 

KVM Host- Read your hypervisor’s Performance using Top

How subscribed is your server?

Here’s a sample of top output.  Here’s some key things that beginners need to know about how your host “sees” the VM’s.  My example is KVM on LTS.  This example is based on a dual proc xeon 8 cores, no hyperthreading, 64GB, HP Gen 5 Proliant Rack Server.

 

Tasks: 238 total, 1 running, 237 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.0 us, 0.7 sy, 0.0 ni, 98.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 1.7 us, 0.7 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 86.7 us, 13.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 2.0 us, 0.7 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.7 us, 0.3 sy, 0.0 ni, 93.4 id, 5.6 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 2.0 us, 0.3 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 1.0 us, 0.3 sy, 0.0 ni, 96.7 id, 2.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 2.7 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 64943228 total, 45933000 free, 13222932 used, 5787296 buff/cache
KiB Swap: 66056188 total, 66056188 free, 0 used. 51145900 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3038 libvirt+ 20 0 4919148 2.192g 23236 S 101.3 3.5 4:10.94 qemu-system-x86_64 -enable-kvm -name landscape -S -machine pc-i440fx-wily,accel+
3130 libvirt+ 20 0 1136988 324008 22948 S 4.7 0.5 1:22.04 qemu-system-x86_64 -enable-kvm -name pfsense -S -machine pc-i440fx-wily,accel=k+
3239 libvirt+ 20 0 17.459g 2.266g 23056 S 4.0 3.7 3:18.80 qemu-system-x86_64 -enable-kvm -name splunk -S -machine pc-i440fx-wily,accel=kv+
3291 libvirt+ 20 0 5571968 1.365g 23160 S 0.7 2.2 1:32.95 qemu-system-x86_64 -enable-kvm -name realworldnumbers -S -machine pc-i440fx-wil+
3356 libvirt+ 20 0 6109132 1.948g 23472 S 0.7 3.1 2:04.28 qemu-system-x86_64 -enable-kvm -name mail -S -machine pc-i440fx-wily,accel=kvm,+
3480 libvirt+ 20 0 2923704 1.225g 23020 S 0.7 2.0 1:47.15 qemu-system-x86_64 -enable-kvm -name cacti -S -machine pc-i440fx-wily,accel=kvm+
51 root 25 5 0 0 0 S 0.3 0.0 0:01.66 [ksmd]
52 root 39 19 0 0 0 S 0.3 0.0 0:00.08 [khugepaged]
1504 snmp 20 0 67912 12560 4036 S 0.3 0.0 0:00.55 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTrigg+
1656 root 20 0 29760 3776 2948 S 0.3 0.0 0:00.25 cmahostd -p 15 -s OK -l /var/log/hp-snmp-agents/cma.log
3184 libvirt+ 20 0 18.610g 1.445g 23224 S 0.3 2.3 1:33.67 qemu-system-x86_64 -enable-kvm -name canary -S -machine pc-i440fx-wily,accel=kv+
9287 root 20 0 41964 3960 3216 R 0.3 0.0 0:00.14 top

Run top.
Select “1” to demonstrate your per-core statistics.
Select “c” to display the process arguments.
Select “t” to cycle through a few different awesome graph style core pictograms.

Right away we’re sorted according to CPU and we can see a few things.

  1. libvirt VM’s show up as a single “process” on the host system.  Notice how the process can access more than 100% of CPUs.  If i provide multiple cores to VM’s, you can access more than hard 100% for one core.
  2. I have an 8 core machine, so maximum MHz percentage is total combined cores of 800%.
  3. “st” is Steal Time.  This is 0% “robbed” on the host (cause there’s no overload robbing it), and should never be more than 20% within any VM .  MOre than 20 would be contention with other VM’s vying for their shot at the real 8 cores in my machine.
  4. It’s my opinion you should give crappy little VM’s only ever 100%(1 core worth) of access, give 2-4 cores for more “busy/bursty” intensive VM’s, and the whole shebang worth of cores (8 for me) to any massive VM like splunk or large databasers.
  5. 238 threads – this is not abnormal for my host machine.

Here’s My system under load.

root@thewind:~# top
top - 20:49:58 up 37 min, 1 user, load average: 6.01, 2.27, 1.89
Tasks: 236 total, 2 running, 234 sleeping, 0 stopped, 0 zombie
%Cpu0 : 30.0/46.1 76[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu1 : 34.8/47.3 82[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu2 : 31.8/45.2 77[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu3 : 98.0/2.0 100[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu4 : 25.7/47.6 73[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu5 : 35.8/43.5 79[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu6 : 30.8/46.8 78[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
%Cpu7 : 19.0/40.3 59[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
KiB Mem : 64943228 total, 43586852 free, 15408264 used, 5948112 buff/cache
KiB Swap: 66056188 total, 66056188 free, 0 used. 48897540 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
 3184 libvirt+ 20 0 18.603g 3.125g 23240 R 497.7 5.0 10:47.01 qemu-system-x86_64 -enable-kvm -name canary -S -machine pc-i440fx-wily,accel=kv+ 
 3239 libvirt+ 20 0 17.459g 2.361g 23056 S 106.0 3.8 5:19.29 qemu-system-x86_64 -enable-kvm -name splunk -S -machine pc-i440fx-wily,accel=kv+ 
 3130 libvirt+ 20 0 1136988 384420 22948 S 12.3 0.6 2:54.41 qemu-system-x86_64 -enable-kvm -name pfsense -S -machine pc-i440fx-wily,accel=k+ 
 7 root 20 0 0 0 0 S 2.0 0.0 0:01.81 [rcu_sched] 
 3526 libvirt+ 20 0 10.027g 1.447g 23164 S 2.0 2.3 2:21.18 qemu-system-x86_64 -enable-kvm -name tenantinvoice -S -machine pc-i440fx-wily,a+ 
 3038 libvirt+ 20 0 4771620 2.228g 23236 S 1.7 3.6 8:22.21 qemu-system-x86_64 -enable-kvm -name landscape -S -machine pc-i440fx-wily,accel+ 
 3291 libvirt+ 20 0 5965184 1.377g 23160 S 1.3 2.2 2:18.26 qemu-system-x86_64 -enable-kvm -name realworldnumbers -S -machine pc-i440fx-wil+ 
 1655 root 20 0 24768 3620 2756 S 1.0 0.0 0:00.79 cmasm2d -p 30 -l /var/log/hp-snmp-agents/cma.log 
 3356 libvirt+ 20 0 6100936 2.329g 23472 S 1.0 3.8 2:50.53 qemu-system-x86_64 -enable-kvm -name mail -S -machine pc-i440fx-wily,accel=kvm,+ 
 3480 libvirt+ 20 0 2923704 1.301g 23020 S 0.7 2.1 2:20.83 qemu-system-x86_64 -enable-kvm -name cacti -S -machine pc-i440fx-wily,accel=kvm+ 
24959 root 20 0 41964 3896 3152 R 0.7 0.0 0:00.06 top 
 51 root 25 5 0 0 0 S 0.3 0.0 0:07.38 [ksmd] 
 3134 root 20 0 0 0 0 S 0.3 0.0 0:00.34 [kvm-pit/3130] 
 1 root 20 0 38084 6148 3988 S 0.0 0.0 0:04.18 /sbin/init

 

Related posts

One thought on “KVM Host- Read your hypervisor’s Performance using Top

  1. Mike Davidson

    Thanks for your excellent tips. I’m obviously not at your level of knowledge and operations but so far two of your articles have saved me hours if not days!

    Cheers!

    Mike Davidson

Leave a Comment