A bit of back story. My Xen experience started out about 5 years ago with
an Aaeon EMB-CV1 with 4GB of memory, running four VMs. As I outgrew that
hardware, I moved to an Aaeon EMB-KB1 with 8GB of memory. I outgrew that
hardware and have moved to an Asrock J5040-ITX with 16GB of memory. These
have all worked beautifully and have been very performant, it's amazed me
the capabilities of paravirtualization and the efficiency attained from the
platform. All have been running with a Debian dom0 and Debian domu's, using
the Debian-maintained Xen version. Each dom0 install has been a fresh
build, migrating the VMs afterwards.
I recently decided to add an additional board into the mix to help with
load so that things requiring more horsepower (like my ELK stack, minecraft
server, nextcloud instance, etc.) can live on the 5040, while lower-CPU
stuff like DNS, VPN server, mail server, etc. can live on a board with a
slower CPU. I wanted to be on the same CPU architecture so that I could
live-migrate VMs for maintenance, so I picked up an Asrock J4205 with 8GB
of memory. After installing Debian 10 (my standard build currently,) the
board was snappy and performant. I then installed xen-tools and
xen-system-amd64, and after rebooting, the system took significantly longer
to boot, and was very laggy from the console (and SSH as well.) At this
point I wasn't running any VMs, didn't have any custom tweaks, etc. Looking
at top, there was a lot of "steal," overall averaging around 10% between
all four cores (all physical cores.)
I tried tying the dom0 to only one CPU and at that point the dom0 was
consistently performant again. However, any domu's I tried to spin up would
be very laggy, with high "steal." Live-migrating them back to the 5040
they'd be fine again. This was also the case if I didn't live-migrate but
just started them up on the 4205.
I thought maybe I'd goofed something up in the build somehow, so I blew
away that installation and rebuilt it from scratch, and experience the same
thing.
I started logging performance with sysstat and this is what I see:
On the 5040 with 6 VMs running:
08:25:01 AM CPU %user %nice %system %iowait %steal
%idle
08:35:01 AM all 0.03 0.00 0.11 0.00 0.05
99.81
08:45:01 AM all 0.03 0.00 0.11 0.00 0.05
99.81
08:55:01 AM all 0.03 0.00 0.17 0.00 0.20
99.60
09:05:01 AM all 0.03 0.00 0.12 0.00 0.05
99.80
09:15:01 AM all 0.03 0.00 0.13 0.00 0.09
99.75
09:25:01 AM all 0.03 0.00 0.18 0.00 0.26
99.53
Average: all 0.03 0.00 0.14 0.00 0.12
99.72
On the 4205 with no VMs running:
08:35:01 AM CPU %user %nice %system %iowait %steal
%idle
08:45:02 AM all 0.03 0.00 0.07 0.01 7.74
92.15
08:55:02 AM all 0.03 0.00 0.09 0.00 8.95
90.93
09:05:01 AM all 0.03 0.00 0.07 0.00 9.19
90.70
09:15:01 AM all 0.03 0.00 0.08 0.00 7.93
91.96
09:25:01 AM all 0.03 0.00 0.07 0.00 8.85
91.05
09:35:01 AM all 0.03 0.00 0.19 0.00 6.73
93.05
Average: all 0.03 0.00 0.09 0.00 8.24
91.63
# top
top - 09:45:55 up 1:29, 1 user, load average: 0.15, 0.11, 0.08
Tasks: 161 total, 2 running, 159 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 1.1 sy, 0.0 ni, 96.6 id, 0.0 wa, 0.0 hi, 0.0 si,
2.3 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 78.9 id, 0.0 wa, 0.0 hi, 0.0 si,
21.1 st
%Cpu2 : 0.0 us, 1.1 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 0.0 si,
9.6 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 62.8 id, 0.0 wa, 0.0 hi, 0.0 si,
37.2 st
Is there any way to tell what's causing the performance degradation, and
what the dom0 is doing when it's "stealing" the CPU? I've been googling the
issue a lot the last few days and haven't found anything useful so far,
only threads saying that this happens when you oversubscribe your domu's,
but as I'm not running any domu's at this point I don't see how that could
be an issue since it's just sitting there looking cool but not doing any
real work.
Local disk storage on both dom0's is a single 20GB Intel 313 SLC SSD. VMs
are stored on a Debian nas box, connecting via iscsi.
# uname -a
Linux vhost2 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64
GNU/Linux
# cat /etc/debian_version
10.9
# xl info
host :
release : 4.19.0-14-amd64
version : #1 SMP Debian 4.19.171-2 (2021-01-30)
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 1497.612
hw_caps :
bfebfbff:47f8e3bf:2c100800:00000101:0000000f:2094e283:00000000:00000100
virt_caps : hvm hvm_directio
total_memory : 8040
free_memory : 7413
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 11
xen_extra : .4
xen_version : 4.11.4
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=512M,max:512M no-real-mode
edd=off
cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by : pkg-xen-devel
cc_compile_domain : lists.alioth.debian.org
cc_compile_date : Fri Dec 11 21:33:51 UTC 2020
build_id : 6d8e0fa3ddb825695eb6c6832631b4fa2331fe41
xend_config_format : 4
Chris
an Aaeon EMB-CV1 with 4GB of memory, running four VMs. As I outgrew that
hardware, I moved to an Aaeon EMB-KB1 with 8GB of memory. I outgrew that
hardware and have moved to an Asrock J5040-ITX with 16GB of memory. These
have all worked beautifully and have been very performant, it's amazed me
the capabilities of paravirtualization and the efficiency attained from the
platform. All have been running with a Debian dom0 and Debian domu's, using
the Debian-maintained Xen version. Each dom0 install has been a fresh
build, migrating the VMs afterwards.
I recently decided to add an additional board into the mix to help with
load so that things requiring more horsepower (like my ELK stack, minecraft
server, nextcloud instance, etc.) can live on the 5040, while lower-CPU
stuff like DNS, VPN server, mail server, etc. can live on a board with a
slower CPU. I wanted to be on the same CPU architecture so that I could
live-migrate VMs for maintenance, so I picked up an Asrock J4205 with 8GB
of memory. After installing Debian 10 (my standard build currently,) the
board was snappy and performant. I then installed xen-tools and
xen-system-amd64, and after rebooting, the system took significantly longer
to boot, and was very laggy from the console (and SSH as well.) At this
point I wasn't running any VMs, didn't have any custom tweaks, etc. Looking
at top, there was a lot of "steal," overall averaging around 10% between
all four cores (all physical cores.)
I tried tying the dom0 to only one CPU and at that point the dom0 was
consistently performant again. However, any domu's I tried to spin up would
be very laggy, with high "steal." Live-migrating them back to the 5040
they'd be fine again. This was also the case if I didn't live-migrate but
just started them up on the 4205.
I thought maybe I'd goofed something up in the build somehow, so I blew
away that installation and rebuilt it from scratch, and experience the same
thing.
I started logging performance with sysstat and this is what I see:
On the 5040 with 6 VMs running:
08:25:01 AM CPU %user %nice %system %iowait %steal
%idle
08:35:01 AM all 0.03 0.00 0.11 0.00 0.05
99.81
08:45:01 AM all 0.03 0.00 0.11 0.00 0.05
99.81
08:55:01 AM all 0.03 0.00 0.17 0.00 0.20
99.60
09:05:01 AM all 0.03 0.00 0.12 0.00 0.05
99.80
09:15:01 AM all 0.03 0.00 0.13 0.00 0.09
99.75
09:25:01 AM all 0.03 0.00 0.18 0.00 0.26
99.53
Average: all 0.03 0.00 0.14 0.00 0.12
99.72
On the 4205 with no VMs running:
08:35:01 AM CPU %user %nice %system %iowait %steal
%idle
08:45:02 AM all 0.03 0.00 0.07 0.01 7.74
92.15
08:55:02 AM all 0.03 0.00 0.09 0.00 8.95
90.93
09:05:01 AM all 0.03 0.00 0.07 0.00 9.19
90.70
09:15:01 AM all 0.03 0.00 0.08 0.00 7.93
91.96
09:25:01 AM all 0.03 0.00 0.07 0.00 8.85
91.05
09:35:01 AM all 0.03 0.00 0.19 0.00 6.73
93.05
Average: all 0.03 0.00 0.09 0.00 8.24
91.63
# top
top - 09:45:55 up 1:29, 1 user, load average: 0.15, 0.11, 0.08
Tasks: 161 total, 2 running, 159 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 1.1 sy, 0.0 ni, 96.6 id, 0.0 wa, 0.0 hi, 0.0 si,
2.3 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 78.9 id, 0.0 wa, 0.0 hi, 0.0 si,
21.1 st
%Cpu2 : 0.0 us, 1.1 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 0.0 si,
9.6 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 62.8 id, 0.0 wa, 0.0 hi, 0.0 si,
37.2 st
Is there any way to tell what's causing the performance degradation, and
what the dom0 is doing when it's "stealing" the CPU? I've been googling the
issue a lot the last few days and haven't found anything useful so far,
only threads saying that this happens when you oversubscribe your domu's,
but as I'm not running any domu's at this point I don't see how that could
be an issue since it's just sitting there looking cool but not doing any
real work.
Local disk storage on both dom0's is a single 20GB Intel 313 SLC SSD. VMs
are stored on a Debian nas box, connecting via iscsi.
# uname -a
Linux vhost2 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64
GNU/Linux
# cat /etc/debian_version
10.9
# xl info
host :
release : 4.19.0-14-amd64
version : #1 SMP Debian 4.19.171-2 (2021-01-30)
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 1497.612
hw_caps :
bfebfbff:47f8e3bf:2c100800:00000101:0000000f:2094e283:00000000:00000100
virt_caps : hvm hvm_directio
total_memory : 8040
free_memory : 7413
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 11
xen_extra : .4
xen_version : 4.11.4
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=512M,max:512M no-real-mode
edd=off
cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by : pkg-xen-devel
cc_compile_domain : lists.alioth.debian.org
cc_compile_date : Fri Dec 11 21:33:51 UTC 2020
build_id : 6d8e0fa3ddb825695eb6c6832631b4fa2331fe41
xend_config_format : 4
Chris