Hi Xen users list,
I've come across something strange. I recently set up Xen (self
compiled, Version 4.18) on a Raspberry Pi 4B. Until yesterday,
everything worked well with one DomU and one bridge.
Then I wanted to actually make use of the virtualization and started to
set up a second Debian Bookworm DomU (using xen-create-image) for
monitoring my systems with zabbix. The bridge used for this setup was
the device bridging the hardware NIC. I installed zabbix, set it up, and
everything went well, I could access the web interface without any problem.
Then I set up VLANs (using VLAN numbers 1 and 2, which I probably
shouldn't, but its a working setup, and therefore...) to separate
network traffic between the DomUs. I made the existing device bridge
VLAN 1 (bridge 1) and created a secondary device for bridging VLAN 2
(bridge 2). Using only bridge 1 / VLAN 1 everything works well, I can
access the zabbix web interface without any noticeable issue. After
switching the zabbix DomU to VLAN 2 / bridge 2, everything seemingly
keeps on working well, I can ping different devices in my network from
the zabbix DomU and vice versa, I can ssh into the machine.
However, as soon as I remotely access the zabbix web interface, the
complete system (DomUs and Dom0) becomes unresponsive and reboots after
a couple of seconds. This is reliably reproducable.
I didn't see any error message in any log (zabbix, DomU syslog, Dom0
syslog) except for the following lines immediately before the system
reboots on the Xen serial console:
(XEN) Watchdog timer fired for domain 0
(XEN) Hardware Dom0 shutdown: watchdog rebooting machine
As soon as I change the bridge to bridge 1 (with or without VLAN setup),
the web interface is accessible again after booting the zabbix DomU.
So I assume that causing high traffic on the virtual NIC when using a
VLAN setup with more than 1 VLAN makes the system hard crash. Of course,
there might be other causes that I'm not aware of, but that seems to be
the most likely assumption.
I'd appreciate any hints how to troubleshoot this and/or how to proceed
otherwise (bug report?).
Thanks,
Paul
xl info:
host : ***
release : 6.1.0-11-arm64
version : #1 SMP Debian 6.1.38-4 (2023-08-08)
machine : aarch64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 1
threads_per_core : 1
cpu_mhz : 54.000
hw_caps :
00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
virt_caps : hvm hap vpmu gnttab-v1
arm_sve_vector_length : 0
total_memory : 8043
free_memory : 5859
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 18
xen_extra : -unstable
xen_version : 4.18-unstable
xen_caps : xen-3.0-aarch64 xen-3.0-armv7l
xen_scheduler : credit2
xen_pagesize : 4096
platform_params : virt_start=0x0
xen_changeset : Fri Aug 11 09:59:49 2023 +0200 git:a9a3b432a8
xen_commandline : placeholder dom0_mem=1024M,max:1024M
console=dtuart dtuart=serial1 sync_console no-real-mode edd=off
cc_compiler : gcc (Debian 12.2.0-14) 12.2.0
cc_compile_by : root
cc_compile_domain : ***
cc_compile_date : Mon Aug 14 22:05:30 CEST 2023
build_id : b21191ae0cd2ff49905c166a665dc30d9a4b1fbf
xend_config_format : 4
cat /etc/network/interfaces (with VLAN setup, redacted for IPs):
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto enabcm6e4ei0
iface enabcm6e4ei0 inet manual
iface enabcm6e4ei0 inet6 manual
VLAN LAN
auto enabcm6e4ei0.1
iface enabcm6e4ei0.1 inet manual
VLAN DMZ_LAN
auto enabcm6e4ei0.2
iface enabcm6e4ei0.2 inet manual
#Bridge LAN
auto xenbr0
iface xenbr0 inet static
bridge_ports enabcm6e4ei0.1
address *.*.*.*/24
gateway *.*.*.*
iface xenbr0 inet6 static
bridge_ports enabcm6e4ei0.1
address *::*/64
gateway *::*
# use SLAAC al IPv6 address from the router
# we may notv6 forwarding, otherwise SLAAC ged
autoconf 1
accept_ra 2
#Bridge DMZ_LAN
auto xenbr1
iface xenbr1 inet manual
bridge_ports enabcm6e4ei0.2
I've come across something strange. I recently set up Xen (self
compiled, Version 4.18) on a Raspberry Pi 4B. Until yesterday,
everything worked well with one DomU and one bridge.
Then I wanted to actually make use of the virtualization and started to
set up a second Debian Bookworm DomU (using xen-create-image) for
monitoring my systems with zabbix. The bridge used for this setup was
the device bridging the hardware NIC. I installed zabbix, set it up, and
everything went well, I could access the web interface without any problem.
Then I set up VLANs (using VLAN numbers 1 and 2, which I probably
shouldn't, but its a working setup, and therefore...) to separate
network traffic between the DomUs. I made the existing device bridge
VLAN 1 (bridge 1) and created a secondary device for bridging VLAN 2
(bridge 2). Using only bridge 1 / VLAN 1 everything works well, I can
access the zabbix web interface without any noticeable issue. After
switching the zabbix DomU to VLAN 2 / bridge 2, everything seemingly
keeps on working well, I can ping different devices in my network from
the zabbix DomU and vice versa, I can ssh into the machine.
However, as soon as I remotely access the zabbix web interface, the
complete system (DomUs and Dom0) becomes unresponsive and reboots after
a couple of seconds. This is reliably reproducable.
I didn't see any error message in any log (zabbix, DomU syslog, Dom0
syslog) except for the following lines immediately before the system
reboots on the Xen serial console:
(XEN) Watchdog timer fired for domain 0
(XEN) Hardware Dom0 shutdown: watchdog rebooting machine
As soon as I change the bridge to bridge 1 (with or without VLAN setup),
the web interface is accessible again after booting the zabbix DomU.
So I assume that causing high traffic on the virtual NIC when using a
VLAN setup with more than 1 VLAN makes the system hard crash. Of course,
there might be other causes that I'm not aware of, but that seems to be
the most likely assumption.
I'd appreciate any hints how to troubleshoot this and/or how to proceed
otherwise (bug report?).
Thanks,
Paul
xl info:
host : ***
release : 6.1.0-11-arm64
version : #1 SMP Debian 6.1.38-4 (2023-08-08)
machine : aarch64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 1
threads_per_core : 1
cpu_mhz : 54.000
hw_caps :
00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
virt_caps : hvm hap vpmu gnttab-v1
arm_sve_vector_length : 0
total_memory : 8043
free_memory : 5859
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 18
xen_extra : -unstable
xen_version : 4.18-unstable
xen_caps : xen-3.0-aarch64 xen-3.0-armv7l
xen_scheduler : credit2
xen_pagesize : 4096
platform_params : virt_start=0x0
xen_changeset : Fri Aug 11 09:59:49 2023 +0200 git:a9a3b432a8
xen_commandline : placeholder dom0_mem=1024M,max:1024M
console=dtuart dtuart=serial1 sync_console no-real-mode edd=off
cc_compiler : gcc (Debian 12.2.0-14) 12.2.0
cc_compile_by : root
cc_compile_domain : ***
cc_compile_date : Mon Aug 14 22:05:30 CEST 2023
build_id : b21191ae0cd2ff49905c166a665dc30d9a4b1fbf
xend_config_format : 4
cat /etc/network/interfaces (with VLAN setup, redacted for IPs):
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto enabcm6e4ei0
iface enabcm6e4ei0 inet manual
iface enabcm6e4ei0 inet6 manual
VLAN LAN
auto enabcm6e4ei0.1
iface enabcm6e4ei0.1 inet manual
VLAN DMZ_LAN
auto enabcm6e4ei0.2
iface enabcm6e4ei0.2 inet manual
#Bridge LAN
auto xenbr0
iface xenbr0 inet static
bridge_ports enabcm6e4ei0.1
address *.*.*.*/24
gateway *.*.*.*
iface xenbr0 inet6 static
bridge_ports enabcm6e4ei0.1
address *::*/64
gateway *::*
# use SLAAC al IPv6 address from the router
# we may notv6 forwarding, otherwise SLAAC ged
autoconf 1
accept_ra 2
#Bridge DMZ_LAN
auto xenbr1
iface xenbr1 inet manual
bridge_ports enabcm6e4ei0.2