I'm using Xen 4.16.1 with latest patches (XSA-408) and Alpine Linux
4.16.2 x86_64 installed in Dom0 on multiple servers with different
hardware. All DomUs are minimal PVHs started with PyGrub.
About a week ago, these otherwise very stable servers began to freeze
randomly, without any changes to the configuration. At the KVM console
the only option was a hard reboot, even SysRq key sequences had any no
effect. Afterward there were no logs, no data in /sys/fs/pstore or
anything to tell what had happened.
The servers will freeze in same moment a DomU is created, either
manually or by rebooting from inside the DomU. However, not always. If
the DomU creation has not managed to freeze the server, the DomU and
server will work without problems as usual.
By either creating a small script in a DomU that automatically reboots
the system after loading, or with a script in Dom0 that continually
creates and destroys a DomU, the host can be crashed reliable in a few
minutes.
I have tried to do an identical installation of Xen in a QEMU VM, and in
this setup I can freeze the QEMU VM in exact same way as the physical
servers. The QEMU VM is configured with a serial console, and if I
switch serial input from Dom0 to Xen (CTRL-A CTRL-A CTRL-A) before
Dom0 freezes, this still works and various debug information can be
printed. I figure this is helpful in finding out what freezes Dom0.
I'm not sure how to proceed from here, what debug information from
Xen that could be helpful or how to interpret it.
Can anyone help?
Thanks in advance.
Best regards,
Mogens Jensen
4.16.2 x86_64 installed in Dom0 on multiple servers with different
hardware. All DomUs are minimal PVHs started with PyGrub.
About a week ago, these otherwise very stable servers began to freeze
randomly, without any changes to the configuration. At the KVM console
the only option was a hard reboot, even SysRq key sequences had any no
effect. Afterward there were no logs, no data in /sys/fs/pstore or
anything to tell what had happened.
The servers will freeze in same moment a DomU is created, either
manually or by rebooting from inside the DomU. However, not always. If
the DomU creation has not managed to freeze the server, the DomU and
server will work without problems as usual.
By either creating a small script in a DomU that automatically reboots
the system after loading, or with a script in Dom0 that continually
creates and destroys a DomU, the host can be crashed reliable in a few
minutes.
I have tried to do an identical installation of Xen in a QEMU VM, and in
this setup I can freeze the QEMU VM in exact same way as the physical
servers. The QEMU VM is configured with a serial console, and if I
switch serial input from Dom0 to Xen (CTRL-A CTRL-A CTRL-A) before
Dom0 freezes, this still works and various debug information can be
printed. I figure this is helpful in finding out what freezes Dom0.
I'm not sure how to proceed from here, what debug information from
Xen that could be helpful or how to interpret it.
Can anyone help?
Thanks in advance.
Best regards,
Mogens Jensen