Mailing List Archive

Update DomU - Debian 11 - no loader found - Invalid kernel
Hello,

i use a Debian 10 Host (Dom0) with Xen 4.14.2.

I update a Debian (i386) DomU from buster to bullseye with the new kernel 5.10.0-8-686-pae.

When i reboot the DomU, there is a fail message and the DomU don't start:
xc: error: panic: xc_dom_core.c:693: xc_dom_find_loader: no loader found: Invalid kernel
libxl: error: libxl_dom.c:578:libxl__build_dom: xc_dom_parse_image failed
libxl: error: libxl_create.c:1576:domcreate_rebuild_done: Domain 40:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain 40:Non-existant domain

DomU: ls -l /boot
-rw-r--r-- 1 root root 83 Aug 3 07:50 System.map-5.10.0-8-686-pae
-rw-r--r-- 1 root root 236162 Aug 3 07:50 config-5.10.0-8-686-pae
-rw-r--r-- 1 root root 23537504 Sep 8 14:44 initrd.img-5.10.0-8-686-pae
-rw-r--r-- 1 root root 4983136 Aug 3 07:50 vmlinuz-5.10.0-8-686-pae

I can use the old kernel linux-image-4.19.0-17-686-pae and boot the DomU.

DomU.cfg
bootloader = '/usr/local/lib/xen/bin/pygrub'
vcpus = '4'
memory = '768'
disk = [
'/dev/vg0/DomU-disk,raw,xvda2,rw',
'/dev/vg0/DomU-swap,raw,xvda1,rw'
]
uuid = "XXXXXX"
vif = [ 'ip=X.X.X.X, mac=XXXXXXXXXXXX' ]
boot="c"
apic=1
acpi=1
pae=1
serial="pty"
sdl=0
keymap="de"
usb=1
stdvga=1
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

Anybody any ideas why and how i solve the problem?

Thanks for any help ??

Ronny
Re: Update DomU - Debian 11 - no loader found - Invalid kernel [ In reply to ]
On Wednesday, September 8, 2021 4:04:08 PM CEST Ronny Wagner wrote:
> Hello,
>
> i use a Debian 10 Host (Dom0) with Xen 4.14.2.
>
> I update a Debian (i386) DomU from buster to bullseye with the new kernel
> 5.10.0-8-686-pae.
>
> When i reboot the DomU, there is a fail message and the DomU don't start:
> xc: error: panic: xc_dom_core.c:693: xc_dom_find_loader: no loader found:
> Invalid kernel libxl: error: libxl_dom.c:578:libxl__build_dom:
> xc_dom_parse_image failed libxl: error:
> libxl_create.c:1576:domcreate_rebuild_done: Domain 40:cannot (re-)build
> domain: -3 libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
> 40:Non-existant domain
>
> DomU: ls -l /boot
> -rw-r--r-- 1 root root 83 Aug 3 07:50 System.map-5.10.0-8-686-pae
> -rw-r--r-- 1 root root 236162 Aug 3 07:50 config-5.10.0-8-686-pae
> -rw-r--r-- 1 root root 23537504 Sep 8 14:44 initrd.img-5.10.0-8-686-pae
> -rw-r--r-- 1 root root 4983136 Aug 3 07:50 vmlinuz-5.10.0-8-686-pae
>
> I can use the old kernel linux-image-4.19.0-17-686-pae and boot the DomU.
>
> DomU.cfg
> bootloader = '/usr/local/lib/xen/bin/pygrub'
> vcpus = '4'
> memory = '768'
> disk = [
> '/dev/vg0/DomU-disk,raw,xvda2,rw',
> '/dev/vg0/DomU-swap,raw,xvda1,rw'
> ]
> uuid = "XXXXXX"
> vif = [ 'ip=X.X.X.X, mac=XXXXXXXXXXXX' ]
> boot="c"
> apic=1
> acpi=1
> pae=1
> serial="pty"
> sdl=0
> keymap="de"
> usb=1
> stdvga=1
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
>
> Anybody any ideas why and how i solve the problem?
>
> Thanks for any help ??
>
> Ronny

Did you update the grub.cfg file (MANUALLY) on the DOMU to point to the new
kernel?
I don't use pygrub myself, but I understand it accesses the grub.cfg file to
determine the kernel-image to use.

--
Joost
AW: Update DomU - Debian 11 - no loader found - Invalid kernel [ In reply to ]
Hi,

i use the /boot/grub/menu.lst:

default 0
timeout 2
title Debian GNU/Linux 10
root (hd0,0)
kernel /vmlinuz root=/dev/xvda2 ro elevator=noop
initrd /initrd.img

title Debian GNU/Linux
root (hd0,0)
kernel /boot/vmlinuz-4.19.0-17-686-pae root=/dev/xvda2 ro elevator=noop
initrd /boot/initrd.img-4.19.0-17-686-pae

$ ls -l /
lrwxrwxrwx 1 root root 28 Sep 8 14:41 initrd.img -> boot/initrd.img-5.10.0-8-686-pae
lrwxrwxrwx 1 root root 25 Sep 8 14:41 vmlinuz -> boot/vmlinuz-5.10.0-8-686-pae

I don't understand "no loader found", i can change to 4.19.0-17 and the DomU started, but not with the 5.10.0-8.

Thanks for helping.

Ronny

-----Urspr?ngliche Nachricht-----
Von: Xen-users <xen-users-bounces@lists.xenproject.org> Im Auftrag von J. Roeleveld
Gesendet: Mittwoch, 8. September 2021 16:35
An: xen-users@lists.xenproject.org
Betreff: Re: Update DomU - Debian 11 - no loader found - Invalid kernel

On Wednesday, September 8, 2021 4:04:08 PM CEST Ronny Wagner wrote:
> Hello,
>
> i use a Debian 10 Host (Dom0) with Xen 4.14.2.
>
> I update a Debian (i386) DomU from buster to bullseye with the new kernel
> 5.10.0-8-686-pae.
>
> When i reboot the DomU, there is a fail message and the DomU don't start:
> xc: error: panic: xc_dom_core.c:693: xc_dom_find_loader: no loader found:
> Invalid kernel libxl: error: libxl_dom.c:578:libxl__build_dom:
> xc_dom_parse_image failed libxl: error:
> libxl_create.c:1576:domcreate_rebuild_done: Domain 40:cannot (re-)build
> domain: -3 libxl: error: libxl_domain.c:1182:libxl__destroy_domid: Domain
> 40:Non-existant domain
>
> DomU: ls -l /boot
> -rw-r--r-- 1 root root 83 Aug 3 07:50 System.map-5.10.0-8-686-pae
> -rw-r--r-- 1 root root 236162 Aug 3 07:50 config-5.10.0-8-686-pae
> -rw-r--r-- 1 root root 23537504 Sep 8 14:44 initrd.img-5.10.0-8-686-pae
> -rw-r--r-- 1 root root 4983136 Aug 3 07:50 vmlinuz-5.10.0-8-686-pae
>
> I can use the old kernel linux-image-4.19.0-17-686-pae and boot the DomU.
>
> DomU.cfg
> bootloader = '/usr/local/lib/xen/bin/pygrub'
> vcpus = '4'
> memory = '768'
> disk = [
> '/dev/vg0/DomU-disk,raw,xvda2,rw',
> '/dev/vg0/DomU-swap,raw,xvda1,rw'
> ]
> uuid = "XXXXXX"
> vif = [ 'ip=X.X.X.X, mac=XXXXXXXXXXXX' ]
> boot="c"
> apic=1
> acpi=1
> pae=1
> serial="pty"
> sdl=0
> keymap="de"
> usb=1
> stdvga=1
> on_poweroff = 'destroy'
> on_reboot = 'restart'
> on_crash = 'restart'
>
> Anybody any ideas why and how i solve the problem?
>
> Thanks for any help ??
>
> Ronny

Did you update the grub.cfg file (MANUALLY) on the DOMU to point to the new
kernel?
I don't use pygrub myself, but I understand it accesses the grub.cfg file to
determine the kernel-image to use.

--
Joost
Re: AW: Update DomU - Debian 11 - no loader found - Invalid kernel [ In reply to ]
On Wednesday, September 8, 2021 5:47:06 PM CEST Ronny Wagner wrote:
> Hi,
>
> i use the /boot/grub/menu.lst:
>
> default 0
> timeout 2
> title Debian GNU/Linux 10
> root (hd0,0)
> kernel /vmlinuz root=/dev/xvda2 ro elevator=noop
> initrd /initrd.img
>
> title Debian GNU/Linux
> root (hd0,0)
> kernel /boot/vmlinuz-4.19.0-17-686-pae root=/dev/xvda2 ro
> elevator=noop initrd /boot/initrd.img-4.19.0-17-686-pae
>
> $ ls -l /
> lrwxrwxrwx 1 root root 28 Sep 8 14:41 initrd.img ->
> boot/initrd.img-5.10.0-8-686-pae lrwxrwxrwx 1 root root 25 Sep 8
> 14:41 vmlinuz -> boot/vmlinuz-5.10.0-8-686-pae
>
> I don't understand "no loader found", i can change to 4.19.0-17 and the DomU
> started, but not with the 5.10.0-8.
>
> Thanks for helping.

Can you test with pointing directly at the kernel and initrd image?
Not the symbolic links?
That could also be the problem.

Do be aware, I don't think pygrub is still being developed and has several
limitations.

--
Joost
Re: Update DomU - Debian 11 - no loader found - Invalid kernel [ In reply to ]
On Wed, Sep 08, 2021 at 02:04:08PM +0000, Ronny Wagner wrote:
> i use a Debian 10 Host (Dom0) with Xen 4.14.2.
> I update a Debian (i386) DomU from buster to bullseye with the new kernel 5.10.0-8-686-pae.

There is no i386 PV support anymore:

| commit a13f2ef168cb2a033a284eb841bcc481ffbc90cf
| Author: Juergen Gross <jgross@suse.com>
| Date: Mon Jun 29 10:35:39 2020 +0200
|
| x86/xen: remove 32-bit Xen PV guest support
|
| Xen is requiring 64-bit machines today and since Xen 4.14 it can be
| built without 32-bit PV guest support. There is no need to carry the
| burden of 32-bit PV guest support in the kernel any longer, as new
| guests can be either HVM or PVH, or they can use a 64 bit kernel.
|
| Remove the 32-bit Xen PV support from the kernel.

Bastian

--
Witch! Witch! They'll burn ya!
-- Hag, "Tomorrow is Yesterday", stardate unknown
Re: Update DomU - Debian 11 - no loader found - Invalid kernel [ In reply to ]
Hi Ronny,

On Wed, Sep 08, 2021 at 02:04:08PM +0000, Ronny Wagner wrote:
> I update a Debian (i386) DomU from buster to bullseye with the new kernel 5.10.0-8-686-pae.
>
> When i reboot the DomU, there is a fail message and the DomU don't start:
> xc: error: panic: xc_dom_core.c:693: xc_dom_find_loader: no loader found: Invalid kernel

As Bastian pointed out, upstream Linux kernel at 5.10 does not
support 32-bit Xen PV guests any more.

I think your least invasive option is to use multiarch and install an
amd64 kernel. You can do that while booted into the working 4.19
kernel.

Follow these instructions only as far as installing and booting the
new amd64 kernel:

https://wiki.debian.org/CrossGrading

You'll then continue with a 64-bit kernel and 32-bit userland. Your
Debian guest will continue to properly upgrade to later versions of
the amd64 kernel.

Optionally you can also crossgrade entire userland to 64-bit but
this is a major undertaking fraught with difficulty and likely a
reinstall is quicker and safer. Changing only the kernel is
minimally invasive.

The above is a quick fix that gets you working again; longer term
you need to stop using 32-bit guests. I think from hypervisor
version 4.15 they are also unsupported in PV mode, can only work in
PVH or HVM mode (or PV-Shim, which is PV in PVH).

Also do note that since XSA-370 there is no security support in Xen
for 32-bit PV guests:

https://xenbits.xen.org/xsa/advisory-370.html
https://github.com/xen-project/xen/blob/master/SUPPORT.md#x86pv

I would also strongly suggest that you move off of pygrub and on to
pvhgrub, though I think you will need to build your own pvhgrub
image from upstream grub sources for this.

At the moment with pygrub you are having your dom0 open the guest
filesystem and search about for a menu.lst file, then parse it and
extract the kernel and initramfs files from the guest. This is
fragile as it relies on pygrub keeping compatibility with actual
grub configuration syntax. It's also a bit of a security worry to be
opening and parsing guest filesystems in dom0 context.

By contrast, pvhgrub is a copy of upstream grub that is booted as
the guest's kernel, so it runs in the guest context and behaves as
grub normally would.

Here are instructions for building a pvhgrub image:

https://wiki.xenproject.org/wiki/Xen_4.12_RC_test_instructions#PVH_domU_boot_via_grub2

You then use the resulting image as guest kernel and add "type=pvh"
to guest xl config file.

Good luck!

Cheers,
Andy
AW: Update DomU - Debian 11 - no loader found - Invalid kernel [ In reply to ]
Thank you very very much!!! Thats it, the crossgrade!!!

Greetinx
Ronny

-----Urspr?ngliche Nachricht-----
Von: Xen-users <xen-users-bounces@lists.xenproject.org> Im Auftrag von Andy Smith
Gesendet: Donnerstag, 9. September 2021 02:11
An: xen-users@lists.xenproject.org
Betreff: Re: Update DomU - Debian 11 - no loader found - Invalid kernel

Hi Ronny,

On Wed, Sep 08, 2021 at 02:04:08PM +0000, Ronny Wagner wrote:
> I update a Debian (i386) DomU from buster to bullseye with the new kernel 5.10.0-8-686-pae.
>
> When i reboot the DomU, there is a fail message and the DomU don't start:
> xc: error: panic: xc_dom_core.c:693: xc_dom_find_loader: no loader found: Invalid kernel

As Bastian pointed out, upstream Linux kernel at 5.10 does not
support 32-bit Xen PV guests any more.

I think your least invasive option is to use multiarch and install an
amd64 kernel. You can do that while booted into the working 4.19
kernel.

Follow these instructions only as far as installing and booting the
new amd64 kernel:

https://wiki.debian.org/CrossGrading

You'll then continue with a 64-bit kernel and 32-bit userland. Your
Debian guest will continue to properly upgrade to later versions of
the amd64 kernel.

Optionally you can also crossgrade entire userland to 64-bit but
this is a major undertaking fraught with difficulty and likely a
reinstall is quicker and safer. Changing only the kernel is
minimally invasive.

The above is a quick fix that gets you working again; longer term
you need to stop using 32-bit guests. I think from hypervisor
version 4.15 they are also unsupported in PV mode, can only work in
PVH or HVM mode (or PV-Shim, which is PV in PVH).

Also do note that since XSA-370 there is no security support in Xen
for 32-bit PV guests:

https://xenbits.xen.org/xsa/advisory-370.html
https://github.com/xen-project/xen/blob/master/SUPPORT.md#x86pv

I would also strongly suggest that you move off of pygrub and on to
pvhgrub, though I think you will need to build your own pvhgrub
image from upstream grub sources for this.

At the moment with pygrub you are having your dom0 open the guest
filesystem and search about for a menu.lst file, then parse it and
extract the kernel and initramfs files from the guest. This is
fragile as it relies on pygrub keeping compatibility with actual
grub configuration syntax. It's also a bit of a security worry to be
opening and parsing guest filesystems in dom0 context.

By contrast, pvhgrub is a copy of upstream grub that is booted as
the guest's kernel, so it runs in the guest context and behaves as
grub normally would.

Here are instructions for building a pvhgrub image:

https://wiki.xenproject.org/wiki/Xen_4.12_RC_test_instructions#PVH_domU_boot_via_grub2

You then use the resulting image as guest kernel and add "type=pvh"
to guest xl config file.

Good luck!

Cheers,
Andy