Mailing List Archive

Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0
Hello! Our setup ir probably antic but reliable. We build PV domU with xen-create-image (fuly automatic with Puppet).

Just today I built the first Jammy domU on a Jammy dom0 (Linux 5.15 with xen-hypervisor-4.16-amd64 and xen-tools 4.9.1). Building the domU worked fine, but I can not start the domU:

When using pygrub the error is:
xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable

So it seems that pygrub does not support the ZSTD compressed kernel.

So I tried to boot the domU with pvgrub2 by setting:
kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"

This brings me to a Grub shell, and manuall booting just leads to:
grub> linux /boot/vmlinuz root=/dev/xvda2
error: not xen image.

https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states "Support for zstd-compressed dom0 (x86) and domU kernels.".

So I need some advice/confirmations:
Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to add ZSTD support to pygrub?

If pygrub will not support ZSTD, what are my options? Should it work with pvgrub2 and I am doing something wrong? How to correctly use pvgrub2.

If pvgrub2 does not support ZSTD, what are my options? Would it help to move from PV to PVH,PVHVM,...? Can I automate building domUs similar to xen-create-image also with non-PV domUs?

Thanks
Klaus
Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0 [ In reply to ]
On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> Hello! Our setup ir probably antic but reliable. We build PV domU with xen-create-image (fuly automatic with Puppet).
>
> Just today I built the first Jammy domU on a Jammy dom0 (Linux 5.15 with xen-hypervisor-4.16-amd64 and xen-tools 4.9.1). Building the domU worked fine, but I can not start the domU:
>
> When using pygrub the error is:
> xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
>
> So it seems that pygrub does not support the ZSTD compressed kernel.
>
> So I tried to boot the domU with pvgrub2 by setting:
> kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"
>
> This brings me to a Grub shell, and manuall booting just leads to:
> grub> linux /boot/vmlinuz root=/dev/xvda2
> error: not xen image.
>
> https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states "Support for zstd-compressed dom0 (x86) and domU kernels.".
>
> So I need some advice/confirmations:
> Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to add ZSTD support to pygrub?
>
> If pygrub will not support ZSTD, what are my options? Should it work with pvgrub2 and I am doing something wrong? How to correctly use pvgrub2.
>
> If pvgrub2 does not support ZSTD, what are my options? Would it help to move from PV to PVH,PVHVM,...? Can I automate building domUs similar to xen-create-image also with non-PV domUs?
>
> Thanks
> Klaus

For my focal based environment I can boot guests with the HWE kernel using
zstd with these options:

type = 'pvh'
kernel = '/usr/lib/grub-xen/grub-i386-xen_pvh.bin'

For guests that need to be 'pv' then I've had to decompress the kernel image
using the extract-vmlinux which is part of the kernel sources.

James
Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0 [ In reply to ]
Hi Klaus,

On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> When using pygrub the error is:
> xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD decompress support unavailable
>
> So it seems that pygrub does not support the ZSTD compressed kernel.

pygrub extracts a kernel and initrd from the guest block device and
then Xen direct boots it. Your Xen does not support zstd-compressed
kernels for direct boot so that doesn't work.

You can use the extract-vmlinux script to decompress a guest kernel
to vmlinux and direct boot that. It isn't hard to bodge a test into
pygrub for zstd (and lz4) compressed kernels and automatically call
extract-vmlinux on them, then boot the resulting vmlinux.

However, pygrub is a really really bad way to boot guests, so you
would be much better off figuring out how to stop using it
altogether. Maybe now is a good time for that, since you're going to
have to make some changes anyway?

TL;DR: PVH mode is probably your saviour as long as you don't need
to boot anything older than kernel version 4.20 or so.

> So I tried to boot the domU with pvgrub2 by setting:
> kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"

pvgrub also doesn't support LZ4 or zstd compressed kernels. It seems
to be abandoned in that no one is interested in adding such support
to it.

Again you can lash something together that decompresses the kernels
first. This time a bit different since it needs to run in guest
context due to pvgrub. Here's something I came up with to make the
guest decompress its kernel when the kernel is installed:

https://github.com/bitfolk/decompress-lz4-kernels

> https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states "Support for zstd-compressed dom0 (x86) and domU kernels.".
>
> So I need some advice/confirmations:
> Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to add ZSTD support to pygrub?

pygrub doesn't boot the kernel, Xen does, so if your Xen supports
direct booting LZ4/zstd then that will work. It's not a question of
what pygrub supports as pygrub does not parse or manipulate the
kernel file in any way, it just passes it to Xen.

However you'd still be using pygrub, which is bad.

> If pygrub will not support ZSTD, what are my options? Should it
> work with pvgrub2 and I am doing something wrong? How to correctly
> use pvgrub2.

pvgrub seems to be a dead end for guests using LZ4/zstd kernels as
there seems to be no interest in supporting that.

> If pvgrub2 does not support ZSTD, what are my options? Would it
> help to move from PV to PVH,PVHVM,...?

Yes. Every guest that uses LZ4 or zstd kernels also supports being
booted in PVH mode, so I switched to PVH for every guest that
supports it and stopped using my extract kernel hack that was
necessary for getting pvgrub to boot them.

One of the other reasons why pvhgrub is better is that it uses the
standard parts of grub for loading images, so if grub supports the
compression format then pvhgrub does too. So no problem loading
lz4/zstd compressed kernels, as long as the grub package you started
with is new enough.

> Can I automate building
> domUs similar to xen-create-image also with non-PV domUs?

As far as I know xen-tools and xen-create-image are also dead
upstream so it would be best not to rely on them. Maybe a project
for a later time though, as I can't see any reason why something
that xen-create-image makes would not boot under PVH mode.

For Debian/Ubuntu-based guests you can just boot the installer
kernel/initrd in PVH mode and run through a normal text mode
install. Or you can use debootstrap from your dom0 to download
everything you need into a directory. Which is basically what
xen-create-image is doing.

(This all changed with Ubuntu >= 21.10 as they have moved away from
the Debian installer. For newer Ubuntu it's going to be either
debootstrap or download the Ubuntu cloud image and boot that (with
optional cloud-init for any configuration you need).)

Cheers,
Andy
Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0 [ In reply to ]
Just tested booting a Jammy domU as pv using pygrub from the build
below (works):

> sudo rpm -qa | grep -P 'xen|qemu'
xen-hypervisor-4.15.2-1.el8.x86_64
xen-doc-4.15.2-1.el8.x86_64
xen-4.15.2-1.el8.x86_64
xen-licenses-4.15.2-1.el8.x86_64
qemut-0.10.2+2-1.el8.x86_64
xen-libs-4.15.2-1.el8.x86_64
qemuu-img-5.1.0+2-1.el8.x86_64
xen-ocaml-4.15.2-1.el8.x86_64
qemuu-5.1.0+2-1.el8.x86_64
ipxe-roms-qemu-20181214-8.git133f4c47.el8.noarch
xen-runtime-4.15.2-1.el8.x86_64

We also boot with the pvgrub from the included stubdom. Both of them
work with zstd compressed kernels dom0+domU since they use the same
xen libraries with latest patches.

Our fedora domU with zstd compressed kernels also work with pygrub and
pvgrub (stubdom).

Your dom0 seems problematic since you don't make clear if you
self-build xen-4.16 or use the Jammy packages from the Universe repo.

best wishes,
PryMar56
As mentioned by Andy S. the grub xen support has no lz4/zstd for
direct kernel boot, as done with pvgrub2.
AW: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0 [ In reply to ]
Sorry for the late repsonse. I just found some time for testing.

Thanks for the detailed answer - that made things much clearer.

As suggested by you and others: using PVH instead of PV solved the problem.

Thanks
Klaus

> -----Urspr?ngliche Nachricht-----
> Von: Xen-users <xen-users-bounces@lists.xenproject.org> Im Auftrag von
> Andy Smith
> Gesendet: Mittwoch, 27. April 2022 23:54
> An: xen-users@lists.xenproject.org
> Betreff: Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy
> dom0
>
> Hi Klaus,
>
> On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> > When using pygrub the error is:
> > xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD
> decompress support unavailable
> >
> > So it seems that pygrub does not support the ZSTD compressed kernel.
>
> pygrub extracts a kernel and initrd from the guest block device and
> then Xen direct boots it. Your Xen does not support zstd-compressed
> kernels for direct boot so that doesn't work.
>
> You can use the extract-vmlinux script to decompress a guest kernel
> to vmlinux and direct boot that. It isn't hard to bodge a test into
> pygrub for zstd (and lz4) compressed kernels and automatically call
> extract-vmlinux on them, then boot the resulting vmlinux.
>
> However, pygrub is a really really bad way to boot guests, so you
> would be much better off figuring out how to stop using it
> altogether. Maybe now is a good time for that, since you're going to
> have to make some changes anyway?
>
> TL;DR: PVH mode is probably your saviour as long as you don't need
> to boot anything older than kernel version 4.20 or so.
>
> > So I tried to boot the domU with pvgrub2 by setting:
> > kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"
>
> pvgrub also doesn't support LZ4 or zstd compressed kernels. It seems
> to be abandoned in that no one is interested in adding such support
> to it.
>
> Again you can lash something together that decompresses the kernels
> first. This time a bit different since it needs to run in guest
> context due to pvgrub. Here's something I came up with to make the
> guest decompress its kernel when the kernel is installed:
>
> https://github.com/bitfolk/decompress-lz4-kernels
>
> > https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states
> "Support for zstd-compressed dom0 (x86) and domU kernels.".
> >
> > So I need some advice/confirmations:
> > Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to add
> ZSTD support to pygrub?
>
> pygrub doesn't boot the kernel, Xen does, so if your Xen supports
> direct booting LZ4/zstd then that will work. It's not a question of
> what pygrub supports as pygrub does not parse or manipulate the
> kernel file in any way, it just passes it to Xen.
>
> However you'd still be using pygrub, which is bad.
>
> > If pygrub will not support ZSTD, what are my options? Should it
> > work with pvgrub2 and I am doing something wrong? How to correctly
> > use pvgrub2.
>
> pvgrub seems to be a dead end for guests using LZ4/zstd kernels as
> there seems to be no interest in supporting that.
>
> > If pvgrub2 does not support ZSTD, what are my options? Would it
> > help to move from PV to PVH,PVHVM,...?
>
> Yes. Every guest that uses LZ4 or zstd kernels also supports being
> booted in PVH mode, so I switched to PVH for every guest that
> supports it and stopped using my extract kernel hack that was
> necessary for getting pvgrub to boot them.
>
> One of the other reasons why pvhgrub is better is that it uses the
> standard parts of grub for loading images, so if grub supports the
> compression format then pvhgrub does too. So no problem loading
> lz4/zstd compressed kernels, as long as the grub package you started
> with is new enough.
>
> > Can I automate building
> > domUs similar to xen-create-image also with non-PV domUs?
>
> As far as I know xen-tools and xen-create-image are also dead
> upstream so it would be best not to rely on them. Maybe a project
> for a later time though, as I can't see any reason why something
> that xen-create-image makes would not boot under PVH mode.
>
> For Debian/Ubuntu-based guests you can just boot the installer
> kernel/initrd in PVH mode and run through a normal text mode
> install. Or you can use debootstrap from your dom0 to download
> everything you need into a directory. Which is basically what
> xen-create-image is doing.
>
> (This all changed with Ubuntu >= 21.10 as they have moved away from
> the Debian installer. For newer Ubuntu it's going to be either
> debootstrap or download the Ubuntu cloud image and boot that (with
> optional cloud-init for any configuration you need).)
>
> Cheers,
> Andy
AW: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0 [ In reply to ]
Just for the records: meanwhile I can also boot 22.04 domUs with 22.04 dom0.
The problem was a build-bug in the Ubuntu Xen package. You can find the details here:
https://bugs.launchpad.net/ubuntu/+source/xen/+bug/1956166

regards
Klaus

> -----Urspr?ngliche Nachricht-----
> Von: Klaus Darilion
> Gesendet: Montag, 19. September 2022 22:45
> An: xen-users@lists.xenproject.org
> Betreff: AW: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy
> dom0
>
> Sorry for the late repsonse. I just found some time for testing.
>
> Thanks for the detailed answer - that made things much clearer.
>
> As suggested by you and others: using PVH instead of PV solved the problem.
>
> Thanks
> Klaus
>
> > -----Urspr?ngliche Nachricht-----
> > Von: Xen-users <xen-users-bounces@lists.xenproject.org> Im Auftrag von
> > Andy Smith
> > Gesendet: Mittwoch, 27. April 2022 23:54
> > An: xen-users@lists.xenproject.org
> > Betreff: Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy
> > dom0
> >
> > Hi Klaus,
> >
> > On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> > > When using pygrub the error is:
> > > xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD
> > decompress support unavailable
> > >
> > > So it seems that pygrub does not support the ZSTD compressed kernel.
> >
> > pygrub extracts a kernel and initrd from the guest block device and
> > then Xen direct boots it. Your Xen does not support zstd-compressed
> > kernels for direct boot so that doesn't work.
> >
> > You can use the extract-vmlinux script to decompress a guest kernel
> > to vmlinux and direct boot that. It isn't hard to bodge a test into
> > pygrub for zstd (and lz4) compressed kernels and automatically call
> > extract-vmlinux on them, then boot the resulting vmlinux.
> >
> > However, pygrub is a really really bad way to boot guests, so you
> > would be much better off figuring out how to stop using it
> > altogether. Maybe now is a good time for that, since you're going to
> > have to make some changes anyway?
> >
> > TL;DR: PVH mode is probably your saviour as long as you don't need
> > to boot anything older than kernel version 4.20 or so.
> >
> > > So I tried to boot the domU with pvgrub2 by setting:
> > > kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"
> >
> > pvgrub also doesn't support LZ4 or zstd compressed kernels. It seems
> > to be abandoned in that no one is interested in adding such support
> > to it.
> >
> > Again you can lash something together that decompresses the kernels
> > first. This time a bit different since it needs to run in guest
> > context due to pvgrub. Here's something I came up with to make the
> > guest decompress its kernel when the kernel is installed:
> >
> > https://github.com/bitfolk/decompress-lz4-kernels
> >
> > > https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states
> > "Support for zstd-compressed dom0 (x86) and domU kernels.".
> > >
> > > So I need some advice/confirmations:
> > > Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to add
> > ZSTD support to pygrub?
> >
> > pygrub doesn't boot the kernel, Xen does, so if your Xen supports
> > direct booting LZ4/zstd then that will work. It's not a question of
> > what pygrub supports as pygrub does not parse or manipulate the
> > kernel file in any way, it just passes it to Xen.
> >
> > However you'd still be using pygrub, which is bad.
> >
> > > If pygrub will not support ZSTD, what are my options? Should it
> > > work with pvgrub2 and I am doing something wrong? How to correctly
> > > use pvgrub2.
> >
> > pvgrub seems to be a dead end for guests using LZ4/zstd kernels as
> > there seems to be no interest in supporting that.
> >
> > > If pvgrub2 does not support ZSTD, what are my options? Would it
> > > help to move from PV to PVH,PVHVM,...?
> >
> > Yes. Every guest that uses LZ4 or zstd kernels also supports being
> > booted in PVH mode, so I switched to PVH for every guest that
> > supports it and stopped using my extract kernel hack that was
> > necessary for getting pvgrub to boot them.
> >
> > One of the other reasons why pvhgrub is better is that it uses the
> > standard parts of grub for loading images, so if grub supports the
> > compression format then pvhgrub does too. So no problem loading
> > lz4/zstd compressed kernels, as long as the grub package you started
> > with is new enough.
> >
> > > Can I automate building
> > > domUs similar to xen-create-image also with non-PV domUs?
> >
> > As far as I know xen-tools and xen-create-image are also dead
> > upstream so it would be best not to rely on them. Maybe a project
> > for a later time though, as I can't see any reason why something
> > that xen-create-image makes would not boot under PVH mode.
> >
> > For Debian/Ubuntu-based guests you can just boot the installer
> > kernel/initrd in PVH mode and run through a normal text mode
> > install. Or you can use debootstrap from your dom0 to download
> > everything you need into a directory. Which is basically what
> > xen-create-image is doing.
> >
> > (This all changed with Ubuntu >= 21.10 as they have moved away from
> > the Debian installer. For newer Ubuntu it's going to be either
> > debootstrap or download the Ubuntu cloud image and boot that (with
> > optional cloud-init for any configuration you need).)
> >
> > Cheers,
> > Andy