Mailing List Archive: Xen on Raspberry Pi 4

Xen on Raspberry Pi 4

Aug 2, 2023, 1:42 PM

Post #1 of 7 (228 views)

Hi list,

Now that the devices are available again, I got hold of a Raspberry Pi 4
(8 GB) which I am currently trying to set up Xen on for using it as a
backup server.

What I did was to install a standard Debian distribution booted via
EFI/GRUB2 (following
https://forums.raspberrypi.com/viewtopic.php?t=282839) and then install
the xen-System-arm64 package via apt. I have output on a console via
UART (at least when booting Debian without Xen, see hypothesis 3 below).

Booting Debian without Xen is working fine. (Although I only see the
Grub selection menu via UART when no HDMI is plugged in while booting.)

Selecting Xen in Grub leads to the following output on the console right
after Grub hands over:

Loading Xen 4.17-arm64.efi ...
Loading Linux 6.1.0-10-arm64 ...
Loading initial ramdisk ...
Using modules provided by bootloader in FDT
Xen 4.17.2-pre (c/s ) EFI loader

And then the output stops.

I have four hypotheses (not mutually exclusive, and there are very
probably unknown unknowns on my side):

1. Booting Xen via EFI is not working, and I need to use U-Boot as
described in
https://xenproject.org/2020/09/29/xen-on-raspberry-pi-4-adventures/

2. The Debian package xen-system-arm64 is not suitable for this purpose
and I need to compile Xen from scratch

3. I got the console settings wrong (I currently use the settings from
https://xenproject.org/2020/09/29/xen-on-raspberry-pi-4-adventures/ for
Xen command line specified in Grub, but I also tried a lot of different
settings), and Xen is actually doing something, but I don't see the output.

4. There is something wrong with my setup and I need to change specific
EFI and/or other settings.

Has anybody got a similar setup working and can give me hints?

I am aware of https://github.com/dornerworks/xen-rpi4-builder, I might
try this as one of the next steps, but I'd rather stick with Debian, if
possible.

Thanks,

Paul

Re: Xen on Raspberry Pi 4 [ In reply to ]

julien at xen

Aug 2, 2023, 2:03 PM

Post #2 of 7 (228 views)

Permalink

On 02/08/2023 21:42, Paul Leiber wrote:
> Hi list,

Hi Paul,

> Now that the devices are available again, I got hold of a Raspberry Pi 4
> (8 GB) which I am currently trying to set up Xen on for using it as a
> backup server.
>
> What I did was to install a standard Debian distribution booted via
> EFI/GRUB2 (following
> https://forums.raspberrypi.com/viewtopic.php?t=282839) and then install
> the xen-System-arm64 package via apt. I have output on a console via
> UART (at least when booting Debian without Xen, see hypothesis 3 below).
>
> Booting Debian without Xen is working fine. (Although I only see the
> Grub selection menu via UART when no HDMI is plugged in while booting.)
>
> Selecting Xen in Grub leads to the following output on the console right
> after Grub hands over:
>
> Loading Xen 4.17-arm64.efi ...
> Loading Linux 6.1.0-10-arm64 ...
> Loading initial ramdisk ...
> Using modules provided by bootloader in FDT
> Xen 4.17.2-pre (c/s ) EFI loader
>
> And then the output stops.
>
> I have four hypotheses (not mutually exclusive, and there are very
> probably unknown unknowns on my side):
>
> 1. Booting Xen via EFI is not working, and I need to use U-Boot as
> described in
> https://xenproject.org/2020/09/29/xen-on-raspberry-pi-4-adventures/
>
> 2. The Debian package xen-system-arm64 is not suitable for this purpose
> and I need to compile Xen from scratch
>
> 3. I got the console settings wrong (I currently use the settings from
> https://xenproject.org/2020/09/29/xen-on-raspberry-pi-4-adventures/ for
> Xen command line specified in Grub, but I also tried a lot of different
> settings), and Xen is actually doing something, but I don't see the output.
>
> 4. There is something wrong with my setup and I need to change specific
> EFI and/or other settings.
>
> Has anybody got a similar setup working and can give me hints?

I have been using Debian + Xen + Grub + UEFI on Rpi4. UEFI firmwares are
mostly shipped with ACPI only (IOW no Device-Tree support) This is not
yet a fully supported configuration on Arm64 and therefore not enabled
by default.

Can you check the kernel log when booting Debian without Xen for
anything referring to ACPI? If so, you will need to Xen build yourself
to enable ACPI (this is protected by EXPERT). Alternatively you could
use U-boot where Device-Tree boot will be available.

Cheers,

--
Julien Grall

Re: Xen on Raspberry Pi 4 [ In reply to ]

paul at onlineschubla

Aug 3, 2023, 10:44 PM

Post #3 of 7 (228 views)

Permalink

Am 02.08.2023 um 23:03 schrieb Julien Grall:
>
>
> On 02/08/2023 21:42, Paul Leiber wrote:
>> Hi list,
>
> Hi Paul,
>
>>
>> What I did was to install a standard Debian distribution booted via
>> EFI/GRUB2 (following
>> https://forums.raspberrypi.com/viewtopic.php?t=282839) and then
>> install the xen-System-arm64 package via apt. I have output on a
>> console via UART (at least when booting Debian without Xen, see
>> hypothesis 3 below).
>>
>> Booting Debian without Xen is working fine. (Although I only see the
>> Grub selection menu via UART when no HDMI is plugged in while booting.)
>>
>> Selecting Xen in Grub leads to the following output on the console
>> right after Grub hands over:
>>
>> Loading Xen 4.17-arm64.efi ...
>> Loading Linux 6.1.0-10-arm64 ...
>> Loading initial ramdisk ...
>> Using modules provided by bootloader in FDT
>> Xen 4.17.2-pre (c/s ) EFI loader
>>
>> And then the output stops.
>>
>
> I have been using Debian + Xen + Grub + UEFI on Rpi4. UEFI firmwares are
> mostly shipped with ACPI only (IOW no Device-Tree support) This is not
> yet a fully supported configuration on Arm64 and therefore not enabled
> by default.
>
> Can you check the kernel log when booting Debian without Xen for
> anything referring to ACPI? If so, you will need to Xen build yourself
> to enable ACPI (this is protected by EXPERT). Alternatively you could
> use U-boot where Device-Tree boot will be available.

Indeed, the device is using ACPI. There is an option in the UEFI
settings to switch to Device-Tree, but that didn't have any effect I
could see, other than plain Debian didn't boot anymore either.

I compiled Xen with the ACPI option for Arm, and got it running. Thank
you very much for your hint, Julien!

For reference:

To enable the ACPI option, I went to the "xen" subdirectory in the Xen
source main directory after "git clone" and did a "make menuconfig".

Out of convience, I initially intended to only rebuild the Debian Xen
hypervisor package with the ACPI switch on, but that didn't work
somehow. I couldn't find out why. I then created a "Frankenxen" with a
newly built hypervisor and standard debian tools, which booted (yay!),
but of course the xen tools didn't work. Using the repo to check out a
Xen 17.1 version and build the hypervisor (so I could still rely on
Debian updates for all the other Xen packages) also didn't work, Xen
didn't boot. I then decided to build a complete Xen version from scratch
from the master repo, which worked well. So perhaps this approach with
enabling ACPI only works on Xen versions > 17.1?

Re: Xen on Raspberry Pi 4 [ In reply to ]

julien at xen

Aug 11, 2023, 11:51 AM

Post #4 of 7 (226 views)

Permalink

Hi Paul,

On 04/08/2023 06:44, Paul Leiber wrote:
> Am 02.08.2023 um 23:03 schrieb Julien Grall:
>>
>>
>> On 02/08/2023 21:42, Paul Leiber wrote:
>>> Hi list,
>>
>> Hi Paul,
>>
>>>
>>> What I did was to install a standard Debian distribution booted via
>>> EFI/GRUB2 (following
>>> https://forums.raspberrypi.com/viewtopic.php?t=282839) and then
>>> install the xen-System-arm64 package via apt. I have output on a
>>> console via UART (at least when booting Debian without Xen, see
>>> hypothesis 3 below).
>>>
>>> Booting Debian without Xen is working fine. (Although I only see the
>>> Grub selection menu via UART when no HDMI is plugged in while booting.)
>>>
>>> Selecting Xen in Grub leads to the following output on the console
>>> right after Grub hands over:
>>>
>>> Loading Xen 4.17-arm64.efi ...
>>> Loading Linux 6.1.0-10-arm64 ...
>>> Loading initial ramdisk ...
>>> Using modules provided by bootloader in FDT
>>> Xen 4.17.2-pre (c/s ) EFI loader
>>>
>>> And then the output stops.
>>>
>>
>> I have been using Debian + Xen + Grub + UEFI on Rpi4. UEFI firmwares
>> are mostly shipped with ACPI only (IOW no Device-Tree support) This is
>> not yet a fully supported configuration on Arm64 and therefore not
>> enabled by default.
>>
>> Can you check the kernel log when booting Debian without Xen for
>> anything referring to ACPI? If so, you will need to Xen build yourself
>> to enable ACPI (this is protected by EXPERT). Alternatively you could
>> use U-boot where Device-Tree boot will be available.
>
> Indeed, the device is using ACPI. There is an option in the UEFI
> settings to switch to Device-Tree, but that didn't have any effect I
> could see, other than plain Debian didn't boot anymore either.
>
> I compiled Xen with the ACPI option for Arm, and got it running. Thank
> you very much for your hint, Julien!
>
> For reference:
>
> To enable the ACPI option, I went to the "xen" subdirectory in the Xen
> source main directory after "git clone" and did a "make menuconfig".
>
> Out of convience, I initially intended to only rebuild the Debian Xen
> hypervisor package with the ACPI switch on, but that didn't work
> somehow. I couldn't find out why.

Is Debian using 4.17.2? Did you see any output from Xen/Linux.

> I then created a "Frankenxen" with a
> newly built hypervisor and standard debian tools, which booted (yay!),
> but of course the xen tools didn't work. Using the repo to check out a
> Xen 17.1 version and build the hypervisor (so I could still rely on
> Debian updates for all the other Xen packages) also didn't work, Xen
> didn't boot. I then decided to build a complete Xen version from scratch
> from the master repo, which worked well. So perhaps this approach with
> enabling ACPI only works on Xen versions > 17.1?

I don't remember any change related to ACPI on Arm that are in master
but not 4.17. I will try to have a look in the next couple of weeks and
find which patches may be missing.

Cheers,

--
Julien Grall

Re: Xen on Raspberry Pi 4 [ In reply to ]

paul at onlineschubla

Aug 13, 2023, 10:46 AM

Post #5 of 7 (225 views)

Permalink

Am 11.08.2023 um 20:51 schrieb Julien Grall:
> Hi Paul,
>
> On 04/08/2023 06:44, Paul Leiber wrote:
>> Am 02.08.2023 um 23:03 schrieb Julien Grall:
>>>
>>>
>>> On 02/08/2023 21:42, Paul Leiber wrote:
>>>> Hi list,
>>>
>>> Hi Paul,
>>>
>>>>
>>>> What I did was to install a standard Debian distribution booted via
>>>> EFI/GRUB2 (following
>>>> https://forums.raspberrypi.com/viewtopic.php?t=282839) and then
>>>> install the xen-System-arm64 package via apt. I have output on a
>>>> console via UART (at least when booting Debian without Xen, see
>>>> hypothesis 3 below).
>>>>
>>>> Booting Debian without Xen is working fine. (Although I only see the
>>>> Grub selection menu via UART when no HDMI is plugged in while booting.)
>>>>
>>>> Selecting Xen in Grub leads to the following output on the console
>>>> right after Grub hands over:
>>>>
>>>> Loading Xen 4.17-arm64.efi ...
>>>> Loading Linux 6.1.0-10-arm64 ...
>>>> Loading initial ramdisk ...
>>>> Using modules provided by bootloader in FDT
>>>> Xen 4.17.2-pre (c/s ) EFI loader
>>>>
>>>> And then the output stops.
>>>>
>>>
>>> I have been using Debian + Xen + Grub + UEFI on Rpi4. UEFI firmwares
>>> are mostly shipped with ACPI only (IOW no Device-Tree support) This
>>> is not yet a fully supported configuration on Arm64 and therefore not
>>> enabled by default.
>>>
>>> Can you check the kernel log when booting Debian without Xen for
>>> anything referring to ACPI? If so, you will need to Xen build
>>> yourself to enable ACPI (this is protected by EXPERT). Alternatively
>>> you could use U-boot where Device-Tree boot will be available.
>>
>> Indeed, the device is using ACPI. There is an option in the UEFI
>> settings to switch to Device-Tree, but that didn't have any effect I
>> could see, other than plain Debian didn't boot anymore either.
>>
>> I compiled Xen with the ACPI option for Arm, and got it running. Thank
>> you very much for your hint, Julien!
>>
>> For reference:
>>
>> To enable the ACPI option, I went to the "xen" subdirectory in the Xen
>> source main directory after "git clone" and did a "make menuconfig".
>>
>> Out of convience, I initially intended to only rebuild the Debian Xen
>> hypervisor package with the ACPI switch on, but that didn't work
>> somehow. I couldn't find out why.
>
> Is Debian using 4.17.2? Did you see any output from Xen/Linux.

xl info on a standard Debian installation gives me version 4.17.2-pre.
If I remember correctly, the output when booting both my self-compiled
Debian xen package or the self-compiled 17.1 version is identical to the
one above, it stops with:

Using modules provided by bootloader in FDT
Xen 4.17.2-pre (c/s ) EFI loader

>> I then created a "Frankenxen" with a newly built hypervisor and
>> standard debian tools, which booted (yay!), but of course the xen
>> tools didn't work. Using the repo to check out a Xen 17.1 version and
>> build the hypervisor (so I could still rely on Debian updates for all
>> the other Xen packages) also didn't work, Xen didn't boot. I then
>> decided to build a complete Xen version from scratch from the master
>> repo, which worked well. So perhaps this approach with enabling ACPI
>> only works on Xen versions > 17.1?
>
> I don't remember any change related to ACPI on Arm that are in master
> but not 4.17. I will try to have a look in the next couple of weeks and
> find which patches may be missing.

Let me know if I can help.

Paul

Re: Xen on Raspberry Pi 4 [ In reply to ]

julien at xen

Oct 22, 2023, 10:43 AM

Post #6 of 7 (174 views)

Permalink

Hi Paul

Sorry for the late reply.

On 13/08/2023 18:46, Paul Leiber wrote:
> Am 11.08.2023 um 20:51 schrieb Julien Grall:
>> Hi Paul,
>>
>> On 04/08/2023 06:44, Paul Leiber wrote:
>>> Am 02.08.2023 um 23:03 schrieb Julien Grall:
>>>>
>>>>
>>>> On 02/08/2023 21:42, Paul Leiber wrote:
>>>>> Hi list,
>>>>
>>>> Hi Paul,
>>>>
>>>>>
>>>>> What I did was to install a standard Debian distribution booted via
>>>>> EFI/GRUB2 (following
>>>>> https://forums.raspberrypi.com/viewtopic.php?t=282839) and then
>>>>> install the xen-System-arm64 package via apt. I have output on a
>>>>> console via UART (at least when booting Debian without Xen, see
>>>>> hypothesis 3 below).
>>>>>
>>>>> Booting Debian without Xen is working fine. (Although I only see
>>>>> the Grub selection menu via UART when no HDMI is plugged in while
>>>>> booting.)
>>>>>
>>>>> Selecting Xen in Grub leads to the following output on the console
>>>>> right after Grub hands over:
>>>>>
>>>>> Loading Xen 4.17-arm64.efi ...
>>>>> Loading Linux 6.1.0-10-arm64 ...
>>>>> Loading initial ramdisk ...
>>>>> Using modules provided by bootloader in FDT
>>>>> Xen 4.17.2-pre (c/s ) EFI loader
>>>>>
>>>>> And then the output stops.
>>>>>
>>>>
>>>> I have been using Debian + Xen + Grub + UEFI on Rpi4. UEFI firmwares
>>>> are mostly shipped with ACPI only (IOW no Device-Tree support) This
>>>> is not yet a fully supported configuration on Arm64 and therefore
>>>> not enabled by default.
>>>>
>>>> Can you check the kernel log when booting Debian without Xen for
>>>> anything referring to ACPI? If so, you will need to Xen build
>>>> yourself to enable ACPI (this is protected by EXPERT). Alternatively
>>>> you could use U-boot where Device-Tree boot will be available.
>>>
>>> Indeed, the device is using ACPI. There is an option in the UEFI
>>> settings to switch to Device-Tree, but that didn't have any effect I
>>> could see, other than plain Debian didn't boot anymore either.
>>>
>>> I compiled Xen with the ACPI option for Arm, and got it running.
>>> Thank you very much for your hint, Julien!
>>>
>>> For reference:
>>>
>>> To enable the ACPI option, I went to the "xen" subdirectory in the
>>> Xen source main directory after "git clone" and did a "make menuconfig".
>>>
>>> Out of convience, I initially intended to only rebuild the Debian Xen
>>> hypervisor package with the ACPI switch on, but that didn't work
>>> somehow. I couldn't find out why.
>>
>> Is Debian using 4.17.2? Did you see any output from Xen/Linux.
>
> xl info on a standard Debian installation gives me version 4.17.2-pre.
> If I remember correctly, the output when booting both my self-compiled
> Debian xen package or the self-compiled 17.1 version is identical to the
> one above, it stops with:
>
> Using modules provided by bootloader in FDT
> Xen 4.17.2-pre (c/s ) EFI loader
>
>>> I then created a "Frankenxen" with a newly built hypervisor and
>>> standard debian tools, which booted (yay!), but of course the xen
>>> tools didn't work. Using the repo to check out a Xen 17.1 version and
>>> build the hypervisor (so I could still rely on Debian updates for all
>>> the other Xen packages) also didn't work, Xen didn't boot. I then
>>> decided to build a complete Xen version from scratch from the master
>>> repo, which worked well. So perhaps this approach with enabling ACPI
>>> only works on Xen versions > 17.1?
>>
>> I don't remember any change related to ACPI on Arm that are in master
>> but not 4.17. I will try to have a look in the next couple of weeks
>> and find which patches may be missing.
>
> Let me know if I can help.

I finally managed to find time to try Xen 4.17.2 on the RPI4. It seems
to work without any issue once I have ACPI enabled.

Is this still a problem for you? If so, would you be able to share your
.config and the Linux kernel version you are using?

Do you also have any logs to share?

Cheers,

--
Julien Grall

Re: Xen on Raspberry Pi 4 [ In reply to ]

paul at onlineschubla

Jan 14, 2024, 8:24 AM

Post #7 of 7 (102 views)

Permalink

Am 22.10.2023 um 19:43 schrieb Julien Grall:
> Hi Paul
>
> Sorry for the late reply.
>
> On 13/08/2023 18:46, Paul Leiber wrote:
>> Am 11.08.2023 um 20:51 schrieb Julien Grall:
>>> Hi Paul,
>>>
>>> On 04/08/2023 06:44, Paul Leiber wrote:
>>>> Am 02.08.2023 um 23:03 schrieb Julien Grall:
>>>>>
>>>>>
>>>>> On 02/08/2023 21:42, Paul Leiber wrote:
>>>>>> Hi list,
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>>>
>>>>>> What I did was to install a standard Debian distribution booted
>>>>>> via EFI/GRUB2 (following
>>>>>> https://forums.raspberrypi.com/viewtopic.php?t=282839) and then
>>>>>> install the xen-System-arm64 package via apt. I have output on a
>>>>>> console via UART (at least when booting Debian without Xen, see
>>>>>> hypothesis 3 below).
>>>>>>
>>>>>> Booting Debian without Xen is working fine. (Although I only see
>>>>>> the Grub selection menu via UART when no HDMI is plugged in while
>>>>>> booting.)
>>>>>>
>>>>>> Selecting Xen in Grub leads to the following output on the console
>>>>>> right after Grub hands over:
>>>>>>
>>>>>> Loading Xen 4.17-arm64.efi ...
>>>>>> Loading Linux 6.1.0-10-arm64 ...
>>>>>> Loading initial ramdisk ...
>>>>>> Using modules provided by bootloader in FDT
>>>>>> Xen 4.17.2-pre (c/s ) EFI loader
>>>>>>
>>>>>> And then the output stops.
>>>>>>
>>>>>
>>>>> I have been using Debian + Xen + Grub + UEFI on Rpi4. UEFI
>>>>> firmwares are mostly shipped with ACPI only (IOW no Device-Tree
>>>>> support) This is not yet a fully supported configuration on Arm64
>>>>> and therefore not enabled by default.
>>>>>
>>>>> Can you check the kernel log when booting Debian without Xen for
>>>>> anything referring to ACPI? If so, you will need to Xen build
>>>>> yourself to enable ACPI (this is protected by EXPERT).
>>>>> Alternatively you could use U-boot where Device-Tree boot will be
>>>>> available.
>>>>
>>>> Indeed, the device is using ACPI. There is an option in the UEFI
>>>> settings to switch to Device-Tree, but that didn't have any effect I
>>>> could see, other than plain Debian didn't boot anymore either.
>>>>
>>>> I compiled Xen with the ACPI option for Arm, and got it running.
>>>> Thank you very much for your hint, Julien!
>>>>
>>>> For reference:
>>>>
>>>> To enable the ACPI option, I went to the "xen" subdirectory in the
>>>> Xen source main directory after "git clone" and did a "make
>>>> menuconfig".
>>>>
>>>> Out of convience, I initially intended to only rebuild the Debian
>>>> Xen hypervisor package with the ACPI switch on, but that didn't work
>>>> somehow. I couldn't find out why.
>>>
>>> Is Debian using 4.17.2? Did you see any output from Xen/Linux.
>>
>> xl info on a standard Debian installation gives me version 4.17.2-pre.
>> If I remember correctly, the output when booting both my self-compiled
>> Debian xen package or the self-compiled 17.1 version is identical to
>> the one above, it stops with:
>>
>> Using modules provided by bootloader in FDT
>> Xen 4.17.2-pre (c/s ) EFI loader
>>
>>>> I then created a "Frankenxen" with a newly built hypervisor and
>>>> standard debian tools, which booted (yay!), but of course the xen
>>>> tools didn't work. Using the repo to check out a Xen 17.1 version
>>>> and build the hypervisor (so I could still rely on Debian updates
>>>> for all the other Xen packages) also didn't work, Xen didn't boot. I
>>>> then decided to build a complete Xen version from scratch from the
>>>> master repo, which worked well. So perhaps this approach with
>>>> enabling ACPI only works on Xen versions > 17.1?
>>>
>>> I don't remember any change related to ACPI on Arm that are in master
>>> but not 4.17. I will try to have a look in the next couple of weeks
>>> and find which patches may be missing.
>>
>> Let me know if I can help.
>
> I finally managed to find time to try Xen 4.17.2 on the RPI4. It seems
> to work without any issue once I have ACPI enabled.
>
> Is this still a problem for you? If so, would you be able to share your
> .config and the Linux kernel version you are using?
>
> Do you also have any logs to share?

Sorry for not getting back to this earlier. For the record: I now tried
again with Xen 4.17.3, and everything seems to be working properly (that
is, apart from the networking problems / xen crashes described in
another thread).

I suspect I didn't execute the right git commands previously when
building Xen 4.17.2-pre.

Thank you for looking into this!

Paul