Mailing List Archive: black screen troubles trying to boot with Xen Hypervisor grub option

black screen troubles trying to boot with Xen Hypervisor grub option

Jul 1, 2023, 9:40 AM

Post #1 of 7 (216 views)

I'm trying to work my way through the beginner guide (
https://wiki.xenproject.org/wiki/Xen_Project_Beginners_Guide ). Things
seem fine all the way down to step 5 "Installing the Xen Project
Software". I do the reboot and select the option with Xen Hypervisor
and instead of getting a console login I get a black screen. The
machine doesn't have a serial port or SOL so I took the suggestion on
the serial console page saying "you could always record a video of the
boot process". Then I transcribed it (which means there might be some
typos). From my limited understanding the log seems to have some
warnings and suggestions about possible changes to make things better,
but nothing that seems to indicate I'm about to be dumped into a black
screen. Log below:

-----------------------------

(XEN) ACPI: 32/64X FACS address mismatch in FADT -
daf5a000/0000000000000000, using 32
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) PCI: Not using MCFG for segment 0000 bus 00-3f
(XEN) Switched to APIC driver x2apic_cluster
(XEN) CPU0: 1200 ... 2600 MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) Speculative mitigation facilities:
(XEN)   Hardware hints:
(XEN)   Hardware features:
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: No, Other:
BRANCH_HARDEN
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 36, Safe
address 1000000000
(XEN)   Support for HVM VMs: RSB EAGER_FPU
(XEN)   XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU enabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 2594.107 MHz processor.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tabled not enabled.
(XEN) I/O virtualization enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Enabling APIC mode: Clustered. Using 1 I/O APICs
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) HVM: ASIDs enabled
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes, 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Initializing Credit2 scheduler
(XEN) Dom0 has maximum 648 PIRQs
(XEN) Xen kernel: 64-bit, lsb
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4a00000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.:   0000000208000000->0000000210000000 (1948531 pages
to be allocated)
(XEN) Init. ramdisk: 000000032c0B4000->000000021e5ffd12
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff84a00000
(XEN) Phys-Mach map: 0000008000000000->0000008000f30778
(XEN) Start info:    ffffffff84a00000->ffffffff84a004b8
(XEN) Page tables:   ffffffff84a01000->ffffffff84a2a000
(XEN) Boot stack:    ffffffff84a2a000->ffffffff84a2b000
(XEN) TOTAL:         ffffffff80000000->ffffffff84c00000
(XEN) ENTRY ADDRESS: ffffffff8306e1c0
(XEN) Dom0 has maximum 4 VCPUS
(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) ***************************************************
(XEN) Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
(XEN) enabled. Please assess your configuration and choose an
(XEN) explicit 'smt=<bool>' setting. See XSA-273.
(XEN) ***************************************************
(XEN) Booted on MLPDS/MFBDS-vulnerable hardware with SMT/Hyperthreading
(XEN) enabled. Mitigations will not be fully effective. Please
(XEN) choose an explicit smt=<bool> setting. See XSA-297
(XEN) ****************************************************
(XEN) 3... 2... 1...

-----------------------------

-Jamie

Re: black screen troubles trying to boot with Xen Hypervisor grub option [ In reply to ]

pbraun at nethence

Jul 8, 2023, 11:08 PM

Post #2 of 7 (210 views)

Permalink

> (XEN) 3... 2... 1...

Hello Jamie, so question is, what does your GRUB config look like
(without the comments please), so we can see if you are having serial
console specifics as for booting the linux kernel and userland.

There are mainly three steps in booting a dom0:

1. xen micro-kernel
2. linux kernel (and eventually initramfs)
3. userland and getty

all three can have different output. Here's a few hints on how to
tackle grub.conf manually.

https://pub.nethence.com/booting/grub

-elge

Re: black screen troubles trying to boot with Xen Hypervisor grub option [ In reply to ]

jamie at parit

Jul 9, 2023, 1:20 PM

Post #3 of 7 (208 views)

Permalink

Hello Elge,

I've attached the grub config prior to the xen metapackage and also the
grub config after xen metapackage install. The pre-xen config might not
be relevant but I'm attaching it in case the diff does turn out to have
relevance. Comments scrubbed from both.

You mention serial console but in my case the computer has no physical
serial port nor the virtual SOL approach.

While digging in to what's going on I also noticed that the base
hypervisor entry has a conditional branch checking $grub_platform so I
added an echo to figure out which branch it's picking, and it turns out
that in my case $grub_platform isn't "pc" and isn't empty but rather is
"efi" so it does the "no-real-mode edd=off" options. When I read about
them in the documentation says that no-real-mode should only be used for
debugging because it prevents vga from working, and I got excited
because I thought "and therefore black screen!" but when I force it to
use no options, like the "pc" branch does, it still gets black screen.

-Jamie

On 2023-07-08 11:08 p.m., Pierre-Philipp Braun wrote:
>> (XEN) 3... 2... 1...
>
> Hello Jamie, so question is, what does your GRUB config look like
> (without the comments please), so we can see if you are having serial
> console specifics as for booting the linux kernel and userland.
>
> There are mainly three steps in booting a dom0:
>
> 1. xen micro-kernel
> 2. linux kernel (and eventually initramfs)
> 3. userland and getty
>
> all three can have different output. Here's a few hints on how to
> tackle grub.conf manually.
>
> https://pub.nethence.com/booting/grub
>
> -elge
>

Re: black screen troubles trying to boot with Xen Hypervisor grub option [ In reply to ]

pbraun at nethence

Jul 9, 2023, 11:57 PM

Post #4 of 7 (208 views)

Permalink

> While digging in to what's going on I also noticed that the base
> hypervisor entry has a conditional branch checking $grub_platform so I
> added an echo to figure out which branch it's picking, and it turns out
> that in my case $grub_platform isn't "pc" and isn't empty but rather is
> "efi" so it does the "no-real-mode edd=off" options. When I read about
> them in the documentation says that no-real-mode should only be used for
> debugging because it prevents vga from working, and I got excited
> because I thought "and therefore black screen!" but when I force it to
> use no options, like the "pc" branch does, it still gets black screen.

About no-real-mode edd=off, you might be right. But to test without it, I would go with my own grub.cfg by overwriting it altogether, to keep it simple (keep insmod part_gpt as for EFI). I find interesting that there is usually `insmod all_video` or a call to function `load_video` with the valid entries (EFI capable), but those are missing in the XEN entries.

So this is the relevant part:

menuentry 'Debian GNU/Linux, with Xen hypervisor' --class debian --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-0e276b18-54a4-4ff9-a770-1bcdd946ea70' {
insmod part_gpt
insmod ext2
set root='hd0,gpt1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt1 --hint-efi=hd0,gpt1 --hint-baremetal=ahci0,gpt1 1dd68754-c4eb-4011-98d4-6df73f99eb81
else
search --no-floppy --fs-uuid --set=root 1dd68754-c4eb-4011-98d4-6df73f99eb81
fi
echo 'Loading Xen 4.17-amd64 ...'
if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
xen_rm_opts=
else
xen_rm_opts="no-real-mode edd=off"
fi
multiboot2 /xen-4.17-amd64.gz placeholder ${xen_rm_opts}
echo 'Loading Linux 6.1.0-10-amd64 ...'
module2 /vmlinuz-6.1.0-10-amd64 placeholder root=UUID=0e276b18-54a4-4ff9-a770-1bcdd946ea70 ro quiet
echo 'Loading initial ramdisk ...'
module2 --nounzip /initrd.img-6.1.0-10-amd64
}

You could try add load_video in top of that and reboot (and without re-generating the entire thing with update-grub).

Besides, note there used to be an issue on Debian systems while setting up XEN. One had to elevate the boot priority.

dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
update-grub

However I am not sure this is still the case. Also, to get the login prompt, some console needs to point hvc0, and I don't see any specific console= argument to the kernel here. You can also disable the casual serial console prompt altogether, since you don't have any.

systemctl disable serial-getty@ttyS0

It's usually harder to setup XEN on EFI systems compared to legacy MBR boot process. The easy way would be to enable CSM in your firmware and go MBR.

Re: black screen troubles trying to boot with Xen Hypervisor grub option [ In reply to ]

jamie at parit

Jul 10, 2023, 10:02 AM

Post #5 of 7 (206 views)

Permalink

I tried load_video, serial disable, and diverting I still got black
screen but then I tried your suggestion of going into BIOS and just
totally suppressing EFI and forcing the system to do strictly legacy
stuff and now it actually boots! I'm curious as to whether the EFI
approach could still get to a bootable system, but not curious enough to
re-break it and be stuck again. Thanks for getting me to bootability!

-Jamie

On 2023-07-09 11:57 p.m., Pierre-Philipp Braun wrote:
>> While digging in to what's going on I also noticed that the base
>> hypervisor entry has a conditional branch checking $grub_platform so
>> I added an echo to figure out which branch it's picking, and it turns
>> out that in my case $grub_platform isn't "pc" and isn't empty but
>> rather is "efi" so it does the "no-real-mode edd=off" options. When
>> I read about them in the documentation says that no-real-mode should
>> only be used for debugging because it prevents vga from working, and
>> I got excited because I thought "and therefore black screen!" but
>> when I force it to use no options, like the "pc" branch does, it
>> still gets black screen.
>
> About no-real-mode edd=off, you might be right. But to test without
> it, I would go with my own grub.cfg by overwriting it altogether, to
> keep it simple (keep insmod part_gpt as for EFI). I find interesting
> that there is usually `insmod all_video` or a call to function
> `load_video` with the valid entries (EFI capable), but those are
> missing in the XEN entries.
>
> So this is the relevant part:
>
> menuentry 'Debian GNU/Linux, with Xen hypervisor' --class debian
> --class gnu-linux --class gnu --class os --class xen
> $menuentry_id_option
> 'xen-gnulinux-simple-0e276b18-54a4-4ff9-a770-1bcdd946ea70' {
>     insmod part_gpt
>     insmod ext2
>     set root='hd0,gpt1'
>     if [ x$feature_platform_search_hint = xy ]; then
>       search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt1
> --hint-efi=hd0,gpt1 --hint-baremetal=ahci0,gpt1
> 1dd68754-c4eb-4011-98d4-6df73f99eb81
>     else
>       search --no-floppy --fs-uuid --set=root
> 1dd68754-c4eb-4011-98d4-6df73f99eb81
>     fi
>     echo    'Loading Xen 4.17-amd64 ...'
>         if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
>             xen_rm_opts=
>         else
>             xen_rm_opts="no-real-mode edd=off"
>         fi
>     multiboot2    /xen-4.17-amd64.gz placeholder   ${xen_rm_opts}
>     echo    'Loading Linux 6.1.0-10-amd64 ...'
>     module2    /vmlinuz-6.1.0-10-amd64 placeholder
> root=UUID=0e276b18-54a4-4ff9-a770-1bcdd946ea70 ro quiet
>     echo    'Loading initial ramdisk ...'
>     module2    --nounzip   /initrd.img-6.1.0-10-amd64
> }
>
> You could try add load_video in top of that and reboot (and without
> re-generating the entire thing with update-grub).
>
> Besides, note there used to be an issue on Debian systems while
> setting up XEN. One had to elevate the boot priority.
>
>     dpkg-divert --divert /etc/grub.d/08_linux_xen --rename
> /etc/grub.d/20_linux_xen
>         update-grub
>
> However I am not sure this is still the case. Also, to get the login
> prompt, some console needs to point hvc0, and I don't see any specific
> console= argument to the kernel here. You can also disable the casual
> serial console prompt altogether, since you don't have any.
>
>         systemctl disable serial-getty@ttyS0
>
> It's usually harder to setup XEN on EFI systems compared to legacy MBR
> boot process. The easy way would be to enable CSM in your firmware
> and go MBR.

Re: black screen troubles trying to boot with Xen Hypervisor grub option [ In reply to ]

slack at rabbit

Jul 10, 2023, 3:47 PM

Post #6 of 7 (205 views)

Permalink

On 10 Jul 2023 19:02, Jamie Campbell wrote:
> I tried load_video, serial disable, and diverting I still got black
> screen but then I tried your suggestion of going into BIOS and just
> totally suppressing EFI and forcing the system to do strictly legacy
> stuff and now it actually boots! I'm curious as to whether the EFI
> approach could still get to a bootable system, but not curious enough to
> re-break it and be stuck again. Thanks for getting me to bootability!
>
> -Jamie
>

(Please bottom-post (; )

I had the same problem booting Xen+Debian on UEFI, found a solution,
maybe it applies to you ?

The problem started when I updated dom0 from Stretch to Buster (at the
time Buster became stable). With the new version of
"/etc/grub.d/20_linux_xen", no more boot.

If you compare the old (stretch) and the actual files (from buster on),
some changes are about multiboot/module to multiboot2/module2 and in the
check you pointed ($grub_platform isn't "pc").
But as you may have eliminated the $grub_platform problem, maybe it's
the use of multiboot versus multiboot2 ?

What I would do is move or rename "/etc/grub.d/20_linux_xen", and then
try with the Stretch version of the file.
(If just renaming, I recommend you to edit the "title" stanzas to better
identify the menus : you'd have identical lines).

If you want to rambo edit "20_linux_xen" in place, below you will find
the concerning differences between two versions.
Basically replace ${xen_loader} and ${module_loader} by respectively
multiboot and module.

I'm still booting a bookworm dom0 with multiboot/module, YMMV !

---
WORKING VERSION
[...]
if [. "\$grub_platform" = "pc" -o "\$grub_platform" = "efi" -o
"\$grub_platform" = "" ]; then
xen_rm_opts=
else
xen_rm_opts="no-real-mode edd=off"
fi
multiboot ${rel_xen_dirname}/${xen_basename} placeholder ${xen_args}
\${xen_rm_opts}
echo '$(echo "$lmessage" | grub_quote)'
module ${rel_dirname}/${basename} placeholder
root=${linux_root_device_thisversion} ro ${args}
[...]
module --nounzip ${rel_dirname}/${initrd}
[...]

---
NON-WORKING VERSION
[...]
if [ "\$grub_platform" = "pc" -o "\$grub_platform" = "" ]; then
xen_rm_opts=
else
xen_rm_opts="no-real-mode edd=off"
fi
${xen_loader} ${rel_xen_dirname}/${xen_basename} placeholder
${xen_args} \${xen_rm_opts}
echo '$(echo "$lmessage" | grub_quote)'
${module_loader} ${rel_dirname}/${basename} placeholder
root=${linux_root_device_thisversion} ro ${args}
[...]
${module_loader} --nounzip ${rel_dirname}/${initrd}
[...]
if ($grub_file --is-x86-multiboot2 $current_xen); then
xen_loader="multiboot2"
module_loader="module2"
else
xen_loader="multiboot"
module_loader="module"
fi
[...]

Re: black screen troubles trying to boot with Xen Hypervisor grub option [ In reply to ]

jamie at parit

Jul 10, 2023, 5:38 PM

Post #7 of 7 (205 views)

Permalink

On 2023-07-10 3:47 p.m., zithro wrote:
> On 10 Jul 2023 19:02, Jamie Campbell wrote:
>> I tried load_video, serial disable, and diverting I still got black
>> screen but then I tried your suggestion of going into BIOS and just
>> totally suppressing EFI and forcing the system to do strictly legacy
>> stuff and now it actually boots! I'm curious as to whether the EFI
>> approach could still get to a bootable system, but not curious enough
>> to re-break it and be stuck again. Thanks for getting me to
>> bootability!
>>
>> -Jamie
>>
>
> (Please bottom-post (; )
>
> I had the same problem booting Xen+Debian on UEFI, found a solution,
> maybe it applies to you ?
>
> The problem started when I updated dom0 from Stretch to Buster (at the
> time Buster became stable). With the new version of
> "/etc/grub.d/20_linux_xen", no more boot.
>
> If you compare the old (stretch) and the actual files (from buster
> on), some changes are about multiboot/module to multiboot2/module2 and
> in the check you pointed ($grub_platform isn't "pc").
> But as you may have eliminated the $grub_platform problem, maybe it's
> the use of multiboot versus multiboot2 ?
>
> What I would do is move or rename "/etc/grub.d/20_linux_xen", and then
> try with the Stretch version of the file.
> (If just renaming, I recommend you to edit the "title" stanzas to
> better identify the menus : you'd have identical lines).
>
> If you want to rambo edit "20_linux_xen" in place, below you will find
> the concerning differences between two versions.
> Basically replace ${xen_loader} and ${module_loader} by respectively
> multiboot and module.
>
> I'm still booting a bookworm dom0 with multiboot/module, YMMV !
>
> ---
> WORKING VERSION
> [...]
> if [. "\$grub_platform" = "pc" -o "\$grub_platform" = "efi" -o
> "\$grub_platform" = "" ]; then
>             xen_rm_opts=
>         else
>             xen_rm_opts="no-real-mode edd=off"
>         fi
>     multiboot    ${rel_xen_dirname}/${xen_basename} placeholder
> ${xen_args} \${xen_rm_opts}
>     echo    '$(echo "$lmessage" | grub_quote)'
>     module    ${rel_dirname}/${basename} placeholder
> root=${linux_root_device_thisversion} ro ${args}
> [...]
>     module    --nounzip   ${rel_dirname}/${initrd}
> [...]
>
> ---
> NON-WORKING VERSION
> [...]
> if [ "\$grub_platform" = "pc" -o "\$grub_platform" = "" ]; then
>             xen_rm_opts=
>         else
>             xen_rm_opts="no-real-mode edd=off"
>         fi
>     ${xen_loader}    ${rel_xen_dirname}/${xen_basename} placeholder
> ${xen_args} \${xen_rm_opts}
>     echo    '$(echo "$lmessage" | grub_quote)'
>     ${module_loader}    ${rel_dirname}/${basename} placeholder
> root=${linux_root_device_thisversion} ro ${args}
> [...]
>     ${module_loader}    --nounzip   ${rel_dirname}/${initrd}
> [...]
> if ($grub_file --is-x86-multiboot2 $current_xen); then
>     xen_loader="multiboot2"
>     module_loader="module2"
>     else
>     xen_loader="multiboot"
>     module_loader="module"
>     fi
> [...]
>

I re-installed with EFI stuff on again, tested that it black screens the
same was as before, then removed the 2s (module, multiboot) and did a
reboot, and it locks up when it gets to "Loading initial ramdisk", and
also warns that "no console will be available to OS". I'll go back to
legacy again, but it was worth a shot, thanks for the suggestion, and
sorry for the netiquette oops!

-Jamie

Mailing List Archive

Mailing List Archive

Attached Files: