Hello,
I am using Xen for Desktop virtualization of Windows and Linux
on an older Intel box with a Haswell (4th generation) Intel
Integrated Graphics Device (IGD) for graphics which is
now fairly old hardware.
I use Debian for the dom0 and I also use Xen's Intel IGD
passthrough feature for accelerated graphics in an HVM domU.
With up-to-date Debian stable version 11.1 (Bullseye) for dom0,
PCI/IGD passthrough works well with Windows HVM domUs, but
not with modern Linux HVM domUs. I reported this several
months ago as a bug to the Debian BTS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988333
A quick synopsis of that bug report:
In the dom0:
Debian 11.1 Bullseye version of Xen: 4.14.3-1~deb11u1
Debian 11.1 Bullseye version of Qemu: 5.2+dfsg-11+deb11u1
Debian 11.1 Bullseye version of Linux kernel: 5.10.70-1
Two kinds of guests configured for passthrough (a domain config
I use for a Debian 11.1 HVM is added to the end of this message):
1. Windows 10 HVM domU: I can configure it to work well with
passthrough of the Intel IGD, the on-board USB card, and the
sound card to Windows 10, but only with the rdm_policy set
to relaxed hack with my old hardware that, AFAICT, does not
isolate passthrough devices optimally.
2. Bullseye Debian 11.1 HVM domU: Passthrough with Intel IGD, USB card
and sound card does not work at all
The main problem I see in the Debian 11.1 dom0 Journal when starting
a Debian 11.1 HVM domU configured for PCI passthrough of the Intel
IGD, the USB 3 controller, and the sound card is that IRQ 16 is
disabled:
----------------- Start of Journal snippet from dom0 --------------
dom0 kernel: irq 16: nobody cared ...
...
dom0 kernel: Call Trace:
dom0 kernel: <IRQ>
dom0 kernel: dump_stack+0x6b/0x83
dom0 kernel: __report_bad_irq+0x35/0xa7
dom0 kernel: note_interrupt.cold+0xb/0x61
dom0 kernel: handle_irq_event+0xa8/0xb0
...
dom0 kernel: handlers:
dom0 kernel: [<00000000ad934388>] usb_hcd_irq [usbcore]
dom0 kernel: [<000000003087e3ca>] ath_isr [ath9k]
dom0 kernel: Disabling IRQ #16
---------------- End of Journal snippet from dom0 ---------------
From what I have read this is caused by an IRQ being assigned
to a device that does not respond to the IRQ. In the dom0, IRQ
#16 has been disabled according to the log, but also according
to the logs, IRQ #16 is assigned to a PCI device that I do not
passthrough. I am thinking perhaps this device assigned to
IRQ #16, which is a USB 2 controller, belongs to the same
IOMMU group (in VFIO terminology) the USB 3 controller that
I do pass through belongs to:
dom0 kernel: ehci-pci 0000:00:1a.0: irq 16, io mem 0xf053c000
0000:00:14.0 is the usb 3 controller (xhci) I do passthrough.
So I think the problem with the Linux HVM domU is caused by the
Linux device drivers in the domU HVM not being as able to deal
with the poor isolation of the devices on this old box as well
as the Windows drivers do. Am I on the right track? Can this be
solved by passing through both this USB 2 controller and the
USB 3 controller to the domU?
A few more questions:
1. Does anyone report Intel IGD passthrough with a Debian 11
dom0 and a Debian 11 domU working on Xen with newer Intel
hardware, such as 8th generation or newer?
2. Since IGD passthrough works fine with a Windows HVM, and
I also discovered [0] an unmodified Debian 11.1 HVM domU works
fine with a slightly modified old version of Debian 8 for
the dom0, which uses the old Xen 4.4 hypervisor, the old
Linux 3.16 kernel, and the old qemu traditional device model
for PCI/IGD passthrough, does anyone know if it is possible
to configure or patch the modern Xen 4.14 hypervisor and/or
the modern qemu upstream device model to get a modern Debian
11.1 HVM domU to enable PCI/IGD passthrough to work with
newer software running in dom0 than what is available from
the old Debian 8 release that is now way beyond EOL? I
would prefer to be able to get PCI/IGD passthrough to Debian
11.1 to work on a supported dom0 configuration such as a
Debian 11.1 dom0 with Xen 4.14 instead of unsupported
Debian 8.x dom0 with unsupported Xen 4.4.
3. Is this problem likely to be only limited to certain hardware
that does not isolate the devices as well as modern Linux-based
software expects?
Thanks in advance for any advice you can give me.
Regards,
Chuck Zmudzinski
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988333#10
My domain config file (private data such as mac and ip addresses
are redacted):
builder = 'hvm'
memory = '3072'
vcpus = '4'
device_model_version = 'qemu-xen'
# device_model_version = 'qemu-xen-traditional'
# This is a bullseye system (Debian 11.1)
disk = ['/dev/systems/linux,,xvda,w','/dev/data/linuxdata,,xvdb,w']
name = 'bullseye-hvm'
vif = [ 'mac=xx:xx:xx:xx:xx:xx,model=e1000,script=vif-route,ip=A.B.X.Y' ]
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
boot = 'c'
acpi = '1'
apic = '1'
viridian = '1'
xen_platform_pci = '1'
serial = 'pty'
vga = 'none'
sdl = '0'
vnc = '0'
gfx_passthru = '1'
pci = [ '00:1b.0', '00:14.0,rdm_policy=relaxed', '00:02.0' ]
I am using Xen for Desktop virtualization of Windows and Linux
on an older Intel box with a Haswell (4th generation) Intel
Integrated Graphics Device (IGD) for graphics which is
now fairly old hardware.
I use Debian for the dom0 and I also use Xen's Intel IGD
passthrough feature for accelerated graphics in an HVM domU.
With up-to-date Debian stable version 11.1 (Bullseye) for dom0,
PCI/IGD passthrough works well with Windows HVM domUs, but
not with modern Linux HVM domUs. I reported this several
months ago as a bug to the Debian BTS:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988333
A quick synopsis of that bug report:
In the dom0:
Debian 11.1 Bullseye version of Xen: 4.14.3-1~deb11u1
Debian 11.1 Bullseye version of Qemu: 5.2+dfsg-11+deb11u1
Debian 11.1 Bullseye version of Linux kernel: 5.10.70-1
Two kinds of guests configured for passthrough (a domain config
I use for a Debian 11.1 HVM is added to the end of this message):
1. Windows 10 HVM domU: I can configure it to work well with
passthrough of the Intel IGD, the on-board USB card, and the
sound card to Windows 10, but only with the rdm_policy set
to relaxed hack with my old hardware that, AFAICT, does not
isolate passthrough devices optimally.
2. Bullseye Debian 11.1 HVM domU: Passthrough with Intel IGD, USB card
and sound card does not work at all
The main problem I see in the Debian 11.1 dom0 Journal when starting
a Debian 11.1 HVM domU configured for PCI passthrough of the Intel
IGD, the USB 3 controller, and the sound card is that IRQ 16 is
disabled:
----------------- Start of Journal snippet from dom0 --------------
dom0 kernel: irq 16: nobody cared ...
...
dom0 kernel: Call Trace:
dom0 kernel: <IRQ>
dom0 kernel: dump_stack+0x6b/0x83
dom0 kernel: __report_bad_irq+0x35/0xa7
dom0 kernel: note_interrupt.cold+0xb/0x61
dom0 kernel: handle_irq_event+0xa8/0xb0
...
dom0 kernel: handlers:
dom0 kernel: [<00000000ad934388>] usb_hcd_irq [usbcore]
dom0 kernel: [<000000003087e3ca>] ath_isr [ath9k]
dom0 kernel: Disabling IRQ #16
---------------- End of Journal snippet from dom0 ---------------
From what I have read this is caused by an IRQ being assigned
to a device that does not respond to the IRQ. In the dom0, IRQ
#16 has been disabled according to the log, but also according
to the logs, IRQ #16 is assigned to a PCI device that I do not
passthrough. I am thinking perhaps this device assigned to
IRQ #16, which is a USB 2 controller, belongs to the same
IOMMU group (in VFIO terminology) the USB 3 controller that
I do pass through belongs to:
dom0 kernel: ehci-pci 0000:00:1a.0: irq 16, io mem 0xf053c000
0000:00:14.0 is the usb 3 controller (xhci) I do passthrough.
So I think the problem with the Linux HVM domU is caused by the
Linux device drivers in the domU HVM not being as able to deal
with the poor isolation of the devices on this old box as well
as the Windows drivers do. Am I on the right track? Can this be
solved by passing through both this USB 2 controller and the
USB 3 controller to the domU?
A few more questions:
1. Does anyone report Intel IGD passthrough with a Debian 11
dom0 and a Debian 11 domU working on Xen with newer Intel
hardware, such as 8th generation or newer?
2. Since IGD passthrough works fine with a Windows HVM, and
I also discovered [0] an unmodified Debian 11.1 HVM domU works
fine with a slightly modified old version of Debian 8 for
the dom0, which uses the old Xen 4.4 hypervisor, the old
Linux 3.16 kernel, and the old qemu traditional device model
for PCI/IGD passthrough, does anyone know if it is possible
to configure or patch the modern Xen 4.14 hypervisor and/or
the modern qemu upstream device model to get a modern Debian
11.1 HVM domU to enable PCI/IGD passthrough to work with
newer software running in dom0 than what is available from
the old Debian 8 release that is now way beyond EOL? I
would prefer to be able to get PCI/IGD passthrough to Debian
11.1 to work on a supported dom0 configuration such as a
Debian 11.1 dom0 with Xen 4.14 instead of unsupported
Debian 8.x dom0 with unsupported Xen 4.4.
3. Is this problem likely to be only limited to certain hardware
that does not isolate the devices as well as modern Linux-based
software expects?
Thanks in advance for any advice you can give me.
Regards,
Chuck Zmudzinski
[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988333#10
My domain config file (private data such as mac and ip addresses
are redacted):
builder = 'hvm'
memory = '3072'
vcpus = '4'
device_model_version = 'qemu-xen'
# device_model_version = 'qemu-xen-traditional'
# This is a bullseye system (Debian 11.1)
disk = ['/dev/systems/linux,,xvda,w','/dev/data/linuxdata,,xvdb,w']
name = 'bullseye-hvm'
vif = [ 'mac=xx:xx:xx:xx:xx:xx,model=e1000,script=vif-route,ip=A.B.X.Y' ]
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
boot = 'c'
acpi = '1'
apic = '1'
viridian = '1'
xen_platform_pci = '1'
serial = 'pty'
vga = 'none'
sdl = '0'
vnc = '0'
gfx_passthru = '1'
pci = [ '00:1b.0', '00:14.0,rdm_policy=relaxed', '00:02.0' ]