Mailing List Archive

Windows DomU: PCIe passthrough issues
Hi everybody,

after getting my Windows Server 2012 DomU up and running, I have problems with reliably passing through my PCIe TV tuner card (Hauppauge HVR-2205).

I mostly followed https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough#:~:text=PCI%20passthrough%20allows%20you%20to,access%20to%20the%20PCI%20device. I need to set "permissive=1" and limit the Dom0 memory size. I then can pass through the card without any problem. It gets detected and works very well under Windows. However, sometimes the card gets "lost" in the DomU, i. e. it disappears from device manager and isn't functional anymore. This seems to happen mostly after a Windows reboot, but I have the impression that there were also instances without a reboot. I can than reattach it in Dom0 with "xl pci-attach" and it works again in the DomU.

Additional information:
- My TV software (MediaPortal) seems to sometimes recognize a new PCIe card instance after reboot/reattachment, but not consistently. (I then need to manually tell the media software to connect the already tuned channels to the "new" tuner card as well as an internal number of the tuner card is incremented.) Both of this didn't happen with the bare metal system.
- I tested the card in a linux DomU (with VDR software) where the card was attached very reliably.
- The tuner card seems not to have function level reset capabilities (at least lspci shows FLReset- in the DevCap field, although I had the impression that I have seen there a FLReset+ in the past...).

What could be the reason for this behavior? What can I do to troubleshoot and/or solve this issue?

The boot entry has the following parameters:
dom0_mem=1024M,max:1024M xen-pciback.hide=(01:00.0)

Xen config file for the Windows domU:

# kernel = "/usr/lib/xen-4.0/boot/hvmloader"
type='hvm'
memory = 4096
vcpus=2
name = "matrix"
vif = ['bridge=xenbr0,mac=00:16:3E:54:A8:2B']
disk = ['phy:/dev/vg0/matrix,hda,w','phy:/dev/vg0/compudms-data,hdb,w']
device_model_version = 'qemu-xen'
boot="c"
hdtype = 'ahci'
acpi = 1
apic = 1
xen_platform_pci = 1
vendor_device = 'xenserver'
# PCI Passthrough
pci=['01:00.0,permissive=1,power_mgmt=1']
viridian = 1
stdvga = 1
sdl = 0
#pae = 1
serial = 'pty'
usb = 1
#usbctrl = ["type = pv"]
usbdevice = 'tablet'
keymap = 'de'
# VNC
#vnc = 1
#vnclisten = "0.0.0.0"
#vncpasswd = ""
# SPICE
spice=1
spicehost='0.0.0.0'
spiceport=6000
# spicedisable_ticketing enabled is for no spice password, instead use spicepasswd
spicedisable_ticketing=1
#spicepasswd="test"
spicevdagent=1
spice_clipboard_sharing=1
# this will automatically redirect up to 4 usb devices from spice client to domUs
#spiceusbredirection=4
# This adds intel hd audio emulated card used for spice audio
soundhw="hda"

xl info:

host : xxx
release : 4.19.0-14-amd64
version : #1 SMP Debian 4.19.171-2 (2021-01-30)
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 1992.095
hw_caps : bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps : hvm hvm_directio
total_memory : 32542
free_memory : 18773
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 11
xen_extra : .4
xen_version : 4.11.4
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=1024M,max:1024M xen-pciback.hide=(01:00.0)
cc_compiler : gcc (Debian 8.3.0-6) 8.3.0
cc_compile_by : pkg-xen-devel
cc_compile_domain : lists.alioth.debian.org
cc_compile_date : Fri Dec 11 21:33:51 UTC 2020
build_id : 6d8e0fa3ddb825695eb6c6832631b4fa2331fe41
xend_config_format : 4

Logs from /var/log/xen, dmesg and /var/log/messages and Windows DomU didn't seem to show relevant errors, but can of course be provided if needed.

Best regards,

Paul
AW: Windows DomU: PCIe passthrough issues [ In reply to ]
> Von: Paul Leiber
> Gesendet: Montag, 24. Mai 2021 22:39
>
> after getting my Windows Server 2012 DomU up and running, I have
> problems with reliably passing through my PCIe TV tuner card (Hauppauge
> HVR-2205).

Further testing has lead me to the suspicion that there is an incorrect behavior, possibly a bug. It seems that every time I do a _reboot_ from within the Windows DomU, the PCI device does not get attached to the DomU. It is immediately available for attachment in the Dom0 when I check for it, and I can reattach it to the DomU with "xl pci-attach" without any problems. Besides the effect that the tuner card gets assigned a new instance number in my TV software and I therefore need to reapply some settings in this software, the card works normally.

If I _shut down_ the DomU from within the DomU (with Windows shutdown mechanism) or the Dom0 (with "xl shutdown) and restart the DomU with "xl create", the PCIe device gets attached automatically at DomU boot and the above mentioned side effects (new instance number, reapply software settings) do not occur.

I could reproduce this behavior with a different TV tuner card from another manufacturer, so it seems unlikely that it is related to the specific devices I am using.

> The boot entry has the following parameters:
> dom0_mem=1024M,max:1024M xen-pciback.hide=(01:00.0)

I figured out that giving kernel boot option " xen-pciback.hide" is not necessary as the driver is not built into the kernel, therefore I changed the parameters to:
dom0_mem=1024M,max:1024M

The device is assigned to xen-pciback via /etc/modprobe.d/xen-pciback.conf (there is no driver on the Dom0 for the tuner card, therefore no precautions for not loading other drivers are necessary, I think):
options xen-pciback hide=(0000:01:00.0)

While doing trial and error, I changed the pci line in the Xen config file to:

pci=['01:00.0,permissive=1,power_mgmt=1,seize=1']

But adding "seize=1" didn't change anything.

> Logs from /var/log/xen, dmesg and /var/log/messages and Windows DomU
> didn't seem to show relevant errors, but can of course be provided if
> needed.

Checking the xl logs more thoroughly, I now found the following lines:

Waiting for domain matrix (domid 8) to die [pid 3113]
Domain 8 has shut down, reason code 1 0x1
Action for shutdown reason code 1 is restart
libxl: warning: libxl_domain.c:1739:libxl_retrieve_domain_configuration: Domain 8:Device present in JSON but not in xenstore, ignored
Domain 8 needs to be cleaned up: destroying the domain
Done. Rebooting now

Searching for this exact error message, I found the following quite old bug report which sounds suspiciously similar to my experience, only for PV DomUs:
https://bugzilla.redhat.com/show_bug.cgi?id=233801

1. Is it possible that this bug is still existing in my rather recent "out of the box" debian setup?
release : 4.19.0-14-amd64
version : #1 SMP Debian 4.19.171-2 (2021-01-30)
xen_version : 4.11.4

2. Do you have any idea what I should do now? Any more testing I could do? Should I mention this in the Xen developer mailing list? File a bug report somewhere (where?)?

I appreciate any advice.

Best regards,

Paul