Hi,
What options are needed to get MSI-X interrupts working with pci passthrough?
Running Linux 5.10 in dom0 and in the guest with XEN 4.14.1.
Passing trough two Intel X553 NICs to the guest with this in xl.conf:
pci = [ '08:00.0','08:00.1' ]
But what else is needed for MSI-X?
Also what about the swiotlb warning? whats needed to suppress fix that?
Thanks,
Henrik
dmesg in domU:
[ 1.509720] Mounting root: ok.
[ 2.614502] cryptd: max_cpu_qlen set to 1000
[ 2.619136] SSE version of gcm_enc/dec engaged.
[ 2.632522] EDAC pnd2: Failed to register device with error -19.
[ 2.667770] input: PC Speaker as /devices/platform/pcspkr/input/input0
[ 2.702441] pcifront pci-0: Installing PCI frontend
[ 2.703239] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[ 2.767077] software IO TLB: mapped [mem 0x0000000005800000-0x0000000005c00000] (4MB)
[ 2.767384] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[ 2.767453] pcifront pci-0: PCI host bridge to bus 0000:00
[ 2.767457] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 2.767459] pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff]
[ 2.767461] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.767869] pci 0000:00:00.0: [8086:15e4] type 00 class 0x020000
[ 2.768544] pci 0000:00:00.0: reg 0x10: [mem 0xdd600000-0xdd7fffff 64bit pref]
[ 2.769524] pci 0000:00:00.0: reg 0x20: [mem 0xdd804000-0xdd807fff 64bit pref]
[ 2.769857] pci 0000:00:00.0: reg 0x30: [mem 0xdf180000-0xdf1fffff pref]
[ 2.771631] pci 0000:00:00.0: [Firmware Bug]: reg 0x184: invalid BAR (can't size)
[ 2.772607] pci 0000:00:00.0: [Firmware Bug]: reg 0x190: invalid BAR (can't size)
[ 2.780194] pci 0000:00:00.1: [8086:15e4] type 00 class 0x020000
[ 2.780845] pci 0000:00:00.1: reg 0x10: [mem 0xdd400000-0xdd5fffff 64bit pref]
[ 2.781816] pci 0000:00:00.1: reg 0x20: [mem 0xdd800000-0xdd803fff 64bit pref]
[ 2.782139] pci 0000:00:00.1: reg 0x30: [mem 0xdf100000-0xdf17ffff pref]
[ 2.783949] pci 0000:00:00.1: [Firmware Bug]: reg 0x184: invalid BAR (can't size)
[ 2.784917] pci 0000:00:00.1: [Firmware Bug]: reg 0x190: invalid BAR (can't size)
[ 2.795870] pcifront pci-0: claiming resource 0000:00:00.0/0
[ 2.795875] pcifront pci-0: claiming resource 0000:00:00.0/4
[ 2.795876] pcifront pci-0: claiming resource 0000:00:00.0/6
[ 2.795878] pcifront pci-0: claiming resource 0000:00:00.1/0
[ 2.795879] pcifront pci-0: claiming resource 0000:00:00.1/4
[ 2.795883] pcifront pci-0: claiming resource 0000:00:00.1/6
[ 2.799838] dca service started, version 1.12.1
[ 2.841452] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 2.841455] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 2.853935] ixgbe 0000:00:00.0: Xen PCI mapped GSI17 to IRQ19
[ 3.097931] EDAC pnd2: Failed to register device with error -19.
[ 3.193603] Interrupt reservation exceeds available resources
[ 3.193683] ixgbe 0000:00:00.0: Failed to allocate MSI-X interrupts. Err: -28
[ 3.193686] ixgbe 0000:00:00.0: Disabling SR-IOV support
[ 3.193688] ixgbe 0000:00:00.0: Disabling RSS support
[ 3.193901] Interrupt reservation exceeds available resources
[ 3.193912] ixgbe 0000:00:00.0: Failed to allocate MSI interrupt, falling back to legacy. Error: -28
[ 3.193918] ixgbe 0000:00:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 3.321556] ixgbe 0000:00:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
[ 3.321560] ixgbe 0000:00:00.0: 3c:ec:ef:03:0b:d6
[ 3.369729] ixgbe 0000:00:00.0: Intel(R) 10 Gigabit Network Connection
[ 3.373883] ixgbe 0000:00:00.1: Xen PCI mapped GSI18 to IRQ20
[ 3.903506] Interrupt reservation exceeds available resources
[ 3.903566] ixgbe 0000:00:00.1: Failed to allocate MSI-X interrupts. Err: -28
[ 3.903569] ixgbe 0000:00:00.1: Disabling SR-IOV support
[ 3.903570] ixgbe 0000:00:00.1: Disabling RSS support
[ 3.903800] Interrupt reservation exceeds available resources
[ 3.903811] ixgbe 0000:00:00.1: Failed to allocate MSI interrupt, falling back to legacy. Error: -28
[ 3.903825] ixgbe 0000:00:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 4.036582] ixgbe 0000:00:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
[ 4.036587] ixgbe 0000:00:00.1: 3c:ec:ef:03:0b:d7
[ 4.080368] ixgbe 0000:00:00.1: Intel(R) 10 Gigabit Network Connection
What options are needed to get MSI-X interrupts working with pci passthrough?
Running Linux 5.10 in dom0 and in the guest with XEN 4.14.1.
Passing trough two Intel X553 NICs to the guest with this in xl.conf:
pci = [ '08:00.0','08:00.1' ]
But what else is needed for MSI-X?
Also what about the swiotlb warning? whats needed to suppress fix that?
Thanks,
Henrik
dmesg in domU:
[ 1.509720] Mounting root: ok.
[ 2.614502] cryptd: max_cpu_qlen set to 1000
[ 2.619136] SSE version of gcm_enc/dec engaged.
[ 2.632522] EDAC pnd2: Failed to register device with error -19.
[ 2.667770] input: PC Speaker as /devices/platform/pcspkr/input/input0
[ 2.702441] pcifront pci-0: Installing PCI frontend
[ 2.703239] xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
[ 2.767077] software IO TLB: mapped [mem 0x0000000005800000-0x0000000005c00000] (4MB)
[ 2.767384] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[ 2.767453] pcifront pci-0: PCI host bridge to bus 0000:00
[ 2.767457] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 2.767459] pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff]
[ 2.767461] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 2.767869] pci 0000:00:00.0: [8086:15e4] type 00 class 0x020000
[ 2.768544] pci 0000:00:00.0: reg 0x10: [mem 0xdd600000-0xdd7fffff 64bit pref]
[ 2.769524] pci 0000:00:00.0: reg 0x20: [mem 0xdd804000-0xdd807fff 64bit pref]
[ 2.769857] pci 0000:00:00.0: reg 0x30: [mem 0xdf180000-0xdf1fffff pref]
[ 2.771631] pci 0000:00:00.0: [Firmware Bug]: reg 0x184: invalid BAR (can't size)
[ 2.772607] pci 0000:00:00.0: [Firmware Bug]: reg 0x190: invalid BAR (can't size)
[ 2.780194] pci 0000:00:00.1: [8086:15e4] type 00 class 0x020000
[ 2.780845] pci 0000:00:00.1: reg 0x10: [mem 0xdd400000-0xdd5fffff 64bit pref]
[ 2.781816] pci 0000:00:00.1: reg 0x20: [mem 0xdd800000-0xdd803fff 64bit pref]
[ 2.782139] pci 0000:00:00.1: reg 0x30: [mem 0xdf100000-0xdf17ffff pref]
[ 2.783949] pci 0000:00:00.1: [Firmware Bug]: reg 0x184: invalid BAR (can't size)
[ 2.784917] pci 0000:00:00.1: [Firmware Bug]: reg 0x190: invalid BAR (can't size)
[ 2.795870] pcifront pci-0: claiming resource 0000:00:00.0/0
[ 2.795875] pcifront pci-0: claiming resource 0000:00:00.0/4
[ 2.795876] pcifront pci-0: claiming resource 0000:00:00.0/6
[ 2.795878] pcifront pci-0: claiming resource 0000:00:00.1/0
[ 2.795879] pcifront pci-0: claiming resource 0000:00:00.1/4
[ 2.795883] pcifront pci-0: claiming resource 0000:00:00.1/6
[ 2.799838] dca service started, version 1.12.1
[ 2.841452] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 2.841455] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 2.853935] ixgbe 0000:00:00.0: Xen PCI mapped GSI17 to IRQ19
[ 3.097931] EDAC pnd2: Failed to register device with error -19.
[ 3.193603] Interrupt reservation exceeds available resources
[ 3.193683] ixgbe 0000:00:00.0: Failed to allocate MSI-X interrupts. Err: -28
[ 3.193686] ixgbe 0000:00:00.0: Disabling SR-IOV support
[ 3.193688] ixgbe 0000:00:00.0: Disabling RSS support
[ 3.193901] Interrupt reservation exceeds available resources
[ 3.193912] ixgbe 0000:00:00.0: Failed to allocate MSI interrupt, falling back to legacy. Error: -28
[ 3.193918] ixgbe 0000:00:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 3.321556] ixgbe 0000:00:00.0: MAC: 6, PHY: 27, PBA No: 030000-000
[ 3.321560] ixgbe 0000:00:00.0: 3c:ec:ef:03:0b:d6
[ 3.369729] ixgbe 0000:00:00.0: Intel(R) 10 Gigabit Network Connection
[ 3.373883] ixgbe 0000:00:00.1: Xen PCI mapped GSI18 to IRQ20
[ 3.903506] Interrupt reservation exceeds available resources
[ 3.903566] ixgbe 0000:00:00.1: Failed to allocate MSI-X interrupts. Err: -28
[ 3.903569] ixgbe 0000:00:00.1: Disabling SR-IOV support
[ 3.903570] ixgbe 0000:00:00.1: Disabling RSS support
[ 3.903800] Interrupt reservation exceeds available resources
[ 3.903811] ixgbe 0000:00:00.1: Failed to allocate MSI interrupt, falling back to legacy. Error: -28
[ 3.903825] ixgbe 0000:00:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 XDP Queue count = 0
[ 4.036582] ixgbe 0000:00:00.1: MAC: 6, PHY: 27, PBA No: 030000-000
[ 4.036587] ixgbe 0000:00:00.1: 3c:ec:ef:03:0b:d7
[ 4.080368] ixgbe 0000:00:00.1: Intel(R) 10 Gigabit Network Connection