Mailing List Archive: PCI pass-through problem for SN570 NVME SSD

PCI pass-through problem for SN570 NVME SSD

Jul 2, 2022, 10:43 AM

Post #1 of 5 (337 views)

Hi everybody,

I run into problems passing through a SN570 NVME SSD to a HVM guest.
So far I have no idea if the problem is with this specific SSD or with
the CPU + motherboard combination or the SW stack.
Looking for some suggestions on troubleshooting.

List of build info:
CPU+motherboard: E-2146G + Gigabyte C246N-WU2
XEN version: 4.14.3
Dom0: Linux Kernel 5.10 (built from Debian 11.2 kernel source package)
The SN570 SSD sits here in the PCI tree:
+-1d.0-[05]----00.0 Sandisk Corp Device 501a

Syndromes observed:
With ASPM enabled, pciback has problem seizing the device.

Jul 2 00:36:54 gaia kernel: [ 1.648270] pciback 0000:05:00.0:
xen_pciback: seizing device
...
Jul 2 00:36:54 gaia kernel: [ 1.768646] pcieport 0000:00:1d.0:
AER: enabled with IRQ 150
Jul 2 00:36:54 gaia kernel: [ 1.768716] pcieport 0000:00:1d.0:
DPC: enabled with IRQ 150
Jul 2 00:36:54 gaia kernel: [ 1.768717] pcieport 0000:00:1d.0:
DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+
SwTrigger+ RP PIO Log 4, DL_ActiveErr+
...
Jul 2 00:36:54 gaia kernel: [ 1.770039] xen: registering gsi 16
triggering 0 polarity 1
Jul 2 00:36:54 gaia kernel: [ 1.770041] Already setup the GSI :16
Jul 2 00:36:54 gaia kernel: [ 1.770314] pcieport 0000:00:1d.0:
DPC: containment event, status:0x1f11 source:0x0000
Jul 2 00:36:54 gaia kernel: [ 1.770315] pcieport 0000:00:1d.0:
DPC: unmasked uncorrectable error detected
Jul 2 00:36:54 gaia kernel: [ 1.770320] pcieport 0000:00:1d.0:
PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction
Layer, (Receiver ID)
Jul 2 00:36:54 gaia kernel: [ 1.770371] pcieport 0000:00:1d.0:
device [8086:a330] error status/mask=00200000/00010000
Jul 2 00:36:54 gaia kernel: [ 1.770413] pcieport 0000:00:1d.0:
[21] ACSViol (First)
Jul 2 00:36:54 gaia kernel: [ 1.770466] pciback 0000:05:00.0:
xen_pciback: device is not found/assigned
Jul 2 00:36:54 gaia kernel: [ 1.920195] pciback 0000:05:00.0:
xen_pciback: device is not found/assigned
Jul 2 00:36:54 gaia kernel: [ 1.920260] pcieport 0000:00:1d.0:
AER: device recovery successful
Jul 2 00:36:54 gaia kernel: [ 1.920263] pcieport 0000:00:1d.0:
DPC: containment event, status:0x1f01 source:0x0000
Jul 2 00:36:54 gaia kernel: [ 1.920264] pcieport 0000:00:1d.0:
DPC: unmasked uncorrectable error detected
Jul 2 00:36:54 gaia kernel: [ 1.920267] pciback 0000:05:00.0:
xen_pciback: device is not found/assigned
Jul 2 00:36:54 gaia kernel: [ 1.938406] xen: registering gsi 16
triggering 0 polarity 1
Jul 2 00:36:54 gaia kernel: [ 1.938408] Already setup the GSI :16
Jul 2 00:36:54 gaia kernel: [ 1.938666] xen_pciback: backend is vpci
...
Jul 2 00:43:48 gaia kernel: [ 420.231955] pcieport 0000:00:1d.0:
DPC: containment event, status:0x1f01 source:0x0000
Jul 2 00:43:48 gaia kernel: [ 420.231961] pcieport 0000:00:1d.0:
DPC: unmasked uncorrectable error detected
Jul 2 00:43:48 gaia kernel: [ 420.231993] pcieport 0000:00:1d.0:
PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction
Layer, (Requester ID)
Jul 2 00:43:48 gaia kernel: [ 420.235775] pcieport 0000:00:1d.0:
device [8086:a330] error status/mask=00100000/00010000
Jul 2 00:43:48 gaia kernel: [ 420.235779] pcieport 0000:00:1d.0:
[20] UnsupReq (First)
Jul 2 00:43:48 gaia kernel: [ 420.235783] pcieport 0000:00:1d.0:
AER: TLP Header: 34000000 05000010 00000000 88458845
Jul 2 00:43:48 gaia kernel: [ 420.235819] pci 0000:05:00.0: AER:
can't recover (no error_detected callback)
Jul 2 00:43:48 gaia kernel: [ 420.384349] pcieport 0000:00:1d.0:
AER: device recovery successful
... // The following might relate to an attempt to assign the device
to guest, not very sure...
Jul 2 00:46:06 gaia kernel: [ 559.147333] pciback 0000:05:00.0:
xen_pciback: seizing device
Jul 2 00:46:06 gaia kernel: [ 559.147435] pciback 0000:05:00.0:
enabling device (0000 -> 0002)
Jul 2 00:46:06 gaia kernel: [ 559.147508] xen: registering gsi 16
triggering 0 polarity 1
Jul 2 00:46:06 gaia kernel: [ 559.147511] Already setup the GSI :16
Jul 2 00:46:06 gaia kernel: [ 559.147558] pciback 0000:05:00.0:
xen_pciback: MSI-X preparation failed (-6)

With pcie_aspm=off, the error log related to pciback goes away.
But I suspect there are still some problems hidden -- since I don't
see any AER enabled messages so errors may be hidden.
I have the xen_pciback built directly into the kernel and assigned the
SSD to it in the kernel command-line.
However, the result from pci-assignable-xxx commands are not very consistent:

root@gaia:~# xl pci-assignable-list
0000:00:17.0
0000:05:00.0
root@gaia:~# xl pci-assignable-remove 05:00.0
libxl: error: libxl_pci.c:853:libxl__device_pci_assignable_remove:
failed to de-quarantine 0000:05:00.0 <===== Here!!!
root@gaia:~# xl pci-assignable-add 05:00.0
libxl: warning: libxl_pci.c:794:libxl__device_pci_assignable_add:
0000:05:00.0 already assigned to pciback <==== Here!!!
root@gaia:~# xl pci-assignable-remove 05:00.0
root@gaia:~# xl pci-assignable-list
0000:00:17.0
root@gaia:~# xl pci-assignable-add 05:00.0
libxl: warning: libxl_pci.c:814:libxl__device_pci_assignable_add:
0000:05:00.0 not bound to a driver, will not be rebound.
root@gaia:~# xl pci-assignable-list
0000:00:17.0
0000:05:00.0

After the 'xl pci-assignable-list' appears to be self-consistent,
creating VM with the SSD assigned still leads to a guest crash:
From qemu log:
[00:06.0] xen_pt_region_update: Error: create new mem mapping failed! (err: 1)
qemu-system-i386: terminating on signal 1 from pid 1192 (xl)

From the 'xl dmesg' output:
(XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
(XEN) domain_crash called from p2m.c:1301
(XEN) Domain 1 reported crashed by domain 0 on cpu#4:
(XEN) memory_map:fail: dom1 gfn=f3078 mfn=a2504 nr=1 ret:-1

Which of the three syndromes are more fundamental?
1. The DPC / AER error log
2. The inconsistency in 'xl pci-assignable-list' state tracking
3. The GFN mapping failure on guest setup

Any suggestions for the next step?

Thanks,
G.R.

Re: PCI pass-through problem for SN570 NVME SSD [ In reply to ]

firemeteor at users

Jul 3, 2022, 11:37 PM

Post #2 of 5 (335 views)

Permalink

Update some findings with extra triage effort...
Detailed log could be found in the attachments.
1. Confirm stock Debian 11.2 kernel (5.10) shares the same syndrome..
2. With loglvl=all, it reveals why the mapping failure happens, looks
like it comes from some duplicated mapping..
(XEN) memory_map:add: dom1 gfn=f3074 mfn=a2610 nr=2
(XEN) memory_map:add: dom1 gfn=f3077 mfn=a2615 nr=1
(XEN) memory_map:add: dom1 gfn=f3078 mfn=a2616 nr=1 <===========Here
(XEN) ioport_map:add: dom1 gport=c140 mport=4060 nr=20
(XEN) ioport_map:add: dom1 gport=c170 mport=4090 nr=8
(XEN) ioport_map:add: dom1 gport=c178 mport=4080 nr=4
(XEN) memory_map:add: dom1 gfn=f3070 mfn=a2500 nr=2
(XEN) memory_map:add: dom1 gfn=f3073 mfn=a2503 nr=1
(XEN) memory_map:add: dom1 gfn=f3078 mfn=a2504 nr=1 <===========Here
(XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
(XEN) domain_crash called from p2m.c:1301
(XEN) Domain 1 reported crashed by domain 0 on cpu#2:
(XEN) memory_map:fail: dom1 gfn=f3078 mfn=a2504 nr=1 ret:-1
(XEN) memory_map:remove: dom1 gfn=f3078 mfn=a2504 nr=1

3. Recompiled kernel with DEBUG enabled for xen_pciback driver and
play with xl pci-assignable-XXX with it
3.1 It's confirmed that the DPC / AER error log happens only when
xen_pciback attempts to seize && release the device
3.1.1 It only happens on each of the first add / remove operations.
3.2 There is still a 'MSI-X preparation failed' message later-on, but
otherwise it appears to be successful to add / remove the device after
the 1st attempt.
3.3 Not necessarily related, but the DPC / AER log looks similar to
this report [1]

[1]: https://patchwork.kernel.org/project/linux-pci/patch/20220127025418.1989642-1-kai.heng.feng@canonical.com/#24713767
PS: Attempting to fix the line-wrapping issue below... Have no idea
what happened about the formatting....

On Sun, Jul 3, 2022 at 1:43 AM G.R. <firemeteor@users.sourceforge.net> wrote:
>
> Hi everybody,
>
> I run into problems passing through a SN570 NVME SSD to a HVM guest.
> So far I have no idea if the problem is with this specific SSD or with
> the CPU + motherboard combination or the SW stack.
> Looking for some suggestions on troubleshooting.
>
> List of build info:
> CPU+motherboard: E-2146G + Gigabyte C246N-WU2
> XEN version: 4.14.3
> Dom0: Linux Kernel 5.10 (built from Debian 11.2 kernel source package)
> The SN570 SSD sits here in the PCI tree:
> +-1d.0-[05]----00.0 Sandisk Corp Device 501a
>
> Syndromes observed:
> With ASPM enabled, pciback has problem seizing the device.
>
> Jul 2 00:36:54 gaia kernel: [ 1.648270] pciback 0000:05:00.0: xen_pciback: seizing device
> ...
> Jul 2 00:36:54 gaia kernel: [ 1.768646] pcieport 0000:00:1d.0: AER: enabled with IRQ 150
> Jul 2 00:36:54 gaia kernel: [ 1.768716] pcieport 0000:00:1d.0: DPC: enabled with IRQ 150
> Jul 2 00:36:54 gaia kernel: [ 1.768717] pcieport 0000:00:1d.0: DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+ SwTrigger+ RP PIO Log 4, DL_ActiveErr+
> ...
> Jul 2 00:36:54 gaia kernel: [ 1.770039] xen: registering gsi 16 triggering 0 polarity 1
> Jul 2 00:36:54 gaia kernel: [ 1.770041] Already setup the GSI :16
> Jul 2 00:36:54 gaia kernel: [ 1.770314] pcieport 0000:00:1d.0: DPC: containment event, status:0x1f11 source:0x0000
> Jul 2 00:36:54 gaia kernel: [ 1.770315] pcieport 0000:00:1d.0: DPC: unmasked uncorrectable error detected
> Jul 2 00:36:54 gaia kernel: [ 1.770320] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver ID)
> Jul 2 00:36:54 gaia kernel: [ 1.770371] pcieport 0000:00:1d.0: device [8086:a330] error status/mask=00200000/00010000
> Jul 2 00:36:54 gaia kernel: [ 1.770413] pcieport 0000:00:1d.0: [21] ACSViol (First)
> Jul 2 00:36:54 gaia kernel: [ 1.770466] pciback 0000:05:00.0: xen_pciback: device is not found/assigned
> Jul 2 00:36:54 gaia kernel: [ 1.920195] pciback 0000:05:00.0: xen_pciback: device is not found/assigned
> Jul 2 00:36:54 gaia kernel: [ 1.920260] pcieport 0000:00:1d.0: AER: device recovery successful
> Jul 2 00:36:54 gaia kernel: [ 1.920263] pcieport 0000:00:1d.0: DPC: containment event, status:0x1f01 source:0x0000
> Jul 2 00:36:54 gaia kernel: [ 1.920264] pcieport 0000:00:1d.0: DPC: unmasked uncorrectable error detected
> Jul 2 00:36:54 gaia kernel: [ 1.920267] pciback 0000:05:00.0: xen_pciback: device is not found/assigned
> Jul 2 00:36:54 gaia kernel: [ 1.938406] xen: registering gsi 16 triggering 0 polarity 1
> Jul 2 00:36:54 gaia kernel: [ 1.938408] Already setup the GSI :16
> Jul 2 00:36:54 gaia kernel: [ 1.938666] xen_pciback: backend is vpci
> ...
> Jul 2 00:43:48 gaia kernel: [ 420.231955] pcieport 0000:00:1d.0: DPC: containment event, status:0x1f01 source:0x0000
> Jul 2 00:43:48 gaia kernel: [ 420.231961] pcieport 0000:00:1d.0: DPC: unmasked uncorrectable error detected
> Jul 2 00:43:48 gaia kernel: [ 420.231993] pcieport 0000:00:1d.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
> Jul 2 00:43:48 gaia kernel: [ 420.235775] pcieport 0000:00:1d.0: device [8086:a330] error status/mask=00100000/00010000
> Jul 2 00:43:48 gaia kernel: [ 420.235779] pcieport 0000:00:1d.0: [20] UnsupReq (First)
> Jul 2 00:43:48 gaia kernel: [ 420.235783] pcieport 0000:00:1d.0: AER: TLP Header: 34000000 05000010 00000000 88458845
> Jul 2 00:43:48 gaia kernel: [ 420.235819] pci 0000:05:00.0: AER: can't recover (no error_detected callback)
> Jul 2 00:43:48 gaia kernel: [ 420.384349] pcieport 0000:00:1d.0: AER: device recovery successful
> ... // The following might relate to an attempt to assign the device to guest, not very sure...
> Jul 2 00:46:06 gaia kernel: [ 559.147333] pciback 0000:05:00.0: xen_pciback: seizing device
> Jul 2 00:46:06 gaia kernel: [ 559.147435] pciback 0000:05:00.0: enabling device (0000 -> 0002)
> Jul 2 00:46:06 gaia kernel: [ 559.147508] xen: registering gsi 16 triggering 0 polarity 1
> Jul 2 00:46:06 gaia kernel: [ 559.147511] Already setup the GSI :16
> Jul 2 00:46:06 gaia kernel: [ 559.147558] pciback 0000:05:00.0: xen_pciback: MSI-X preparation failed (-6)
>
>
> With pcie_aspm=off, the error log related to pciback goes away.
> But I suspect there are still some problems hidden -- since I don't
> see any AER enabled messages so errors may be hidden.
> I have the xen_pciback built directly into the kernel and assigned the
> SSD to it in the kernel command-line.
> However, the result from pci-assignable-xxx commands are not very consistent:
>
> root@gaia:~# xl pci-assignable-list
> 0000:00:17.0
> 0000:05:00.0
> root@gaia:~# xl pci-assignable-remove 05:00.0
> libxl: error: libxl_pci.c:853:libxl__device_pci_assignable_remove: failed to de-quarantine 0000:05:00.0 <===== Here!!!
> root@gaia:~# xl pci-assignable-add 05:00.0
> libxl: warning: libxl_pci.c:794:libxl__device_pci_assignable_add: 0000:05:00.0 already assigned to pciback <==== Here!!!
> root@gaia:~# xl pci-assignable-remove 05:00.0
> root@gaia:~# xl pci-assignable-list 0000:00:17.0
> root@gaia:~# xl pci-assignable-add 05:00.0
> libxl: warning: libxl_pci.c:814:libxl__device_pci_assignable_add: 0000:05:00.0 not bound to a driver, will not be rebound.
> root@gaia:~# xl pci-assignable-list
> 0000:00:17.0
> 0000:05:00.0
>
>
> After the 'xl pci-assignable-list' appears to be self-consistent, creating VM with the SSD assigned still leads to a guest crash:
> From qemu log:
> [00:06.0] xen_pt_region_update: Error: create new mem mapping failed! (err: 1)
> qemu-system-i386: terminating on signal 1 from pid 1192 (xl)
>
> From the 'xl dmesg' output:
> (XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
> (XEN) domain_crash called from p2m.c:1301
> (XEN) Domain 1 reported crashed by domain 0 on cpu#4:
> (XEN) memory_map:fail: dom1 gfn=f3078 mfn=a2504 nr=1 ret:-1
>
>
> Which of the three syndromes are more fundamental?
> 1. The DPC / AER error log
> 2. The inconsistency in 'xl pci-assignable-list' state tracking
> 3. The GFN mapping failure on guest setup
>
> Any suggestions for the next step?
>
>
> Thanks,
> G.R.

Re: PCI pass-through problem for SN570 NVME SSD [ In reply to ]

roger.pau at citrix

Jul 4, 2022, 2:50 AM

Post #3 of 5 (335 views)

Permalink

On Sun, Jul 03, 2022 at 01:43:11AM +0800, G.R. wrote:
> Hi everybody,
>
> I run into problems passing through a SN570 NVME SSD to a HVM guest.
> So far I have no idea if the problem is with this specific SSD or with
> the CPU + motherboard combination or the SW stack.
> Looking for some suggestions on troubleshooting.
>
> List of build info:
> CPU+motherboard: E-2146G + Gigabyte C246N-WU2
> XEN version: 4.14.3

Are you using a debug build of Xen? (if not it would be helpful to do
so).

> Dom0: Linux Kernel 5.10 (built from Debian 11.2 kernel source package)
> The SN570 SSD sits here in the PCI tree:
> +-1d.0-[05]----00.0 Sandisk Corp Device 501a

Could be helpful to post the output with -vvv so we can see the
capabilities of the device.

> Syndromes observed:
> With ASPM enabled, pciback has problem seizing the device.
>
> Jul 2 00:36:54 gaia kernel: [ 1.648270] pciback 0000:05:00.0:
> xen_pciback: seizing device
> ...
> Jul 2 00:36:54 gaia kernel: [ 1.768646] pcieport 0000:00:1d.0:
> AER: enabled with IRQ 150
> Jul 2 00:36:54 gaia kernel: [ 1.768716] pcieport 0000:00:1d.0:
> DPC: enabled with IRQ 150
> Jul 2 00:36:54 gaia kernel: [ 1.768717] pcieport 0000:00:1d.0:
> DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+
> SwTrigger+ RP PIO Log 4, DL_ActiveErr+

Is there a device reset involved here? It's possible the device
doesn't reset properly and hence the Uncorrectable Error Status
Register ends up with inconsistent bits set.

> ...
> Jul 2 00:36:54 gaia kernel: [ 1.770039] xen: registering gsi 16
> triggering 0 polarity 1
> Jul 2 00:36:54 gaia kernel: [ 1.770041] Already setup the GSI :16
> Jul 2 00:36:54 gaia kernel: [ 1.770314] pcieport 0000:00:1d.0:
> DPC: containment event, status:0x1f11 source:0x0000
> Jul 2 00:36:54 gaia kernel: [ 1.770315] pcieport 0000:00:1d.0:
> DPC: unmasked uncorrectable error detected
> Jul 2 00:36:54 gaia kernel: [ 1.770320] pcieport 0000:00:1d.0:
> PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction
> Layer, (Receiver ID)
> Jul 2 00:36:54 gaia kernel: [ 1.770371] pcieport 0000:00:1d.0:
> device [8086:a330] error status/mask=00200000/00010000
> Jul 2 00:36:54 gaia kernel: [ 1.770413] pcieport 0000:00:1d.0:
> [21] ACSViol (First)
> Jul 2 00:36:54 gaia kernel: [ 1.770466] pciback 0000:05:00.0:
> xen_pciback: device is not found/assigned
> Jul 2 00:36:54 gaia kernel: [ 1.920195] pciback 0000:05:00.0:
> xen_pciback: device is not found/assigned
> Jul 2 00:36:54 gaia kernel: [ 1.920260] pcieport 0000:00:1d.0:
> AER: device recovery successful
> Jul 2 00:36:54 gaia kernel: [ 1.920263] pcieport 0000:00:1d.0:
> DPC: containment event, status:0x1f01 source:0x0000
> Jul 2 00:36:54 gaia kernel: [ 1.920264] pcieport 0000:00:1d.0:
> DPC: unmasked uncorrectable error detected
> Jul 2 00:36:54 gaia kernel: [ 1.920267] pciback 0000:05:00.0:
> xen_pciback: device is not found/assigned

That's from a different device (05:00.0).

> Jul 2 00:36:54 gaia kernel: [ 1.938406] xen: registering gsi 16
> triggering 0 polarity 1
> Jul 2 00:36:54 gaia kernel: [ 1.938408] Already setup the GSI :16
> Jul 2 00:36:54 gaia kernel: [ 1.938666] xen_pciback: backend is vpci
> ...
> Jul 2 00:43:48 gaia kernel: [ 420.231955] pcieport 0000:00:1d.0:
> DPC: containment event, status:0x1f01 source:0x0000
> Jul 2 00:43:48 gaia kernel: [ 420.231961] pcieport 0000:00:1d.0:
> DPC: unmasked uncorrectable error detected
> Jul 2 00:43:48 gaia kernel: [ 420.231993] pcieport 0000:00:1d.0:
> PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction
> Layer, (Requester ID)
> Jul 2 00:43:48 gaia kernel: [ 420.235775] pcieport 0000:00:1d.0:
> device [8086:a330] error status/mask=00100000/00010000
> Jul 2 00:43:48 gaia kernel: [ 420.235779] pcieport 0000:00:1d.0:
> [20] UnsupReq (First)
> Jul 2 00:43:48 gaia kernel: [ 420.235783] pcieport 0000:00:1d.0:
> AER: TLP Header: 34000000 05000010 00000000 88458845
> Jul 2 00:43:48 gaia kernel: [ 420.235819] pci 0000:05:00.0: AER:
> can't recover (no error_detected callback)
> Jul 2 00:43:48 gaia kernel: [ 420.384349] pcieport 0000:00:1d.0:
> AER: device recovery successful
> ... // The following might relate to an attempt to assign the device
> to guest, not very sure...
> Jul 2 00:46:06 gaia kernel: [ 559.147333] pciback 0000:05:00.0:
> xen_pciback: seizing device
> Jul 2 00:46:06 gaia kernel: [ 559.147435] pciback 0000:05:00.0:
> enabling device (0000 -> 0002)
> Jul 2 00:46:06 gaia kernel: [ 559.147508] xen: registering gsi 16
> triggering 0 polarity 1
> Jul 2 00:46:06 gaia kernel: [ 559.147511] Already setup the GSI :16
> Jul 2 00:46:06 gaia kernel: [ 559.147558] pciback 0000:05:00.0:
> xen_pciback: MSI-X preparation failed (-6)
>
>
> With pcie_aspm=off, the error log related to pciback goes away.
> But I suspect there are still some problems hidden -- since I don't
> see any AER enabled messages so errors may be hidden.
> I have the xen_pciback built directly into the kernel and assigned the
> SSD to it in the kernel command-line.
> However, the result from pci-assignable-xxx commands are not very consistent:
>
> root@gaia:~# xl pci-assignable-list
> 0000:00:17.0
> 0000:05:00.0
> root@gaia:~# xl pci-assignable-remove 05:00.0
> libxl: error: libxl_pci.c:853:libxl__device_pci_assignable_remove:
> failed to de-quarantine 0000:05:00.0 <===== Here!!!
> root@gaia:~# xl pci-assignable-add 05:00.0
> libxl: warning: libxl_pci.c:794:libxl__device_pci_assignable_add:
> 0000:05:00.0 already assigned to pciback <==== Here!!!
> root@gaia:~# xl pci-assignable-remove 05:00.0
> root@gaia:~# xl pci-assignable-list
> 0000:00:17.0
> root@gaia:~# xl pci-assignable-add 05:00.0
> libxl: warning: libxl_pci.c:814:libxl__device_pci_assignable_add:
> 0000:05:00.0 not bound to a driver, will not be rebound.
> root@gaia:~# xl pci-assignable-list
> 0000:00:17.0
> 0000:05:00.0

I'm confused, the log above is mostly from a device at 0000:00:1d.0,
while here you only have 0000:00:17.0 and 0000:05:00.0. I assume
0000:00:1d.0 never gets to appear on the output of `xl
pci-assignable-list`?

Also you seem to be trying to assign 0000:05:00.0 which is not the
same device that's giving the errors above. From the text above I've
assumed 0000:00:1d.0 was the NVME that you wanted to assign to a
guest.

Could you attempt the same with only the single device that's causing
issues as assignable? (having other devices just makes the output
confusing).

>
> After the 'xl pci-assignable-list' appears to be self-consistent,
> creating VM with the SSD assigned still leads to a guest crash:
> From qemu log:
> [00:06.0] xen_pt_region_update: Error: create new mem mapping failed! (err: 1)
> qemu-system-i386: terminating on signal 1 from pid 1192 (xl)
>
> From the 'xl dmesg' output:
> (XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted

Seems like QEMU is attempting to remap a p2m_mmio_direct region.

Can you paste the full output of `xl dmesg`? (as that will contain the
memory map).

Would also be helpful if you could get the RMRR regions from that
box. Booting with `iommu=verbose` on the Xen command line should print
those.

> (XEN) domain_crash called from p2m.c:1301
> (XEN) Domain 1 reported crashed by domain 0 on cpu#4:
> (XEN) memory_map:fail: dom1 gfn=f3078 mfn=a2504 nr=1 ret:-1
>
>
> Which of the three syndromes are more fundamental?
> 1. The DPC / AER error log
> 2. The inconsistency in 'xl pci-assignable-list' state tracking
> 3. The GFN mapping failure on guest setup
>
> Any suggestions for the next step?

I'm slightly confused by the fact that the DPC / AER errors seem to be
from a device that's different from what you attempt to passthrough?
(0000:00:1d.0 vs 0000:05:00.0)

Might be helpful to start by only attempting to passthrough the device
you are having issues with, and leaving any other device out.

Roger.

Re: PCI pass-through problem for SN570 NVME SSD [ In reply to ]

firemeteor at users

Jul 4, 2022, 4:34 AM

Post #4 of 5 (335 views)

Permalink

On Mon, Jul 4, 2022 at 5:53 PM Roger Pau Monné <roger.pau@citrix.com> wrote:
>
> On Sun, Jul 03, 2022 at 01:43:11AM +0800, G.R. wrote:
> > Hi everybody,
> >
> > I run into problems passing through a SN570 NVME SSD to a HVM guest.
> > So far I have no idea if the problem is with this specific SSD or with
> > the CPU + motherboard combination or the SW stack.
> > Looking for some suggestions on troubleshooting.
> >
> > List of build info:
> > CPU+motherboard: E-2146G + Gigabyte C246N-WU2
> > XEN version: 4.14.3
>
> Are you using a debug build of Xen? (if not it would be helpful to do
> so).
It's a release version at this moment. I can switch to a debug version
later when I get my hands free.
BTW, I got a DEBUG build of the xen_pciback driver to see how it plays
with 'xl pci-assignable-xxx' commands.
You can find this in my 2nd email in the chain.

>
> > Dom0: Linux Kernel 5.10 (built from Debian 11.2 kernel source package)
> > The SN570 SSD sits here in the PCI tree:
> > +-1d.0-[05]----00.0 Sandisk Corp Device 501a
>
> Could be helpful to post the output with -vvv so we can see the
> capabilities of the device.
Sure, please find the -vvv output from the attachment.
This one is just to indicate the connection in the PCI tree.
I.e. 05:00.0 is attached under 00:1d.0.

>
> > Syndromes observed:
> > With ASPM enabled, pciback has problem seizing the device.
> >
> > Jul 2 00:36:54 gaia kernel: [ 1.648270] pciback 0000:05:00.0:
> > xen_pciback: seizing device
> > ...
> > Jul 2 00:36:54 gaia kernel: [ 1.768646] pcieport 0000:00:1d.0:
> > AER: enabled with IRQ 150
> > Jul 2 00:36:54 gaia kernel: [ 1.768716] pcieport 0000:00:1d.0:
> > DPC: enabled with IRQ 150
> > Jul 2 00:36:54 gaia kernel: [ 1.768717] pcieport 0000:00:1d.0:
> > DPC: error containment capabilities: Int Msg #0, RPExt+ PoisonedTLP+
> > SwTrigger+ RP PIO Log 4, DL_ActiveErr+
>
> Is there a device reset involved here? It's possible the device
> doesn't reset properly and hence the Uncorrectable Error Status
> Register ends up with inconsistent bits set.

xen_pciback appears to force a FLR whenever it attempts to seize a
capable device.
As shown in pciback_dbg_xl-pci_assignable_XXX.log attached in my 2nd mail.
[ 323.448115] xen_pciback: wants to seize 0000:05:00.0
[ 323.448136] pciback 0000:05:00.0: xen_pciback: probing...
[ 323.448137] pciback 0000:05:00.0: xen_pciback: seizing device
[ 323.448162] pciback 0000:05:00.0: xen_pciback: pcistub_device_alloc
[ 323.448162] pciback 0000:05:00.0: xen_pciback: initializing...
[ 323.448163] pciback 0000:05:00.0: xen_pciback: initializing config
[ 323.448344] pciback 0000:05:00.0: xen_pciback: enabling device
[ 323.448425] xen: registering gsi 16 triggering 0 polarity 1
[ 323.448428] Already setup the GSI :16
[ 323.448497] pciback 0000:05:00.0: xen_pciback: save state of device
[ 323.448642] pciback 0000:05:00.0: xen_pciback: resetting (FLR, D3,
etc) the device
[ 323.448707] pcieport 0000:00:1d.0: DPC: containment event,
status:0x1f11 source:0x0000
[ 323.448730] pcieport 0000:00:1d.0: DPC: unmasked uncorrectable error detected
[ 323.448760] pcieport 0000:00:1d.0: PCIe Bus Error:
severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Receiver
ID)
[ 323.448786] pcieport 0000:00:1d.0: device [8086:a330] error
status/mask=00200000/00010000
[ 323.448813] pcieport 0000:00:1d.0: [21] ACSViol (First)
[ 324.690979] pciback 0000:05:00.0: not ready 1023ms after FLR;
waiting <============ HERE
[ 325.730706] pciback 0000:05:00.0: not ready 2047ms after FLR; waiting
[ 327.997638] pciback 0000:05:00.0: not ready 4095ms after FLR; waiting
[ 332.264251] pciback 0000:05:00.0: not ready 8191ms after FLR; waiting
[ 340.584320] pciback 0000:05:00.0: not ready 16383ms after FLR;
waiting
[ 357.010896] pciback 0000:05:00.0: not ready 32767ms after FLR; waiting
[ 391.143951] pciback 0000:05:00.0: not ready 65535ms after FLR; giving up
[ 392.249252] pciback 0000:05:00.0: xen_pciback: reset device
[ 392.249392] pciback 0000:05:00.0: xen_pciback:
xen_pcibk_error_detected(bus:5,devfn:0)
[ 392.249393] pciback 0000:05:00.0: xen_pciback: device is not found/assigned
[ 392.397074] pciback 0000:05:00.0: xen_pciback:
xen_pcibk_error_resume(bus:5,devfn:0)
[ 392.397080] pciback 0000:05:00.0: xen_pciback: device is not found/assigned
[ 392.397284] pcieport 0000:00:1d.0: AER: device recovery successful
Note, I only see this in FLR action the 1st attempt.
And my SATA controller which doesn't support FLR appears to pass
through just fine...

>
> > ...
> > Jul 2 00:36:54 gaia kernel: [ 1.770039] xen: registering gsi 16
> > triggering 0 polarity 1
> > Jul 2 00:36:54 gaia kernel: [ 1.770041] Already setup the GSI :16
> > Jul 2 00:36:54 gaia kernel: [ 1.770314] pcieport 0000:00:1d.0:
> > DPC: containment event, status:0x1f11 source:0x0000
> > Jul 2 00:36:54 gaia kernel: [ 1.770315] pcieport 0000:00:1d.0:
> > DPC: unmasked uncorrectable error detected
> > Jul 2 00:36:54 gaia kernel: [ 1.770320] pcieport 0000:00:1d.0:
> > PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction
> > Layer, (Receiver ID)
> > Jul 2 00:36:54 gaia kernel: [ 1.770371] pcieport 0000:00:1d.0:
> > device [8086:a330] error status/mask=00200000/00010000
> > Jul 2 00:36:54 gaia kernel: [ 1.770413] pcieport 0000:00:1d.0:
> > [21] ACSViol (First)
> > Jul 2 00:36:54 gaia kernel: [ 1.770466] pciback 0000:05:00.0:
> > xen_pciback: device is not found/assigned
> > Jul 2 00:36:54 gaia kernel: [ 1.920195] pciback 0000:05:00.0:
> > xen_pciback: device is not found/assigned
> > Jul 2 00:36:54 gaia kernel: [ 1.920260] pcieport 0000:00:1d.0:
> > AER: device recovery successful
> > Jul 2 00:36:54 gaia kernel: [ 1.920263] pcieport 0000:00:1d.0:
> > DPC: containment event, status:0x1f01 source:0x0000
> > Jul 2 00:36:54 gaia kernel: [ 1.920264] pcieport 0000:00:1d.0:
> > DPC: unmasked uncorrectable error detected
> > Jul 2 00:36:54 gaia kernel: [ 1.920267] pciback 0000:05:00.0:
> > xen_pciback: device is not found/assigned
>
> That's from a different device (05:00.0).
00:1d.0 is the bridge port that 05:00.0 attaches to.

> >
> > After the 'xl pci-assignable-list' appears to be self-consistent,
> > creating VM with the SSD assigned still leads to a guest crash:
> > From qemu log:
> > [00:06.0] xen_pt_region_update: Error: create new mem mapping failed! (err: 1)
> > qemu-system-i386: terminating on signal 1 from pid 1192 (xl)
> >
> > From the 'xl dmesg' output:
> > (XEN) d1: GFN 0xf3078 (0xa2616,0,5,7) -> (0xa2504,0,5,7) not permitted
>
> Seems like QEMU is attempting to remap a p2m_mmio_direct region.
>
> Can you paste the full output of `xl dmesg`? (as that will contain the
> memory map).
Attached.

>
> Would also be helpful if you could get the RMRR regions from that
> box. Booting with `iommu=verbose` on the Xen command line should print
> those.
Coming in my next reply...

Re: PCI pass-through problem for SN570 NVME SSD [ In reply to ]

firemeteor at users

Jul 4, 2022, 4:44 AM

Post #5 of 5 (335 views)

Permalink

On Mon, Jul 4, 2022 at 7:34 PM G.R. <firemeteor@users.sourceforge.net> wrote:
>
> On Mon, Jul 4, 2022 at 5:53 PM Roger Pau Monné <roger.pau@citrix.com> wrote:

> >
> > Would also be helpful if you could get the RMRR regions from that
> > box. Booting with `iommu=verbose` on the Xen command line should print
> > those.
> Coming in my next reply...
See attached.

Mailing List Archive

Attached Files:

Attached Files:

Attached Files: