Adding Bertrand, Oleksandr, Julien, and others -- they have a more
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.
Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.
Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?
On Tue, 17 Nov 2020, Leo Krueger wrote:
> Hi,
>
> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...) but then had to add the following node to my device tree
>
> gic_lpi_base: syscon@0x80000000 {
> compatible = "gic-lpi-base";
> reg = <0x0 0x80000000 0x0 0x100000>;
> max-gic-redistributors = <2>;
> };
>
> to somehow change something in regard to the ITS and MSI/MSI-X
>
> (XEN) GICv3 initialization:
> (XEN) gic_dist_addr=0x00000006000000
> (XEN) gic_maintenance_irq=25
> (XEN) gic_rdist_stride=0
> (XEN) gic_rdist_regions=1
> (XEN) redistributor regions:
> (XEN) - region 0: 0x00000006040000 - 0x00000006080000
> (XEN) GICv3: using at most 57344 LPIs on the host.
> (XEN) GICv3: 288 lines, (IID 0001143b).
> (XEN) GICv3: Found ITS @0x6020000
> (XEN) using non-cacheable ITS command queue
> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>
> [ 0.000000] GICv3: Distributor has no Range Selector support
> [ 0.000000] GICv3: no VLPI support, no direct LPI support
> [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
> [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1)
> [ 0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
> [ 0.000000] GIC: using LPI property table @0x00000000dc830000
> [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
> [ 0.000000] CPU0: using LPI pending table @0x00000000dc840000
> ...
> [ 0.040080] Platform MSI: gic-its domain created
> [ 0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> [ 0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>
>
> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might fail!" log messages for some PCI devices, but at least the on-board ethernet ports (fsl_enetc ) are initialized.
> I can set the link up and a link is successfully established.
>
> But (!) I cannot receive or transmit anything (no error message...) and there seem to be no interrupts:
>
> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
>
> (from /proc/interrupts).
>
> Any idea on this one? I keep digging and checking whether my device tree needs some additional fixes.
>
> Kind regards,
> Leo
>
> --
> Leo Krüger, M.Sc.
> Senior Systems Engineer Distributed Systems
> Intelligent Digital Cabin
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Hein-Saß-Weg 22
> 21129 Hamburg
>
> +49 (0) 40 248 595-154
>
> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Sitz der Gesellschaft / Legal Domicile: Hamburg
> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR Andreas Rieckhof
> Geschäftsführung / Board of Management: Roland Gerhards
>
> Disclaimer:
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have
> received this mail in error), please notify the sender immediately and destroy this e-mail. Any unauthorised copying,
> disclosure or distribution of the material in this e-mail is strictly forbidden.
>
> > -----Ursprüngliche Nachricht-----
> > Von: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Gesendet: Dienstag, 17. November 2020 01:59
> > An: Leo Krueger <leo.krueger@zal.aero>
> > Cc: Peng Fan <peng.fan@nxp.com>; Stefano Stabellini
> > <stefano.stabellini@xilinx.com>; brucea@xilinx.com; Cornelia Bruelhart
> > <cornelia.bruelhart@zal.aero>
> > Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer
> >
> > Replies inline below
> >
> >
> > On Sun, 15 Nov 2020, Leo Krueger wrote:
> > > Hi Peng, hi Stefano,
> > >
> > >
> > >
> > > sorry for the long silence…
> > >
> > >
> > >
> > > I tried the change suggested (and hope I didn’t do anything wrong
> > > while doing so…) on top of XEN 4.13.2 (before, I always tried with
> > > 4.12 but wanted to give 4.13.2 a try as well) but I do not see any difference,
> > still the same “unhandled context fault” log entries pop up and I cannot
> > access my sdcard.
> > >
> > >
> > >
> > > As it seems to work without respectively disabled iommu, that would be
> > > fine for me for now. What I am worried about a bit more is PCIe or
> > MSI/MSIX to be exact.
> > >
> > >
> > >
> > > Here is the gic-v3 and its node from my device tree:
> > >
> > >
> > >
> > > interrupt-controller@6000000 {
> > >
> > > compatible = "arm,gic-v3";
> > >
> > > #address-cells = <0x2>;
> > >
> > > #size-cells = <0x2>;
> > >
> > > ranges;
> > >
> > > reg = <0x0 0x6000000 0x0 0x10000 0x0 0x6040000 0x0 0x40000>;
> > >
> > > #interrupt-cells = <0x3>;
> > >
> > > interrupt-controller;
> > >
> > > interrupts = <0x1 0x9 0xf08>;
> > >
> > > phandle = <0x1>;
> > >
> > >
> > >
> > > gic-its@6020000 {
> > >
> > > compatible = "arm,gic-v3-its";
> > >
> > > msi-controller;
> > >
> > > reg = <0x0 0x6020000 0x0 0x20000>;
> > >
> > > phandle = <0xd>;
> > >
> > > };
> > >
> > > };
> > >
> > >
> > >
> > > And here are some kernel log excerpts related to GIC when booting
> > without (!) XEN:
> > >
> > >
> > >
> > > [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
> > >
> > > [ 0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [ 0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
> > >
> > > [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> > > @20fb880000 (flat, esz 8, psz 64K, shr 0)
> > >
> > > [ 0.000000] ITS: using cache flushing for cmd queue
> > >
> > > [ 0.000000] GIC: using LPI property table @0x00000020fb830000
> > >
> > > [ 0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> > >
> > > [ 0.000000] CPU0: using LPI pending table @0x00000020fb840000
> > >
> > > [ 0.000000] GIC: using cache flushing for LPI property table
> > >
> > >
> > >
> > > However, when booting with XEN, only the following three lines are
> > contained in the kernel log:
> > >
> > >
> > >
> > > [ 0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [ 0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [ 0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> >
> > "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS
> > support.
> >
> > Can you double check that you have the ITS driver in Xen built-in? That would
> > be CONFIG_HAS_ITS. If you do "make menuconfig" and enable "Configure
> > standard Xen features (expert users)" you should get a new option "GICv3
> > ITS MSI controller support" under "Architecture Features".
> > Make sure to enable it.
> >
> > Let me know if that works!
>
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.
Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.
Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?
On Tue, 17 Nov 2020, Leo Krueger wrote:
> Hi,
>
> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...) but then had to add the following node to my device tree
>
> gic_lpi_base: syscon@0x80000000 {
> compatible = "gic-lpi-base";
> reg = <0x0 0x80000000 0x0 0x100000>;
> max-gic-redistributors = <2>;
> };
>
> to somehow change something in regard to the ITS and MSI/MSI-X
>
> (XEN) GICv3 initialization:
> (XEN) gic_dist_addr=0x00000006000000
> (XEN) gic_maintenance_irq=25
> (XEN) gic_rdist_stride=0
> (XEN) gic_rdist_regions=1
> (XEN) redistributor regions:
> (XEN) - region 0: 0x00000006040000 - 0x00000006080000
> (XEN) GICv3: using at most 57344 LPIs on the host.
> (XEN) GICv3: 288 lines, (IID 0001143b).
> (XEN) GICv3: Found ITS @0x6020000
> (XEN) using non-cacheable ITS command queue
> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>
> [ 0.000000] GICv3: Distributor has no Range Selector support
> [ 0.000000] GICv3: no VLPI support, no direct LPI support
> [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
> [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1)
> [ 0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
> [ 0.000000] GIC: using LPI property table @0x00000000dc830000
> [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
> [ 0.000000] CPU0: using LPI pending table @0x00000000dc840000
> ...
> [ 0.040080] Platform MSI: gic-its domain created
> [ 0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> [ 0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>
>
> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might fail!" log messages for some PCI devices, but at least the on-board ethernet ports (fsl_enetc ) are initialized.
> I can set the link up and a link is successfully established.
>
> But (!) I cannot receive or transmit anything (no error message...) and there seem to be no interrupts:
>
> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
>
> (from /proc/interrupts).
>
> Any idea on this one? I keep digging and checking whether my device tree needs some additional fixes.
>
> Kind regards,
> Leo
>
> --
> Leo Krüger, M.Sc.
> Senior Systems Engineer Distributed Systems
> Intelligent Digital Cabin
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Hein-Saß-Weg 22
> 21129 Hamburg
>
> +49 (0) 40 248 595-154
>
> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Sitz der Gesellschaft / Legal Domicile: Hamburg
> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR Andreas Rieckhof
> Geschäftsführung / Board of Management: Roland Gerhards
>
> Disclaimer:
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have
> received this mail in error), please notify the sender immediately and destroy this e-mail. Any unauthorised copying,
> disclosure or distribution of the material in this e-mail is strictly forbidden.
>
> > -----Ursprüngliche Nachricht-----
> > Von: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Gesendet: Dienstag, 17. November 2020 01:59
> > An: Leo Krueger <leo.krueger@zal.aero>
> > Cc: Peng Fan <peng.fan@nxp.com>; Stefano Stabellini
> > <stefano.stabellini@xilinx.com>; brucea@xilinx.com; Cornelia Bruelhart
> > <cornelia.bruelhart@zal.aero>
> > Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer
> >
> > Replies inline below
> >
> >
> > On Sun, 15 Nov 2020, Leo Krueger wrote:
> > > Hi Peng, hi Stefano,
> > >
> > >
> > >
> > > sorry for the long silence…
> > >
> > >
> > >
> > > I tried the change suggested (and hope I didn’t do anything wrong
> > > while doing so…) on top of XEN 4.13.2 (before, I always tried with
> > > 4.12 but wanted to give 4.13.2 a try as well) but I do not see any difference,
> > still the same “unhandled context fault” log entries pop up and I cannot
> > access my sdcard.
> > >
> > >
> > >
> > > As it seems to work without respectively disabled iommu, that would be
> > > fine for me for now. What I am worried about a bit more is PCIe or
> > MSI/MSIX to be exact.
> > >
> > >
> > >
> > > Here is the gic-v3 and its node from my device tree:
> > >
> > >
> > >
> > > interrupt-controller@6000000 {
> > >
> > > compatible = "arm,gic-v3";
> > >
> > > #address-cells = <0x2>;
> > >
> > > #size-cells = <0x2>;
> > >
> > > ranges;
> > >
> > > reg = <0x0 0x6000000 0x0 0x10000 0x0 0x6040000 0x0 0x40000>;
> > >
> > > #interrupt-cells = <0x3>;
> > >
> > > interrupt-controller;
> > >
> > > interrupts = <0x1 0x9 0xf08>;
> > >
> > > phandle = <0x1>;
> > >
> > >
> > >
> > > gic-its@6020000 {
> > >
> > > compatible = "arm,gic-v3-its";
> > >
> > > msi-controller;
> > >
> > > reg = <0x0 0x6020000 0x0 0x20000>;
> > >
> > > phandle = <0xd>;
> > >
> > > };
> > >
> > > };
> > >
> > >
> > >
> > > And here are some kernel log excerpts related to GIC when booting
> > without (!) XEN:
> > >
> > >
> > >
> > > [ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
> > >
> > > [ 0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [ 0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
> > >
> > > [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> > > @20fb880000 (flat, esz 8, psz 64K, shr 0)
> > >
> > > [ 0.000000] ITS: using cache flushing for cmd queue
> > >
> > > [ 0.000000] GIC: using LPI property table @0x00000020fb830000
> > >
> > > [ 0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> > >
> > > [ 0.000000] CPU0: using LPI pending table @0x00000020fb840000
> > >
> > > [ 0.000000] GIC: using cache flushing for LPI property table
> > >
> > >
> > >
> > > However, when booting with XEN, only the following three lines are
> > contained in the kernel log:
> > >
> > >
> > >
> > > [ 0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [ 0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [ 0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> >
> > "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS
> > support.
> >
> > Can you double check that you have the ITS driver in Xen built-in? That would
> > be CONFIG_HAS_ITS. If you do "make menuconfig" and enable "Configure
> > standard Xen features (expert users)" you should get a new option "GICv3
> > ITS MSI controller support" under "Architecture Features".
> > Make sure to enable it.
> >
> > Let me know if that works!
>