Mailing List Archive

Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
Adding Bertrand, Oleksandr, Julien, and others -- they have a more
recent experience with GICv3 ITS than me and might be able to help.
I am attaching the device tree Leo sent a few days ago for reference.


Typically when you can set the ethernet link up and no packets are
exchanged it is because of a missing interrupt. In this case a missing
MSI.

Bertrand, I believe you tried the GIC ITS driver with PCI devices
recently. It is expected to work correctly with MSIs in Dom0, right?



On Tue, 17 Nov 2020, Leo Krueger wrote:
> Hi,
>
> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...) but then had to add the following node to my device tree
>
> gic_lpi_base: syscon@0x80000000 {
> compatible = "gic-lpi-base";
> reg = <0x0 0x80000000 0x0 0x100000>;
> max-gic-redistributors = <2>;
> };
>
> to somehow change something in regard to the ITS and MSI/MSI-X
>
> (XEN) GICv3 initialization:
> (XEN) gic_dist_addr=0x00000006000000
> (XEN) gic_maintenance_irq=25
> (XEN) gic_rdist_stride=0
> (XEN) gic_rdist_regions=1
> (XEN) redistributor regions:
> (XEN) - region 0: 0x00000006040000 - 0x00000006080000
> (XEN) GICv3: using at most 57344 LPIs on the host.
> (XEN) GICv3: 288 lines, (IID 0001143b).
> (XEN) GICv3: Found ITS @0x6020000
> (XEN) using non-cacheable ITS command queue
> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>
> [ 0.000000] GICv3: Distributor has no Range Selector support
> [ 0.000000] GICv3: no VLPI support, no direct LPI support
> [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
> [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1)
> [ 0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
> [ 0.000000] GIC: using LPI property table @0x00000000dc830000
> [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
> [ 0.000000] CPU0: using LPI pending table @0x00000000dc840000
> ...
> [ 0.040080] Platform MSI: gic-its domain created
> [ 0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> [ 0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>
>
> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might fail!" log messages for some PCI devices, but at least the on-board ethernet ports (fsl_enetc ) are initialized.
> I can set the link up and a link is successfully established.
>
> But (!) I cannot receive or transmit anything (no error message...) and there seem to be no interrupts:
>
> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
>
> (from /proc/interrupts).
>
> Any idea on this one? I keep digging and checking whether my device tree needs some additional fixes.
>
> Kind regards,
> Leo
>
> --
> Leo Krüger, M.Sc.
> Senior Systems Engineer Distributed Systems
> Intelligent Digital Cabin
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Hein-Saß-Weg 22
> 21129 Hamburg
>  
> +49 (0) 40 248 595-154
>
> zal.aero | twitter.com/ZALTechCenter | facebook.com/ZALTechCenter
>
> ZAL Zentrum für Angewandte Luftfahrtforschung GmbH
> Sitz der Gesellschaft / Legal Domicile: Hamburg
> Registergericht / Registration Court: Amtsgericht Hamburg HRB 110232
> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: StR Andreas Rieckhof
> Geschäftsführung / Board of Management: Roland Gerhards
>
> Disclaimer:
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have
> received this mail in error), please notify the sender immediately and destroy this e-mail. Any unauthorised copying,
> disclosure or distribution of the material in this e-mail is strictly forbidden.
>
> > -----Ursprüngliche Nachricht-----
> > Von: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Gesendet: Dienstag, 17. November 2020 01:59
> > An: Leo Krueger <leo.krueger@zal.aero>
> > Cc: Peng Fan <peng.fan@nxp.com>; Stefano Stabellini
> > <stefano.stabellini@xilinx.com>; brucea@xilinx.com; Cornelia Bruelhart
> > <cornelia.bruelhart@zal.aero>
> > Betreff: Re: AW: AW: AW: Xen data from meta-virtualization layer
> >
> > Replies inline below
> >
> >
> > On Sun, 15 Nov 2020, Leo Krueger wrote:
> > > Hi Peng, hi Stefano,
> > >
> > >
> > >
> > > sorry for the long silence…
> > >
> > >
> > >
> > > I tried the change suggested (and hope I didn’t do anything wrong
> > > while doing so…) on top of XEN 4.13.2 (before, I always tried with
> > > 4.12 but wanted to give 4.13.2 a try as well) but I do not see any difference,
> > still the same “unhandled context fault” log entries pop up and I cannot
> > access my sdcard.
> > >
> > >
> > >
> > > As it seems to work without respectively disabled iommu, that would be
> > > fine for me for now. What I am worried about a bit more is PCIe or
> > MSI/MSIX to be exact.
> > >
> > >
> > >
> > > Here is the gic-v3 and its node from my device tree:
> > >
> > >
> > >
> > > interrupt-controller@6000000 {
> > >
> > >         compatible = "arm,gic-v3";
> > >
> > >         #address-cells = <0x2>;
> > >
> > >         #size-cells = <0x2>;
> > >
> > >         ranges;
> > >
> > >         reg = <0x0 0x6000000 0x0 0x10000 0x0 0x6040000 0x0 0x40000>;
> > >
> > >         #interrupt-cells = <0x3>;
> > >
> > >         interrupt-controller;
> > >
> > >         interrupts = <0x1 0x9 0xf08>;
> > >
> > >         phandle = <0x1>;
> > >
> > >
> > >
> > >         gic-its@6020000 {
> > >
> > >                 compatible = "arm,gic-v3-its";
> > >
> > >                 msi-controller;
> > >
> > >                 reg = <0x0 0x6020000 0x0 0x20000>;
> > >
> > >                 phandle = <0xd>;
> > >
> > >         };
> > >
> > > };
> > >
> > >
> > >
> > > And here are some kernel log excerpts related to GIC when booting
> > without (!) XEN:
> > >
> > >
> > >
> > > [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
> > >
> > > [    0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [    0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [    0.000000] ITS [mem 0x06020000-0x0603ffff]
> > >
> > > [    0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> > > @20fb880000 (flat, esz 8, psz 64K, shr 0)
> > >
> > > [    0.000000] ITS: using cache flushing for cmd queue
> > >
> > > [    0.000000] GIC: using LPI property table @0x00000020fb830000
> > >
> > > [    0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> > >
> > > [    0.000000] CPU0: using LPI pending table @0x00000020fb840000
> > >
> > > [    0.000000] GIC: using cache flushing for LPI property table
> > >
> > >
> > >
> > > However, when booting with XEN, only the following three lines are
> > contained in the kernel log:
> > >
> > >
> > >
> > > [    0.000000] GICv3: Distributor has no Range Selector support
> > >
> > > [    0.000000] GICv3: no VLPI support, no direct LPI support
> > >
> > > [    0.000000] GICv3: CPU0: found redistributor 0 region
> > > 0:0x0000000006040000
> >
> > "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS
> > support.
> >
> > Can you double check that you have the ITS driver in Xen built-in? That would
> > be CONFIG_HAS_ITS. If you do "make menuconfig" and enable "Configure
> > standard Xen features (expert users)" you should get a new option "GICv3
> > ITS MSI controller support" under "Architecture Features".
> > Make sure to enable it.
> >
> > Let me know if that works!
>
Re: AW: AW: AW: AW: Xen data from meta-virtualization layer [ In reply to ]
Hi,

On 17/11/2020 23:53, Stefano Stabellini wrote:
> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> recent experience with GICv3 ITS than me and might be able to help.
> I am attaching the device tree Leo sent a few days ago for reference.
>
>
> Typically when you can set the ethernet link up and no packets are
> exchanged it is because of a missing interrupt. In this case a missing
> MSI.
>
> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> recently. It is expected to work correctly with MSIs in Dom0, right?

OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
onwards on Thunder-X.

However, it may be possible that some more work is necessary for other
hardware (e.g. workaround, missing code...). See more below.

>
>
>
> On Tue, 17 Nov 2020, Leo Krueger wrote:
>> Hi,
>>
>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it before...) but then had to add the following node to my device tree
>>
>> gic_lpi_base: syscon@0x80000000 {
>> compatible = "gic-lpi-base";

I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
could you clarify which flavor/version of Linux you are using?

>> reg = <0x0 0x80000000 0x0 0x100000>;
>> max-gic-redistributors = <2>;
>> };
>>
>> to somehow change something in regard to the ITS and MSI/MSI-X
>>
>> (XEN) GICv3 initialization:
>> (XEN) gic_dist_addr=0x00000006000000
>> (XEN) gic_maintenance_irq=25
>> (XEN) gic_rdist_stride=0
>> (XEN) gic_rdist_regions=1
>> (XEN) redistributor regions:
>> (XEN) - region 0: 0x00000006040000 - 0x00000006080000
>> (XEN) GICv3: using at most 57344 LPIs on the host.
>> (XEN) GICv3: 288 lines, (IID 0001143b).
>> (XEN) GICv3: Found ITS @0x6020000
>> (XEN) using non-cacheable ITS command queue
>> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
>>
>> [ 0.000000] GICv3: Distributor has no Range Selector support
>> [ 0.000000] GICv3: no VLPI support, no direct LPI support
>> [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
>> [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000 (flat, esz 8, psz 64K, shr 1)
>> [ 0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
>> [ 0.000000] GIC: using LPI property table @0x00000000dc830000
>> [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000006040000
>> [ 0.000000] CPU0: using LPI pending table @0x00000000dc840000
>> ...
>> [ 0.040080] Platform MSI: gic-its domain created
>> [ 0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
>> [ 0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
>>
>>
>> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X might fail!" log messages for some PCI devices, but at least the on-board ethernet ports (fsl_enetc ) are initialized.
>> I can set the link up and a link is successfully established.

This message is normal. Xen on Arm is not yet aware of PCI devices and
therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.

However, this is not an issue because the virtual ITS implementation
will allow dom0 to configure any devices.

>>
>> But (!) I cannot receive or transmit anything (no error message...) and there seem to be no interrupts:
>>
>> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
>> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
>>
>> (from /proc/interrupts).
>>
>> Any idea on this one? I keep digging and checking whether my device tree needs some additional fixes.

Can you apply patch [1] and provide the logs? This will dump more
information about the command received by the vITS and the one send to
the host ITS.

Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
some of the messages.

[...]

>>>> [    0.000000] GICv3: Distributor has no Range Selector support
>>>>
>>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
>>>>
>>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
>>>> 0:0x0000000006040000
>>>
>>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't find any ITS
>>> support.
VLPI is a feature that was introduced in GICv4 to directly inject LPI in
the guest. So this is normal to see this message when running on Xen
because we are going to only expose a GICv3 to a domain until at least
we support nested virt.

However, you were right about that Xen didn't expose the ITS because the
following lines were missing:

[ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000
(flat, esz 8, psz 64K, shr 1)

Cheers,

[1]
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 9558bad96ac3..8a0a02308e74 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
const void *its_cmd)
/* No ITS commands from an interrupt handler (at the moment). */
ASSERT(!in_irq());

+ printk(XENLOG_WARNING, "pITS cmd 0x%02lx: %016lx %016lx %016lx
%016lx\n",
+ its_cmd_get_command(command),
+ command[0], command[1], command[2], command[3]);
+
spin_lock(&hw_its->cmd_lock);

do {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 869bc97fa1aa..e7c5bcd8d423 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -183,7 +183,10 @@ void gicv3_do_LPI(unsigned int lpi)
/* Find out if a guest mapped something to this physical LPI. */
hlpip = gic_get_host_lpi(lpi);
if ( !hlpip )
+ {
+ printk("%s: Received LPI %u but it is not mapped?\n", __func__,
lpi);
goto out;
+ }

hlpi.data = read_u64_atomic(&hlpip->data);

@@ -222,6 +225,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi,
int domain_id,
{
union host_lpi *hlpip, hlpi;

+ printk("%s: host_lpi %u domain %d virq_lpi %u\n",
+ __func__, host_lpi, domain_id, virq_lpi);
+
ASSERT(host_lpi >= LPI_OFFSET);

host_lpi -= LPI_OFFSET;
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 58d939b85f92..89ef137b3e6b 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -897,7 +897,7 @@ out_unlock:

static void dump_its_command(uint64_t *command)
{
- gdprintk(XENLOG_WARNING, " cmd 0x%02lx: %016lx %016lx %016lx
%016lx\n",
+ gdprintk(XENLOG_WARNING, "vITS cmd 0x%02lx: %016lx %016lx %016lx
%016lx\n",
its_cmd_get_command(command),
command[0], command[1], command[2], command[3]);
}
@@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d,
struct virt_its *its)
if ( ret )
return ret;

+ dump_its_command(command);
+
switch ( its_cmd_get_command(command) )
{
case GITS_CMD_CLEAR:


--
Julien Grall
AW: AW: AW: AW: AW: Xen data from meta-virtualization layer [ In reply to ]
Hi Julien,

finally I could try out what you suggested, please find my answers inline.

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xen.org>
> Gesendet: Mittwoch, 18. November 2020 13:24
> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
> <leo.krueger@zal.aero>
> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>
> Hi,
>
> On 17/11/2020 23:53, Stefano Stabellini wrote:
> > Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> > recent experience with GICv3 ITS than me and might be able to help.
> > I am attaching the device tree Leo sent a few days ago for reference.
> >
> >
> > Typically when you can set the ethernet link up and no packets are
> > exchanged it is because of a missing interrupt. In this case a missing
> > MSI.
> >
> > Bertrand, I believe you tried the GIC ITS driver with PCI devices
> > recently. It is expected to work correctly with MSIs in Dom0, right?
>
> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
> onwards on Thunder-X.
>
> However, it may be possible that some more work is necessary for other
> hardware (e.g. workaround, missing code...). See more below.
>
> >
> >
> >
> > On Tue, 17 Nov 2020, Leo Krueger wrote:
> >> Hi,
> >>
> >> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >> before...) but then had to add the following node to my device tree
> >>
> >> gic_lpi_base: syscon@0x80000000 {
> >> compatible = "gic-lpi-base";
>
> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
> you clarify which flavor/version of Linux you are using?

It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
While searching around the Internet for any solution, I came across [0] which contained the gic-lpi-base node.
So I just tried adding it (quite desperate I know) and voila, it at least brought me one step further (XEN exposing the ITS)...

>
> >> reg = <0x0 0x80000000 0x0 0x100000>;
> >> max-gic-redistributors = <2>;
> >> };
> >>
> >> to somehow change something in regard to the ITS and MSI/MSI-X
> >>
> >> (XEN) GICv3 initialization:
> >> (XEN) gic_dist_addr=0x00000006000000
> >> (XEN) gic_maintenance_irq=25
> >> (XEN) gic_rdist_stride=0
> >> (XEN) gic_rdist_regions=1
> >> (XEN) redistributor regions:
> >> (XEN) - region 0: 0x00000006040000 - 0x00000006080000
> >> (XEN) GICv3: using at most 57344 LPIs on the host.
> >> (XEN) GICv3: 288 lines, (IID 0001143b).
> >> (XEN) GICv3: Found ITS @0x6020000
> >> (XEN) using non-cacheable ITS command queue
> >> (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000
> >>
> >> [ 0.000000] GICv3: Distributor has no Range Selector support
> >> [ 0.000000] GICv3: no VLPI support, no direct LPI support
> >> [ 0.000000] ITS [mem 0x06020000-0x0603ffff]
> >> [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices
> @dc880000 (flat, esz 8, psz 64K, shr 1)
> >> [ 0.000000] ITS@0x0000000006020000: allocated 32768 Interrupt
> Collections @dc820000 (flat, esz 2, psz 64K, shr 1)
> >> [ 0.000000] GIC: using LPI property table @0x00000000dc830000
> >> [ 0.000000] GICv3: CPU0: found redistributor 0 region
> 0:0x0000000006040000
> >> [ 0.000000] CPU0: using LPI pending table @0x00000000dc840000
> >> ...
> >> [ 0.040080] Platform MSI: gic-its domain created
> >> [ 0.040136] PCI/MSI: /interrupt-controller/gic-its domain created
> >> [ 0.040181] fsl-mc MSI: /interrupt-controller/gic-its domain created
> >>
> >>
> >> Still I am ending up with the " Failed to add - passthrough or MSI/MSI-X
> might fail!" log messages for some PCI devices, but at least the on-board
> ethernet ports (fsl_enetc ) are initialized.
> >> I can set the link up and a link is successfully established.
>
> This message is normal. Xen on Arm is not yet aware of PCI devices and
> therefore the hypercalls to add PCI devices will return -EOPNOTSUPP.
>
> However, this is not an issue because the virtual ITS implementation will
> allow dom0 to configure any devices.
>
> >>
> >> But (!) I cannot receive or transmit anything (no error message...) and
> there seem to be no interrupts:
> >>
> >> 29: 0 ITS-MSI 1 Edge gbe0-rxtx0
> >> 32: 0 ITS-MSI 8192 Edge ptp_qoriq
> >>
> >> (from /proc/interrupts).
> >>
> >> Any idea on this one? I keep digging and checking whether my device tree
> needs some additional fixes.
>
> Can you apply patch [1] and provide the logs? This will dump more
> information about the command received by the vITS and the one send to
> the host ITS.

For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, quite ugly in parts).
Find attached the boot log and an output of "xl dmesg" which is truncated due to the large amount of messages.

When enabling the network interface (gbe0), the following output is visible:

root@kontron-sal28:~# ip link set up dev gbe0
(XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
(XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
[ 34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
[ 34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
[ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
root@kontron-sal28:~# [ 35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
[ 38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
[ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready

Does that tell you anything?

>
> Note that Xen will need to be build with CONFIG_DEBUG=y in order to get
> some of the messages.
>
> [...]
>
> >>>> [    0.000000] GICv3: Distributor has no Range Selector support
> >>>>
> >>>> [    0.000000] GICv3: no VLPI support, no direct LPI support
> >>>>
> >>>> [    0.000000] GICv3: CPU0: found redistributor 0 region
> >>>> 0:0x0000000006040000
> >>>
> >>> "no VLPI support" is very suspicious, it looks like Dom0 doesn't
> >>> find any ITS support.
> VLPI is a feature that was introduced in GICv4 to directly inject LPI in the
> guest. So this is normal to see this message when running on Xen because
> we are going to only expose a GICv3 to a domain until at least we support
> nested virt.
>
> However, you were right about that Xen didn't expose the ITS because the
> following lines were missing:
>
> [ 0.000000] ITS@0x0000000006020000: allocated 65536 Devices @dc880000
> (flat, esz 8, psz 64K, shr 1)
>
> Cheers,

Best regards,
Leo

>
> [1]
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c index
> 9558bad96ac3..8a0a02308e74 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its,
> const void *its_cmd)
> /* No ITS commands from an interrupt handler (at the moment). */
> ASSERT(!in_irq());
>
> + printk(XENLOG_WARNING, "pITS cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> + its_cmd_get_command(command),
> + command[0], command[1], command[2], command[3]);
> +
> spin_lock(&hw_its->cmd_lock);
>
> do {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index
> 869bc97fa1aa..e7c5bcd8d423 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -183,7 +183,10 @@ void gicv3_do_LPI(unsigned int lpi)
> /* Find out if a guest mapped something to this physical LPI. */
> hlpip = gic_get_host_lpi(lpi);
> if ( !hlpip )
> + {
> + printk("%s: Received LPI %u but it is not mapped?\n", __func__,
> lpi);
> goto out;
> + }
>
> hlpi.data = read_u64_atomic(&hlpip->data);
>
> @@ -222,6 +225,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi,
> int domain_id,
> {
> union host_lpi *hlpip, hlpi;
>
> + printk("%s: host_lpi %u domain %d virq_lpi %u\n",
> + __func__, host_lpi, domain_id, virq_lpi);
> +
> ASSERT(host_lpi >= LPI_OFFSET);
>
> host_lpi -= LPI_OFFSET;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c index
> 58d939b85f92..89ef137b3e6b 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -897,7 +897,7 @@ out_unlock:
>
> static void dump_its_command(uint64_t *command)
> {
> - gdprintk(XENLOG_WARNING, " cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> + gdprintk(XENLOG_WARNING, "vITS cmd 0x%02lx: %016lx %016lx %016lx
> %016lx\n",
> its_cmd_get_command(command),
> command[0], command[1], command[2], command[3]);
> }
> @@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d,
> struct virt_its *its)
> if ( ret )
> return ret;
>
> + dump_its_command(command);
> +
> switch ( its_cmd_get_command(command) )
> {
> case GITS_CMD_CLEAR:
>
>
> --
> Julien Grall

[0] https://www.mail-archive.com/u-boot@lists.denx.de/msg379708.html
[1]
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 9558bad96a..d175ba52b0 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -87,6 +87,10 @@ static int its_send_command(struct host_its *hw_its, const void *its_cmd)
/* No ITS commands from an interrupt handler (at the moment). */
ASSERT(!in_irq());

+ printk(XENLOG_WARNING "pITS cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
+ (((uint64_t *) its_cmd)[0] >> 0) & GENMASK(8 - 1, 0),
+ ((uint64_t *) its_cmd)[0], ((uint64_t *) its_cmd)[1], ((uint64_t *) its_cmd)[2], ((uint64_t *) its_cmd)[3]);
+
spin_lock(&hw_its->cmd_lock);

do {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 78b9521b21..2c3b0fc9e5 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -181,8 +181,10 @@ void gicv3_do_LPI(unsigned int lpi)

/* Find out if a guest mapped something to this physical LPI. */
hlpip = gic_get_host_lpi(lpi);
- if ( !hlpip )
+ if ( !hlpip ) {
+ printk("%s: Received LPI %u but it is not mapped?\n", __func__, lpi);
goto out;
+ }

hlpi.data = read_u64_atomic(&hlpip->data);

@@ -221,6 +223,9 @@ void gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
{
union host_lpi *hlpip, hlpi;

+ printk("%s: host_lpi %u domain %d virt_lpi %u\n",
+ __func__, host_lpi, domain_id, virt_lpi);
+
ASSERT(host_lpi >= LPI_OFFSET);

host_lpi -= LPI_OFFSET;
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 6e153c698d..dd5081ef80 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -897,7 +897,7 @@ out_unlock:

static void dump_its_command(uint64_t *command)
{
- gdprintk(XENLOG_WARNING, " cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
+ gdprintk(XENLOG_WARNING, "vITS cmd 0x%02lx: %016lx %016lx %016lx %016lx\n",
its_cmd_get_command(command),
command[0], command[1], command[2], command[3]);
}
@@ -926,6 +926,8 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its)
if ( ret )
return ret;

+ dump_its_command(command);
+
switch ( its_cmd_get_command(command) )
{
case GITS_CMD_CLEAR:
Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer [ In reply to ]
On 22/11/2020 22:55, Leo Krueger wrote:
> Hi Julien,

Hi Leo,

>
> finally I could try out what you suggested, please find my answers inline.

Thank you for sending the logs!

>
>> -----Ursprüngliche Nachricht-----
>> Von: Julien Grall <julien@xen.org>
>> Gesendet: Mittwoch, 18. November 2020 13:24
>> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
>> <leo.krueger@zal.aero>
>> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
>> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
>> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
>> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
>>
>> Hi,
>>
>> On 17/11/2020 23:53, Stefano Stabellini wrote:
>>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
>>> recent experience with GICv3 ITS than me and might be able to help.
>>> I am attaching the device tree Leo sent a few days ago for reference.
>>>
>>>
>>> Typically when you can set the ethernet link up and no packets are
>>> exchanged it is because of a missing interrupt. In this case a missing
>>> MSI.
>>>
>>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
>>> recently. It is expected to work correctly with MSIs in Dom0, right?
>>
>> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to boot
>> Dom0. I haven't seen any failure on recent Xen. We are testing 4.11 and
>> onwards on Thunder-X.
>>
>> However, it may be possible that some more work is necessary for other
>> hardware (e.g. workaround, missing code...). See more below.
>>
>>>
>>>
>>>
>>> On Tue, 17 Nov 2020, Leo Krueger wrote:
>>>> Hi,
>>>>
>>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
>>>> before...) but then had to add the following node to my device tree
>>>>
>>>> gic_lpi_base: syscon@0x80000000 {
>>>> compatible = "gic-lpi-base";
>>
>> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo, could
>> you clarify which flavor/version of Linux you are using?
>
> It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.

Do you have a link to the Linux tree? Is there any additional patches on
top of vanilla?

> While searching around the Internet for any solution, I came across [0] which contained the gic-lpi-base node.
> So I just tried adding it (quite desperate I know) and voila, it at least brought me one step further (XEN exposing the ITS)...

I am slightly confused to how this would help. Xen and, AFAICT, Linux
don't understand gic-lpi-base. Do you have modification in your Linux to
use it?

Looking at the DT changes in [0], it looks like the node is not a child
of gic@. So I think Xen will map the region to Dom0.

There are two things that I can notice:
1) This region is RAM, but I can't find any reserve node. Is there
any specific code in Linux to reserve it?
2) The implementation in U-boot seems to suggest that the firmware
will configure the LPIs and then enable it. If that's the case, then Xen
needs to re-use the table in the DT rather than allocating a new one.
However, I would have expected an error message in the log:

"GICv3: CPUx: Cannot initialize LPIs"

At least Xen should not expose gic-lpi-base to the kernel, but I will
wait on more details about the Linux kernel used before commenting more.

I would also be interested to know more details about the failure when
gic-lpi-base is not added in your DT. In particular, I am interested to
understand why Xen would not expose the ITS as we don't parse that node.

[...]

> For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know, quite ugly in parts).

No worries, debug patches are not meant to be nice to read ;).

> Find attached the boot log and an output of "xl dmesg" which is truncated due to the large amount of messages.
>
> When enabling the network interface (gbe0), the following output is visible:
>
> root@kontron-sal28:~# ip link set up dev gbe0
> (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 000000170000000c 0000000000000001 0000000000000000 0000000000000000
> (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0000000000000005 0000000000000000 0000000000000000 0000000000000000

0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt
by writing in the property table (access are not trapped to Xen) and
then requested to invalidate the cache state.

> [ 34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> [ 34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> [ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> root@kontron-sal28:~# [ 35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is Down
> [ 38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow control off
> [ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes ready
>
> Does that tell you anything?

It is at least a good sign because it means Linux is able to
initialize/talk to the vITS.

I would lean towards one (or multiple) issue with pITS and/or the
device-tree exposed to Linux. I am not entirely what exactly... I think
having more details about the Linux setup would be helpful.

I will reply on Rahul's e-mail separately.

Cheers,

--
Julien Grall
AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer [ In reply to ]
Hi,

> -----Ursprüngliche Nachricht-----
> Von: Julien Grall <julien@xen.org>
> Gesendet: Montag, 23. November 2020 19:27
> An: Leo Krueger <leo.krueger@zal.aero>; Stefano Stabellini
> <stefano.stabellini@xilinx.com>
> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia Bruelhart
> <cornelia.bruelhart@zal.aero>; oleksandr_andrushchenko@epam.com; xen-
> devel@lists.xenproject.org; Bertrand.Marquis@arm.com
> Betreff: Re: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer
>
>
>
> On 22/11/2020 22:55, Leo Krueger wrote:
> > Hi Julien,
>
> Hi Leo,
>
> >
> > finally I could try out what you suggested, please find my answers inline.
>
> Thank you for sending the logs!
>
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Julien Grall <julien@xen.org>
> >> Gesendet: Mittwoch, 18. November 2020 13:24
> >> An: Stefano Stabellini <stefano.stabellini@xilinx.com>; Leo Krueger
> >> <leo.krueger@zal.aero>
> >> Cc: Peng Fan <peng.fan@nxp.com>; brucea@xilinx.com; Cornelia
> >> Bruelhart <cornelia.bruelhart@zal.aero>;
> >> oleksandr_andrushchenko@epam.com; xen- devel@lists.xenproject.org;
> >> Bertrand.Marquis@arm.com
> >> Betreff: Re: AW: AW: AW: AW: Xen data from meta-virtualization layer
> >>
> >> Hi,
> >>
> >> On 17/11/2020 23:53, Stefano Stabellini wrote:
> >>> Adding Bertrand, Oleksandr, Julien, and others -- they have a more
> >>> recent experience with GICv3 ITS than me and might be able to help.
> >>> I am attaching the device tree Leo sent a few days ago for reference.
> >>>
> >>>
> >>> Typically when you can set the ethernet link up and no packets are
> >>> exchanged it is because of a missing interrupt. In this case a
> >>> missing MSI.
> >>>
> >>> Bertrand, I believe you tried the GIC ITS driver with PCI devices
> >>> recently. It is expected to work correctly with MSIs in Dom0, right?
> >>
> >> OSSTest has some hardware (e.g. Thunder-X) where ITS is required to
> >> boot Dom0. I haven't seen any failure on recent Xen. We are testing
> >> 4.11 and onwards on Thunder-X.
> >>
> >> However, it may be possible that some more work is necessary for
> >> other hardware (e.g. workaround, missing code...). See more below.
> >>
> >>>
> >>>
> >>>
> >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> >>>> Hi,
> >>>>
> >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> >>>> before...) but then had to add the following node to my device tree
> >>>>
> >>>> gic_lpi_base: syscon@0x80000000 {
> >>>> compatible = "gic-lpi-base";
> >>
> >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> >> could you clarify which flavor/version of Linux you are using?
> >
> > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
>
> Do you have a link to the Linux tree? Is there any additional patches on top of
> vanilla?

Linux tree is found here: https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
(up to the latest commit in that branch)

>
> > While searching around the Internet for any solution, I came across [0]
> which contained the gic-lpi-base node.
> > So I just tried adding it (quite desperate I know) and voila, it at least
> brought me one step further (XEN exposing the ITS)...
>
> I am slightly confused to how this would help. Xen and, AFAICT, Linux don't
> understand gic-lpi-base. Do you have modification in your Linux to use it?

I have no idea, to be honest. Maybe it is about the memory being reserved due to that node or something like that?

>
> Looking at the DT changes in [0], it looks like the node is not a child of gic@.
> So I think Xen will map the region to Dom0.
>
> There are two things that I can notice:
> 1) This region is RAM, but I can't find any reserve node. Is there any specific
> code in Linux to reserve it?
> 2) The implementation in U-boot seems to suggest that the firmware will
> configure the LPIs and then enable it. If that's the case, then Xen needs to
> re-use the table in the DT rather than allocating a new one.
> However, I would have expected an error message in the log:
>
> "GICv3: CPUx: Cannot initialize LPIs"
>
> At least Xen should not expose gic-lpi-base to the kernel, but I will wait on
> more details about the Linux kernel used before commenting more.
>
> I would also be interested to know more details about the failure when gic-
> lpi-base is not added in your DT. In particular, I am interested to understand
> why Xen would not expose the ITS as we don't parse that node.

How can I supply you with more information in regard to that? Without that node, ITS was not exposed at all.

>
> [...]
>
> > For XEN 4.13.2 I had to adapt your patch slightly [1], see below (yes I know,
> quite ugly in parts).
>
> No worries, debug patches are not meant to be nice to read ;).
>
> > Find attached the boot log and an output of "xl dmesg" which is truncated
> due to the large amount of messages.
> >
> > When enabling the network interface (gbe0), the following output is
> visible:
> >
> > root@kontron-sal28:~# ip link set up dev gbe0
> > (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x0c: 000000170000000c
> > 0000000000000001 0000000000000000 0000000000000000
> > (XEN) vgic-v3-its.c:902:d0v0 vITS cmd 0x05: 0000000000000005
> > 0000000000000000 0000000000000000 0000000000000000
>
> 0xc is INV and 0x5 is SYNC. Most likely the driver unmask the interrupt by
> writing in the property table (access are not trapped to Xen) and then
> requested to invalidate the cache state.
>
> > [ 34.034598] Atheros 8031 ethernet 0000:00:00.3:05: attached PHY driver
> [Atheros 8031 ethernet] (mii_bus:phy_addr=0000:00:00.3:05, irq=POLL)
> > [ 34.041111] 8021q: adding VLAN 0 to HW filter on device gbe0
> > [ 34.041209] IPv6: ADDRCONF(NETDEV_UP): gbe0: link is not ready
> > root@kontron-sal28:~# [ 35.041951] fsl_enetc 0000:00:00.0 gbe0: Link is
> Down
> > [ 38.114426] fsl_enetc 0000:00:00.0 gbe0: Link is Up - 1Gbps/Full - flow
> control off
> > [ 38.114508] IPv6: ADDRCONF(NETDEV_CHANGE): gbe0: link becomes
> ready
> >
> > Does that tell you anything?
>
> It is at least a good sign because it means Linux is able to initialize/talk to the
> vITS.
>
> I would lean towards one (or multiple) issue with pITS and/or the device-tree
> exposed to Linux. I am not entirely what exactly... I think having more details
> about the Linux setup would be helpful.

Ok let me know what you need from my side. I was considering the following things to try out next:

- a more recent u-boot version as this might fix problems with the msi-map (at least that is what I think, I am not an expert here)
- a different device tree (a more recent one, ...)
- ...

>
> I will reply on Rahul's e-mail separately.
>
> Cheers,

Best wishes!

>
> --
> Julien Grall
Re: AW: AW: AW: AW: AW: AW: Xen data from meta-virtualization layer [ In reply to ]
+ Zhiqiang Hou

On Tue, 24 Nov 2020, Leo Krueger wrote:
> > >>> On Tue, 17 Nov 2020, Leo Krueger wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I enabled CONFIG_HAS_ITS (what a stupid mistake by me to not set it
> > >>>> before...) but then had to add the following node to my device tree
> > >>>>
> > >>>> gic_lpi_base: syscon@0x80000000 {
> > >>>> compatible = "gic-lpi-base";
> > >>
> > >> I couldn't find this compatible defined/used in Linux 5.10-rc4. @Leo,
> > >> could you clarify which flavor/version of Linux you are using?
> > >
> > > It is Linux 4.19 from Yocto (Warror release). XEN 4.13.2.
> >
> > Do you have a link to the Linux tree? Is there any additional patches on top of
> > vanilla?
>
> Linux tree is found here: https://github.com/kontron/linux-smarc-sal28/commits/master-LSDK-19.09
> (up to the latest commit in that branch)

[...]

> > Looking at the DT changes in [0], it looks like the node is not a child of gic@.
> > So I think Xen will map the region to Dom0.
> >
> > There are two things that I can notice:
> > 1) This region is RAM, but I can't find any reserve node. Is there any specific
> > code in Linux to reserve it?
> > 2) The implementation in U-boot seems to suggest that the firmware will
> > configure the LPIs and then enable it. If that's the case, then Xen needs to
> > re-use the table in the DT rather than allocating a new one.

That Linux tree has no mentions of gic-lpi-base. That means that
gic-lpi-base is only used in u-boot, not in Linux. In particular the
most relevant commit is af288cb291da3abef6be0875527729296f7de7a0.

In regards to the reserved-memory regions, maybe we are not seeing them
because Leo posted the host device tree, not the one passed at runtime
from u-boot to Linux?

If so, Leo, could you please boot Linux on native (no Xen) and get the
device tree from there at runtime using dtc -I fs -O dts
/proc/device-tree ?


However, the name of the reserved-memory region created by u-boot seems
to be "lpi_rd_table". I cannot find any mentions of lpi_rd_table in the
Linux kernel tree either.

Zhiqiang, Leo is trying to boot Xen on sAL28. Linux booting on Xen
throws errors in regards to GIC/ITS initialization. On other hardware
Xen can use and virtualize GICv3 and ITS just fine. Could you please
explain what is different about sAL28 and how Xen/Linux is expected to
use the lpi_rd_table reserved-memory region?