Mailing List Archive

On 26/04/2021 15:41, Prem Harikrishnan wrote:
> Hello everyone,

Hi Prem,

> I am using the latest version of the xilinx mpsoc zcu102 development
> board and i am trying to create a barematal application that uses the
> AXI DMA on the programmable logic, using the scatter gather example from
> the xilinx repository. This works perfectly well for the baremetal
> application. The diagram below represents the information flow. The AXI
> DMA and the AXI data FIFO are in the Programable logic (FPGA). This AXI
> DMA is out of the box IP from Vivado and uses a single channel (Does not
> use AxUSER) which means the AXI ID cannot manually be set.
>
>
> Now i want to make this as a baremetal hypervisor guest, in future this
> will be replaced by a custom IP with cache colouring enabled. I want to
> pass through the AXI DMA in the PL. (Not to be used by Dom0)
>
>
> I followed this tutorial Baremetal + DomU
> <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> image and use tftp to boot.
>
> From what i understand from the xilinx tutorials Xen+and+PL+Masters
> <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> section 3 and MPSOC SMMU
> <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> section 4, stream ids are required by the SMMU to get the correct
> address translation.
>
> I am still unable to figure out how to get the iommu property for my AXI
> DMA and how to set the dtdev config for my hypervisor guest.
>
>
> I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> which i can be useful to my application
>
> * ?1:1 memory mappings for Xen virtual machines, enabling device
> assignments without SMMU
> *
> ?Static assignment of PL blocks, including bus-mastering blocks, to
> Xen virtual machines
> * ?Support for PL Device Passthrough to DomU in Xen

You are usiong a version of Xen that has been customised by Xilinx. So I
would recommend to contact them directly.

I have CCed Stefano who might be able to help you.

Best regards,

--
Julien Grall

On Mon, 10 May 2021, Julien Grall wrote:
> On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > Hello everyone,
>
> Hi Prem,
>
> > I am using the latest version of the xilinx mpsoc zcu102 development board
> > and i am trying to create a barematal application that uses the AXI DMA on
> > the programmable logic, using the scatter gather example from the xilinx
> > repository. This works perfectly well for the baremetal application. The
> > diagram below represents the information flow. The AXI DMA and the AXI data
> > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > AXI ID cannot manually be set.
> >
> >
> > Now i want to make this as a baremetal hypervisor guest, in future this will
> > be replaced by a custom IP with cache colouring enabled. I want to pass
> > through the AXI DMA in the PL. (Not to be used by Dom0)
> >
> >
> > I followed this tutorial Baremetal + DomU
> > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > image and use tftp to boot.
> >
> > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > section 3 and MPSOC SMMU
> > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > section 4, stream ids are required by the SMMU to get the correct address
> > translation.
> >
> > I am still unable to figure out how to get the iommu property for my AXI DMA
> > and how to set the dtdev config for my hypervisor guest.
> >
> >
> > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > which i can be useful to my application
> >
> > *  1:1 memory mappings for Xen virtual machines, enabling device
> > assignments without SMMU
> > *
> >  Static assignment of PL blocks, including bus-mastering blocks, to
> > Xen virtual machines
> > *  Support for PL Device Passthrough to DomU in Xen
>
> You are usiong a version of Xen that has been customised by Xilinx. So I would
> recommend to contact them directly.
>
> I have CCed Stefano who might be able to help you.

Hi Julien,

Many thanks for CC'ing me, I missed the email.


Hi Prem,

Let me CC a few people that might be able to help.

Prem,

Sorry for the late email, I've been under the weather and have been away
from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
calculated:

14-10: TBU bits
9-6: Master ID bits
5-0: AXI ID bits

https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
To look up the TBU value, see UG1085 and look for the "System Masters"
table which is table 16-4 on page 409.

For the master ID, see UG1085 and look for the "Master IDs List" table
which is table 16-13 on page 429.

The AXI ID bits is a bit trickery since it's going to be out in PL.
See: https://www.xilinx.com/support/answers/69447.html
With SmartConnect all the AXI IDs are 0, they aren't passed on. There
are work arounds. Here's a couple of examples (with example numbers):

AXI master -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001

AXI master -> SmartConnect -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000

AXI master -> SmartConnect -> PS slave port 0
-> PS slave port 1
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000

Two AXI masters -> SmartConnect -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)

Two AXI masters -> SmartConnect -> PS slave port 0
-> PS slave port 1
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)

Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)

Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
-> PS slave port 1
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)

The "sideband -> SmartConenct -> sideband" can also be functionally
replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
fairly simple but with some caveats.

You can also read in the SMMU faults from Xen and use:
https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
to decode them. That should tell you the SMMU IDs as well.

Let me know if this isn't clear or you have any other questions.

Brian

On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> On Mon, 10 May 2021, Julien Grall wrote:
> > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > Hello everyone,
> >
> > Hi Prem,
> >
> > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > and i am trying to create a barematal application that uses the AXI DMA on
> > > the programmable logic, using the scatter gather example from the xilinx
> > > repository. This works perfectly well for the baremetal application. The
> > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > AXI ID cannot manually be set.
> > >
> > >
> > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > >
> > >
> > > I followed this tutorial Baremetal + DomU
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > image and use tftp to boot.
> > >
> > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > section 3 and MPSOC SMMU
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > section 4, stream ids are required by the SMMU to get the correct address
> > > translation.
> > >
> > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > and how to set the dtdev config for my hypervisor guest.
> > >
> > >
> > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > which i can be useful to my application
> > >
> > > * ?1:1 memory mappings for Xen virtual machines, enabling device
> > > assignments without SMMU
> > > *
> > > ?Static assignment of PL blocks, including bus-mastering blocks, to
> > > Xen virtual machines
> > > * ?Support for PL Device Passthrough to DomU in Xen
> >
> > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > recommend to contact them directly.
> >
> > I have CCed Stefano who might be able to help you.
>
> Hi Julien,
>
> Many thanks for CC'ing me, I missed the email.
>
>
> Hi Prem,
>
> Let me CC a few people that might be able to help.

Dear Brian, Stefano and Julien,

Thank you very much for your replies.
Hi Brian and Stefano, Can you please help me with the following questions?

@Brian Woods<mailto:brian.woods@xilinx.com> Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i tried it with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1 master and I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs. https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually tied of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I can update you soon with an ILA trace on Vivado.

@Stefano Stabellini<mailto:stefano.stabellini@xilinx.com> Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding xen.dtsi. https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this SATA passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my xen configuration.

Thanks and Best Regards,
Prem
________________________________
From: Brian Woods <brian.woods@xilinx.com>
Sent: Thursday, May 13, 2021 17:13
To: Stefano Stabellini <stefano.stabellini@xilinx.com>
Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

Prem,

Sorry for the late email, I've been under the weather and have been away
from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
calculated:

14-10: TBU bits
9-6: Master ID bits
5-0: AXI ID bits

https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
To look up the TBU value, see UG1085 and look for the "System Masters"
table which is table 16-4 on page 409.

For the master ID, see UG1085 and look for the "Master IDs List" table
which is table 16-13 on page 429.

The AXI ID bits is a bit trickery since it's going to be out in PL.
See: https://www.xilinx.com/support/answers/69447.html
With SmartConnect all the AXI IDs are 0, they aren't passed on. There
are work arounds. Here's a couple of examples (with example numbers):

AXI master -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001

AXI master -> SmartConnect -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000

AXI master -> SmartConnect -> PS slave port 0
-> PS slave port 1
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000

Two AXI masters -> SmartConnect -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)

Two AXI masters -> SmartConnect -> PS slave port 0
-> PS slave port 1
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)

Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)

Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
-> PS slave port 1
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)

The "sideband -> SmartConenct -> sideband" can also be functionally
replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
fairly simple but with some caveats.

You can also read in the SMMU faults from Xen and use:
https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
to decode them. That should tell you the SMMU IDs as well.

Let me know if this isn't clear or you have any other questions.

Brian

On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> On Mon, 10 May 2021, Julien Grall wrote:
> > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > Hello everyone,
> >
> > Hi Prem,
> >
> > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > and i am trying to create a barematal application that uses the AXI DMA on
> > > the programmable logic, using the scatter gather example from the xilinx
> > > repository. This works perfectly well for the baremetal application. The
> > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > AXI ID cannot manually be set.
> > >
> > >
> > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > >
> > >
> > > I followed this tutorial Baremetal + DomU
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > image and use tftp to boot.
> > >
> > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > section 3 and MPSOC SMMU
> > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > section 4, stream ids are required by the SMMU to get the correct address
> > > translation.
> > >
> > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > and how to set the dtdev config for my hypervisor guest.
> > >
> > >
> > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > which i can be useful to my application
> > >
> > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > assignments without SMMU
> > > *
> > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > Xen virtual machines
> > > * Support for PL Device Passthrough to DomU in Xen
> >
> > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > recommend to contact them directly.
> >
> > I have CCed Stefano who might be able to help you.
>
> Hi Julien,
>
> Many thanks for CC'ing me, I missed the email.
>
>
> Hi Prem,
>
> Let me CC a few people that might be able to help.

I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
a baremetal application? Can you run it successfully, just printing
"hello world", if you only assign the UART and nothing else?

I am asking just to verify that you can run the baremetal VM
successfully, aside from the PL device assignment.


In regards to the PL device assignment, I'll let Brian answer on the
Vivado configuration and SMID calculation. But I noticed that you only
assigned IRQ 54 to the DomU which is the one for UART1. You haven't
assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
looks like there are two related IRQs: 121 and 122.

Finally, why did you mention SATA passthrough in your reply? I don't
follow: I cannot see any SATA controller assignment in your
configuration.


On Fri, 14 May 2021, Prem Harikrishnan wrote:
> Dear Brian, Stefano and Julien,
>
> Thank you very much for your replies.
> Hi Brian and Stefano, Can you please help me with the following questions?
>
> @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i tried it
> with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1 master and
> I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually tied
> of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I can
> update you soon with an ILA trace on Vivado.
>
> @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding xen.dtsi.
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this SATA
> passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my xen
> configuration.
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Brian Woods <brian.woods@xilinx.com>
> Sent: Thursday, May 13, 2021 17:13
> To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>;
> xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> Prem,
>
> Sorry for the late email, I've been under the weather and have been away
> from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> calculated:
>
> 14-10: TBU bits
> 9-6: Master ID bits
> 5-0: AXI ID bits
>
> https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> To look up the TBU value, see UG1085 and look for the "System Masters"
> table which is table 16-4 on page 409.
>
> For the master ID, see UG1085 and look for the "Master IDs List" table
> which is table 16-13 on page 429.
>
> The AXI ID bits is a bit trickery since it's going to be out in PL.
> See: https://www.xilinx.com/support/answers/69447.html
> With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> are work arounds.  Here's a couple of examples (with example numbers):
>
> AXI master -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
>
> AXI master -> SmartConnect -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
>
> AXI master -> SmartConnect -> PS slave port 0
>                            -> PS slave port 1
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
>
> Two AXI masters -> SmartConnect -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
>
> Two AXI masters -> SmartConnect -> PS slave port 0
>                                 -> PS slave port 1
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
>
> Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
>
> Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
>                                                         -> PS slave port 1
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
>         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
>         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
>
> The "sideband -> SmartConenct -> sideband" can also be functionally
> replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> fairly simple but with some caveats.
>
> You can also read in the SMMU faults from Xen and use:
> https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> to decode them. That should tell you the SMMU IDs as well.
>
> Let me know if this isn't clear or you have any other questions.
>
> Brian
>
> On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > On Mon, 10 May 2021, Julien Grall wrote:
> > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > Hello everyone,
> > >
> > > Hi Prem,
> > >
> > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > the programmable logic, using the scatter gather example from the xilinx
> > > > repository. This works perfectly well for the baremetal application. The
> > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > AXI ID cannot manually be set.
> > > >
> > > >
> > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > >
> > > >
> > > > I followed this tutorial Baremetal + DomU
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > image and use tftp to boot.
> > > >
> > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > section 3 and MPSOC SMMU
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > translation.
> > > >
> > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > and how to set the dtdev config for my hypervisor guest.
> > > >
> > > >
> > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > which i can be useful to my application
> > > >
> > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > >     assignments without SMMU
> > > >   *
> > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > >     Xen virtual machines
> > > >   *   Support for PL Device Passthrough to DomU in Xen
> > >
> > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > recommend to contact them directly.
> > >
> > > I have CCed Stefano who might be able to help you.
> >
> > Hi Julien,
> >
> > Many thanks for CC'ing me, I missed the email.
> >
> >
> > Hi Prem,
> >
> > Let me CC a few people that might be able to help.
>
>

Hi Stefano,
Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
Also yes, I have already run hello world as DomU guest, with UART1.

I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the guest config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the function xil_dcacheflushrange(). Please find the c code attached in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c

I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass through in the guest config.(from pl.dtsi, which I manually created).

Thanks and Best Regards,
Prem

Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com>
Sent: Friday 14 May 2021, 23:16
To: Prem Harikrishnan
Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com; edgari@xilinx.com; bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
a baremetal application? Can you run it successfully, just printing
"hello world", if you only assign the UART and nothing else?

I am asking just to verify that you can run the baremetal VM
successfully, aside from the PL device assignment.


In regards to the PL device assignment, I'll let Brian answer on the
Vivado configuration and SMID calculation. But I noticed that you only
assigned IRQ 54 to the DomU which is the one for UART1. You haven't
assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
looks like there are two related IRQs: 121 and 122.

Finally, why did you mention SATA passthrough in your reply? I don't
follow: I cannot see any SATA controller assignment in your
configuration.


On Fri, 14 May 2021, Prem Harikrishnan wrote:
> Dear Brian, Stefano and Julien,
>
> Thank you very much for your replies.
> Hi Brian and Stefano, Can you please help me with the following questions?
>
> @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i tried it
> with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1 master and
> I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually tied
> of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I can
> update you soon with an ILA trace on Vivado.
>
> @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding xen.dtsi.
> https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this SATA
> passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my xen
> configuration.
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Brian Woods <brian.woods@xilinx.com>
> Sent: Thursday, May 13, 2021 17:13
> To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>;
> xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> Prem,
>
> Sorry for the late email, I've been under the weather and have been away
> from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> calculated:
>
> 14-10: TBU bits
> 9-6: Master ID bits
> 5-0: AXI ID bits
>
> https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> To look up the TBU value, see UG1085 and look for the "System Masters"
> table which is table 16-4 on page 409.
>
> For the master ID, see UG1085 and look for the "Master IDs List" table
> which is table 16-13 on page 429.
>
> The AXI ID bits is a bit trickery since it's going to be out in PL.
> See: https://www.xilinx.com/support/answers/69447.html
> With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> are work arounds. Here's a couple of examples (with example numbers):
>
> AXI master -> PS slave port 0
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
>
> AXI master -> SmartConnect -> PS slave port 0
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
>
> AXI master -> SmartConnect -> PS slave port 0
> -> PS slave port 1
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
>
> Two AXI masters -> SmartConnect -> PS slave port 0
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
>
> Two AXI masters -> SmartConnect -> PS slave port 0
> -> PS slave port 1
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
>
> Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
>
> Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> -> PS slave port 1
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
>
> The "sideband -> SmartConenct -> sideband" can also be functionally
> replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> fairly simple but with some caveats.
>
> You can also read in the SMMU faults from Xen and use:
> https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> to decode them. That should tell you the SMMU IDs as well.
>
> Let me know if this isn't clear or you have any other questions.
>
> Brian
>
> On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > On Mon, 10 May 2021, Julien Grall wrote:
> > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > Hello everyone,
> > >
> > > Hi Prem,
> > >
> > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > the programmable logic, using the scatter gather example from the xilinx
> > > > repository. This works perfectly well for the baremetal application. The
> > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > AXI ID cannot manually be set.
> > > >
> > > >
> > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > >
> > > >
> > > > I followed this tutorial Baremetal + DomU
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > image and use tftp to boot.
> > > >
> > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > section 3 and MPSOC SMMU
> > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > translation.
> > > >
> > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > and how to set the dtdev config for my hypervisor guest.
> > > >
> > > >
> > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > which i can be useful to my application
> > > >
> > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > assignments without SMMU
> > > > *
> > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > Xen virtual machines
> > > > * Support for PL Device Passthrough to DomU in Xen
> > >
> > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > recommend to contact them directly.
> > >
> > > I have CCed Stefano who might be able to help you.
> >
> > Hi Julien,
> >
> > Many thanks for CC'ing me, I missed the email.
> >
> >
> > Hi Prem,
> >
> > Let me CC a few people that might be able to help.
>
>

Hi Prem,

Some older versions of Xil_DCacheFlushRange were implemented using
deprecated set/way instructions that are known to cause problems in
virtualized environments.

More recent implementations use the "dc civac" intruction that shouldn't
have any issues.

Can you double-check that you are calling the 64-bit version implemented
by CIVAC?

https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L414



On Fri, 14 May 2021, Prem Harikrishnan wrote:
> Hi Stefano, 
> Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example. 
> Also yes, I have already run hello world as DomU guest, with UART1.
>
> I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the guest
> config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> function xil_dcacheflushrange(). Please find the c code attached
> in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c 
>
> I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass through
> in the guest config.(from pl.dtsi, which I manually created).
>
> Thanks and Best Regards,
> Prem
>
> Get Outlook for Android
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Friday 14 May 2021, 23:16
> To: Prem Harikrishnan
> Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com; edgari@xilinx.com;
> bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
>
> I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> a baremetal application? Can you run it successfully, just printing
> "hello world", if you only assign the UART and nothing else?
>
> I am asking just to verify that you can run the baremetal VM
> successfully, aside from the PL device assignment.
>
>
> In regards to the PL device assignment, I'll let Brian answer on the
> Vivado configuration and SMID calculation. But I noticed that you only
> assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> looks like there are two related IRQs: 121 and 122.
>
> Finally, why did you mention SATA passthrough in your reply? I don't
> follow: I cannot see any SATA controller assignment in your
> configuration.
>
>
> On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > Dear Brian, Stefano and Julien,
> >
> > Thank you very much for your replies.
> > Hi Brian and Stefano, Can you please help me with the following questions?
> >
> > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i tried
> it
> > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1 master
> and
> > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually tied
> > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I can
> > update you soon with an ILA trace on Vivado.
> >
> > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> xen.dtsi.
> > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this SATA
> > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my xen
> > configuration.
> >
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Brian Woods <brian.woods@xilinx.com>
> > Sent: Thursday, May 13, 2021 17:13
> > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>;
> > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > Prem,
> >
> > Sorry for the late email, I've been under the weather and have been away
> > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > calculated:
> >
> > 14-10: TBU bits
> > 9-6: Master ID bits
> > 5-0: AXI ID bits
> >
> > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > To look up the TBU value, see UG1085 and look for the "System Masters"
> > table which is table 16-4 on page 409.
> >
> > For the master ID, see UG1085 and look for the "Master IDs List" table
> > which is table 16-13 on page 429.
> >
> > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > See: https://www.xilinx.com/support/answers/69447.html
> > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > are work arounds.  Here's a couple of examples (with example numbers):
> >
> > AXI master -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> >
> > AXI master -> SmartConnect -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> >
> > AXI master -> SmartConnect -> PS slave port 0
> >                            -> PS slave port 1
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> >
> > Two AXI masters -> SmartConnect -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> >
> > Two AXI masters -> SmartConnect -> PS slave port 0
> >                                 -> PS slave port 1
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> >
> > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> >
> > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> >                                                         -> PS slave port 1
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> >
> > The "sideband -> SmartConenct -> sideband" can also be functionally
> > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> > fairly simple but with some caveats.
> >
> > You can also read in the SMMU faults from Xen and use:
> > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > to decode them. That should tell you the SMMU IDs as well.
> >
> > Let me know if this isn't clear or you have any other questions.
> >
> > Brian
> >
> > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > Hello everyone,
> > > >
> > > > Hi Prem,
> > > >
> > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > repository. This works perfectly well for the baremetal application. The
> > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > AXI ID cannot manually be set.
> > > > >
> > > > >
> > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > >
> > > > >
> > > > > I followed this tutorial Baremetal + DomU
> > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > image and use tftp to boot.
> > > > >
> > > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > section 3 and MPSOC SMMU
> > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > translation.
> > > > >
> > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > and how to set the dtdev config for my hypervisor guest.
> > > > >
> > > > >
> > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > which i can be useful to my application
> > > > >
> > > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > > >     assignments without SMMU
> > > > >   *
> > > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > > >     Xen virtual machines
> > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > >
> > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > recommend to contact them directly.
> > > >
> > > > I have CCed Stefano who might be able to help you.
> > >
> > > Hi Julien,
> > >
> > > Many thanks for CC'ing me, I missed the email.
> > >
> > >
> > > Hi Prem,
> > >
> > > Let me CC a few people that might be able to help.
> >
> >
>
>
>

Dear Stefano,
Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed out. I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the DomU memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you pointed out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.

Thanks again i was able get my custom IP also working as an AXI master from PL.

Best Regards,
Prem
________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com>
Sent: Tuesday, May 18, 2021 02:35
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>; JOVITAC@xilinx.com <JOVITAC@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

Your configuration looks correct as far as I can tell.

You need dtdev because it triggers the SMMU configuration for your PL
block. You don't need device_tree because your baremetal application is
not going to read it, but it is not going to cause any issues either.

When you say "the execution does not complete the loop at line 244", it
means that you see the execution entering the loop and then never
leaving? Like an infinite loop?

If so, then I imagine the issue could be that the application is
crashing when trying to access TxBufferPtr, which is pointing to
TX_BUFFER_BASE, which is:


#ifndef DDR_BASE_ADDR
#warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
DEFAULT SET TO 0x01000000
#define MEM_BASE_ADDR 0x01000000
#else
#define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
#endif

#define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)



Can you double check that TX_BUFFER_BASE is pointing to a correct
address? Memory starts at 0x40000000 in a Xen DomU.

Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?


On Mon, 17 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works only
> in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> I am still unable to find out why??
>
> Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file when it
> is already added to xen.dtsi?
>
> Best Regards,
> Prem
>
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Saturday, May 15, 2021 00:25
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> Hi Prem,
>
> Some older versions of Xil_DCacheFlushRange were implemented using
> deprecated set/way instructions that are known to cause problems in
> virtualized environments.
>
> More recent implementations use the "dc civac" intruction that shouldn't
> have any issues.
>
> Can you double-check that you are calling the 64-bit version implemented
> by CIVAC?
>
> https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L414
>
>
>
>
> On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > Also yes, I have already run hello world as DomU guest, with UART1.
> >
> > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the guest
> > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > function xil_dcacheflushrange(). Please find the c code attached
> > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> >
> > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> through
> > in the guest config.(from pl.dtsi, which I manually created).
> >
> > Thanks and Best Regards,
> > Prem
> >
> > Get Outlook for Android
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Friday 14 May 2021, 23:16
> > To: Prem Harikrishnan
> > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> edgari@xilinx.com;
> > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> >
> > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > a baremetal application? Can you run it successfully, just printing
> > "hello world", if you only assign the UART and nothing else?
> >
> > I am asking just to verify that you can run the baremetal VM
> > successfully, aside from the PL device assignment.
> >
> >
> > In regards to the PL device assignment, I'll let Brian answer on the
> > Vivado configuration and SMID calculation. But I noticed that you only
> > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > looks like there are two related IRQs: 121 and 122.
> >
> > Finally, why did you mention SATA passthrough in your reply? I don't
> > follow: I cannot see any SATA controller assignment in your
> > configuration.
> >
> >
> > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > Dear Brian, Stefano and Julien,
> > >
> > > Thank you very much for your replies.
> > > Hi Brian and Stefano, Can you please help me with the following questions?
> > >
> > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i
> tried
> > it
> > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1 master
> > and
> > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually
> tied
> > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I can
> > > update you soon with an ILA trace on Vivado.
> > >
> > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > xen.dtsi.
> > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this
> SATA
> > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my xen
> > > configuration.
> > >
> > > Thanks and Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Brian Woods <brian.woods@xilinx.com>
> > > Sent: Thursday, May 13, 2021 17:13
> > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>;
> > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Prem,
> > >
> > > Sorry for the late email, I've been under the weather and have been away
> > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > calculated:
> > >
> > > 14-10: TBU bits
> > > 9-6: Master ID bits
> > > 5-0: AXI ID bits
> > >
> > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > table which is table 16-4 on page 409.
> > >
> > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > which is table 16-13 on page 429.
> > >
> > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > See: https://www.xilinx.com/support/answers/69447.html
> > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > are work arounds. Here's a couple of examples (with example numbers):
> > >
> > > AXI master -> PS slave port 0
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > >
> > > AXI master -> SmartConnect -> PS slave port 0
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > >
> > > AXI master -> SmartConnect -> PS slave port 0
> > > -> PS slave port 1
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > >
> > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > >
> > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > -> PS slave port 1
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > >
> > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > >
> > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > -> PS slave port 1
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > >
> > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > fairly simple but with some caveats.
> > >
> > > You can also read in the SMMU faults from Xen and use:
> > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > to decode them. That should tell you the SMMU IDs as well.
> > >
> > > Let me know if this isn't clear or you have any other questions.
> > >
> > > Brian
> > >
> > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > Hello everyone,
> > > > >
> > > > > Hi Prem,
> > > > >
> > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > AXI ID cannot manually be set.
> > > > > >
> > > > > >
> > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > >
> > > > > >
> > > > > > I followed this tutorial Baremetal + DomU
> > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > image and use tftp to boot.
> > > > > >
> > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > section 3 and MPSOC SMMU
> > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > translation.
> > > > > >
> > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > >
> > > > > >
> > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > which i can be useful to my application
> > > > > >
> > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > assignments without SMMU
> > > > > > *
> > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > Xen virtual machines
> > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > >
> > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > recommend to contact them directly.
> > > > >
> > > > > I have CCed Stefano who might be able to help you.
> > > >
> > > > Hi Julien,
> > > >
> > > > Many thanks for CC'ing me, I missed the email.
> > > >
> > > >
> > > > Hi Prem,
> > > >
> > > > Let me CC a few people that might be able to help.
> > >
> > >
> >
> >
> >
>
>

Great to hear! Thank you for reporting back.


On Tue, 18 May 2021, Prem Harikrishnan wrote:
> Dear Stefano,
> Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed out.
> I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the DomU
> memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you pointed
> out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
>
> Thanks again i was able get my custom IP also working as an AXI master from PL.
>
> Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Tuesday, May 18, 2021 02:35
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> Your configuration looks correct as far as I can tell.
>
> You need dtdev because it triggers the SMMU configuration for your PL
> block. You don't need device_tree because your baremetal application is
> not going to read it, but it is not going to cause any issues either.
>
> When you say "the execution does not complete the loop at line 244", it
> means that you see the execution entering the loop and then never
> leaving? Like an infinite loop?
>
> If so, then I imagine the issue could be that the application is
> crashing when trying to access TxBufferPtr, which is pointing to
> TX_BUFFER_BASE, which is:
>
>
> #ifndef DDR_BASE_ADDR
> #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
>                  DEFAULT SET TO 0x01000000
> #define MEM_BASE_ADDR           0x01000000
> #else
> #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> #endif
>
> #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
>
>
>
> Can you double check that TX_BUFFER_BASE is pointing to a correct
> address? Memory starts at 0x40000000 in a Xen DomU.
>
> Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
>
>
> On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Thanks again for your reply and your suggestion.  I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works only
> > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c  and
> > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > I am still unable to find out why??
> >
> > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file when
> it
> > is already added to xen.dtsi?
> >
> > Best Regards,
> > Prem
> >
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Saturday, May 15, 2021 00:25
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > Hi Prem,
> >
> > Some older versions of Xil_DCacheFlushRange were implemented using
> > deprecated set/way instructions that are known to cause problems in
> > virtualized environments.
> >
> > More recent implementations use the "dc civac" intruction that shouldn't
> > have any issues.
> >
> > Can you double-check that you are calling the 64-bit version implemented
> > by CIVAC?
> >
> >https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L414
>
> >
> >
> >
> >
> > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano, 
> > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example. 
> > > Also yes, I have already run hello world as DomU guest, with UART1.
> > >
> > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> guest
> > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > function xil_dcacheflushrange(). Please find the c code attached
> > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c 
> > >
> > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > through
> > > in the guest config.(from pl.dtsi, which I manually created).
> > >
> > > Thanks and Best Regards,
> > > Prem
> > >
> > > Get Outlook for Android
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Friday 14 May 2021, 23:16
> > > To: Prem Harikrishnan
> > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > edgari@xilinx.com;
> > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > >
> > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > a baremetal application? Can you run it successfully, just printing
> > > "hello world", if you only assign the UART and nothing else?
> > >
> > > I am asking just to verify that you can run the baremetal VM
> > > successfully, aside from the PL device assignment.
> > >
> > >
> > > In regards to the PL device assignment, I'll let Brian answer on the
> > > Vivado configuration and SMID calculation. But I noticed that you only
> > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > looks like there are two related IRQs: 121 and 122.
> > >
> > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > follow: I cannot see any SATA controller assignment in your
> > > configuration.
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Dear Brian, Stefano and Julien,
> > > >
> > > > Thank you very much for your replies.
> > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > >
> > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i
> > tried
> > > it
> > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> master
> > > and
> > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually
> > tied
> > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I
> can
> > > > update you soon with an ILA trace on Vivado.
> > > >
> > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > xen.dtsi.
> > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this
> > SATA
> > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my
> xen
> > > > configuration.
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > Sent: Thursday, May 13, 2021 17:13
> > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>;
> > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > Prem,
> > > >
> > > > Sorry for the late email, I've been under the weather and have been away
> > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > calculated:
> > > >
> > > > 14-10: TBU bits
> > > > 9-6: Master ID bits
> > > > 5-0: AXI ID bits
> > > >
> > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > table which is table 16-4 on page 409.
> > > >
> > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > which is table 16-13 on page 429.
> > > >
> > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > > > are work arounds.  Here's a couple of examples (with example numbers):
> > > >
> > > > AXI master -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > >
> > > > AXI master -> SmartConnect -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > >
> > > > AXI master -> SmartConnect -> PS slave port 0
> > > >                            -> PS slave port 1
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > >
> > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > >
> > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > >                                 -> PS slave port 1
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > >
> > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > >
> > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > >                                                         -> PS slave port 1
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > >
> > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> > > > fairly simple but with some caveats.
> > > >
> > > > You can also read in the SMMU faults from Xen and use:
> > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > to decode them. That should tell you the SMMU IDs as well.
> > > >
> > > > Let me know if this isn't clear or you have any other questions.
> > > >
> > > > Brian
> > > >
> > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > Hello everyone,
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > AXI ID cannot manually be set.
> > > > > > >
> > > > > > >
> > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > >
> > > > > > >
> > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > image and use tftp to boot.
> > > > > > >
> > > > > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > section 3 and MPSOC SMMU
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > translation.
> > > > > > >
> > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > >
> > > > > > >
> > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > which i can be useful to my application
> > > > > > >
> > > > > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > >     assignments without SMMU
> > > > > > >   *
> > > > > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > >     Xen virtual machines
> > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > >
> > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > recommend to contact them directly.
> > > > > >
> > > > > > I have CCed Stefano who might be able to help you.
> > > > >
> > > > > Hi Julien,
> > > > >
> > > > > Many thanks for CC'ing me, I missed the email.
> > > > >
> > > > >
> > > > > Hi Prem,
> > > > >
> > > > > Let me CC a few people that might be able to help.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>

Hi Stefano,
Sorry to bother you again.
I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled my guest running with FreeRTOS and it generates a trap.
Petalinux and VITIS IDE Version is 2020.2.

Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1

root@pilbara:/etc/xen# xl create -c apu_freertos.cfg
Parsing config from apu_freertos.cfg
(XEN) Dom1 colors: [ 8 ]
(XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
(XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
(XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
(XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
(XEN) sched_null.c:344: 1 <-- d1v0
(XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
(XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
(XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
(XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
(XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1

The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.

Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN for the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
Thanks and Best Regards,
Prem

________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com>
Sent: Tuesday, May 18, 2021 20:09
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com <JOVITAC@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

Great to hear! Thank you for reporting back.


On Tue, 18 May 2021, Prem Harikrishnan wrote:
> Dear Stefano,
> Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed out.
> I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the DomU
> memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you pointed
> out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
>
> Thanks again i was able get my custom IP also working as an AXI master from PL.
>
> Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Tuesday, May 18, 2021 02:35
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> Your configuration looks correct as far as I can tell.
>
> You need dtdev because it triggers the SMMU configuration for your PL
> block. You don't need device_tree because your baremetal application is
> not going to read it, but it is not going to cause any issues either.
>
> When you say "the execution does not complete the loop at line 244", it
> means that you see the execution entering the loop and then never
> leaving? Like an infinite loop?
>
> If so, then I imagine the issue could be that the application is
> crashing when trying to access TxBufferPtr, which is pointing to
> TX_BUFFER_BASE, which is:
>
>
> #ifndef DDR_BASE_ADDR
> #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> DEFAULT SET TO 0x01000000
> #define MEM_BASE_ADDR 0x01000000
> #else
> #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> #endif
>
> #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
>
>
>
> Can you double check that TX_BUFFER_BASE is pointing to a correct
> address? Memory starts at 0x40000000 in a Xen DomU.
>
> Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
>
>
> On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works only
> > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > I am still unable to find out why??
> >
> > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file when
> it
> > is already added to xen.dtsi?
> >
> > Best Regards,
> > Prem
> >
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Saturday, May 15, 2021 00:25
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Hi Prem,
> >
> > Some older versions of Xil_DCacheFlushRange were implemented using
> > deprecated set/way instructions that are known to cause problems in
> > virtualized environments.
> >
> > More recent implementations use the "dc civac" intruction that shouldn't
> > have any issues.
> >
> > Can you double-check that you are calling the 64-bit version implemented
> > by CIVAC?
> >
> >https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L414
>
> >
> >
> >
> >
> > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > Also yes, I have already run hello world as DomU guest, with UART1.
> > >
> > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> guest
> > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > function xil_dcacheflushrange(). Please find the c code attached
> > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > >
> > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > through
> > > in the guest config.(from pl.dtsi, which I manually created).
> > >
> > > Thanks and Best Regards,
> > > Prem
> > >
> > > Get Outlook for Android
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Friday 14 May 2021, 23:16
> > > To: Prem Harikrishnan
> > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > edgari@xilinx.com;
> > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > >
> > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > a baremetal application? Can you run it successfully, just printing
> > > "hello world", if you only assign the UART and nothing else?
> > >
> > > I am asking just to verify that you can run the baremetal VM
> > > successfully, aside from the PL device assignment.
> > >
> > >
> > > In regards to the PL device assignment, I'll let Brian answer on the
> > > Vivado configuration and SMID calculation. But I noticed that you only
> > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > looks like there are two related IRQs: 121 and 122.
> > >
> > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > follow: I cannot see any SATA controller assignment in your
> > > configuration.
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Dear Brian, Stefano and Julien,
> > > >
> > > > Thank you very much for your replies.
> > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > >
> > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i
> > tried
> > > it
> > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> master
> > > and
> > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have manually
> > tied
> > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I
> can
> > > > update you soon with an ILA trace on Vivado.
> > > >
> > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > xen.dtsi.
> > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this
> > SATA
> > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my
> xen
> > > > configuration.
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > Sent: Thursday, May 13, 2021 17:13
> > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <sstabellini@kernel.org>;
> > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > Prem,
> > > >
> > > > Sorry for the late email, I've been under the weather and have been away
> > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > calculated:
> > > >
> > > > 14-10: TBU bits
> > > > 9-6: Master ID bits
> > > > 5-0: AXI ID bits
> > > >
> > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > table which is table 16-4 on page 409.
> > > >
> > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > which is table 16-13 on page 429.
> > > >
> > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > are work arounds. Here's a couple of examples (with example numbers):
> > > >
> > > > AXI master -> PS slave port 0
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > >
> > > > AXI master -> SmartConnect -> PS slave port 0
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > >
> > > > AXI master -> SmartConnect -> PS slave port 0
> > > > -> PS slave port 1
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > >
> > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > >
> > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > -> PS slave port 1
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > >
> > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > >
> > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > -> PS slave port 1
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > >
> > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > fairly simple but with some caveats.
> > > >
> > > > You can also read in the SMMU faults from Xen and use:
> > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > to decode them. That should tell you the SMMU IDs as well.
> > > >
> > > > Let me know if this isn't clear or you have any other questions.
> > > >
> > > > Brian
> > > >
> > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > Hello everyone,
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > AXI ID cannot manually be set.
> > > > > > >
> > > > > > >
> > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > >
> > > > > > >
> > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > image and use tftp to boot.
> > > > > > >
> > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > section 3 and MPSOC SMMU
> > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > translation.
> > > > > > >
> > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > >
> > > > > > >
> > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > which i can be useful to my application
> > > > > > >
> > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > assignments without SMMU
> > > > > > > *
> > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > Xen virtual machines
> > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > >
> > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > recommend to contact them directly.
> > > > > >
> > > > > > I have CCed Stefano who might be able to help you.
> > > > >
> > > > > Hi Julien,
> > > > >
> > > > > Many thanks for CC'ing me, I missed the email.
> > > > >
> > > > >
> > > > > Hi Prem,
> > > > >
> > > > > Let me CC a few people that might be able to help.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>

+Mubin


Hi Prem,

Did you make sure to build FreeRTOS as "hypervisor guest"? I think
FreeRTOS is built to run at EL3 by default so if that is not changed
there are errors when run on any hypervisor. It is possible to build
FreeRTOS to run on Xen but it requires a couple of manual steps:

- source settings64.sh from Vitis builds
- xsct
- set WORKSPACE .
- setws -switch $WORKSPACE
- app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
- bsp config stdin psu_uart_1
- bsp config stdout psu_uart_1
- bsp config hypervisor_guest true
- Go to application source path and modify the linker script start address to 0x40000000,
- app build -name freertos-hello-world
- Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"

The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
might be able to provide additional info.



On Thu, 27 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Sorry to bother you again.
> I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled my
> guest running with FreeRTOS and it generates a trap.
> Petalinux and VITIS IDE Version is 2020.2.
>
> Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
>
> root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> (XEN) Dom1 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> (XEN) sched_null.c:344: 1 <-- d1v0
> (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
>
> The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
>
> Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN for
> the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Tuesday, May 18, 2021 20:09
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com
> <JOVITAC@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> Great to hear! Thank you for reporting back.
>
>
> On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > Dear Stefano,
> > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> out.
> > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the DomU
> > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> pointed
> > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> >
> > Thanks again i was able get my custom IP also working as an AXI master from PL.
> >
> > Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Tuesday, May 18, 2021 02:35
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > Your configuration looks correct as far as I can tell.
> >
> > You need dtdev because it triggers the SMMU configuration for your PL
> > block. You don't need device_tree because your baremetal application is
> > not going to read it, but it is not going to cause any issues either.
> >
> > When you say "the execution does not complete the loop at line 244", it
> > means that you see the execution entering the loop and then never
> > leaving? Like an infinite loop?
> >
> > If so, then I imagine the issue could be that the application is
> > crashing when trying to access TxBufferPtr, which is pointing to
> > TX_BUFFER_BASE, which is:
> >
> >
> > #ifndef DDR_BASE_ADDR
> > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> >                  DEFAULT SET TO 0x01000000
> > #define MEM_BASE_ADDR           0x01000000
> > #else
> > #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> > #endif
> >
> > #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> >
> >
> >
> > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > address? Memory starts at 0x40000000 in a Xen DomU.
> >
> > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> >
> >
> > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Thanks again for your reply and your suggestion.  I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> only
> > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c  and
> > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > I am still unable to find out why??
> > >
> > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file when
> > it
> > > is already added to xen.dtsi?
> > >
> > > Best Regards,
> > > Prem
> > >
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Saturday, May 15, 2021 00:25
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> <fnuv@xilinx.com>;
> > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > Hi Prem,
> > >
> > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > deprecated set/way instructions that are known to cause problems in
> > > virtualized environments.
> > >
> > > More recent implementations use the "dc civac" intruction that shouldn't
> > > have any issues.
> > >
> > > Can you double-check that you are calling the 64-bit version implemented
> > > by CIVAC?
> > >
> >>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L41
> 4
> >
> > >
> > >
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano, 
> > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example. 
> > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > >
> > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > guest
> > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c 
> > > >
> > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > through
> > > > in the guest config.(from pl.dtsi, which I manually created).
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> > > > Get Outlook for Android
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Sent: Friday 14 May 2021, 23:16
> > > > To: Prem Harikrishnan
> > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > > edgari@xilinx.com;
> > > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > >
> > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > a baremetal application? Can you run it successfully, just printing
> > > > "hello world", if you only assign the UART and nothing else?
> > > >
> > > > I am asking just to verify that you can run the baremetal VM
> > > > successfully, aside from the PL device assignment.
> > > >
> > > >
> > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > looks like there are two related IRQs: 121 and 122.
> > > >
> > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > follow: I cannot see any SATA controller assignment in your
> > > > configuration.
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Dear Brian, Stefano and Julien,
> > > > >
> > > > > Thank you very much for your replies.
> > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > >
> > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i
> > > tried
> > > > it
> > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > master
> > > > and
> > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> manually
> > > tied
> > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I
> > can
> > > > > update you soon with an ILA trace on Vivado.
> > > > >
> > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > xen.dtsi.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this
> > > SATA
> > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my
> > xen
> > > > > configuration.
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini
> <sstabellini@kernel.org>;
> > > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > > Prem,
> > > > >
> > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > calculated:
> > > > >
> > > > > 14-10: TBU bits
> > > > > 9-6: Master ID bits
> > > > > 5-0: AXI ID bits
> > > > >
> > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > table which is table 16-4 on page 409.
> > > > >
> > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > which is table 16-13 on page 429.
> > > > >
> > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > > > > are work arounds.  Here's a couple of examples (with example numbers):
> > > > >
> > > > > AXI master -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > >                            -> PS slave port 1
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > >                                 -> PS slave port 1
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > >                                                         -> PS slave port 1
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > >
> > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> > > > > fairly simple but with some caveats.
> > > > >
> > > > > You can also read in the SMMU faults from Xen and use:
> > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > >
> > > > > Let me know if this isn't clear or you have any other questions.
> > > > >
> > > > > Brian
> > > > >
> > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > AXI ID cannot manually be set.
> > > > > > > >
> > > > > > > >
> > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > >
> > > > > > > >
> > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > image and use tftp to boot.
> > > > > > > >
> > > > > > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > translation.
> > > > > > > >
> > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > >
> > > > > > > >
> > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > which i can be useful to my application
> > > > > > > >
> > > > > > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > >     assignments without SMMU
> > > > > > > >   *
> > > > > > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > >     Xen virtual machines
> > > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > > >
> > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > recommend to contact them directly.
> > > > > > >
> > > > > > > I have CCed Stefano who might be able to help you.
> > > > > >
> > > > > > Hi Julien,
> > > > > >
> > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > >
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > Let me CC a few people that might be able to help.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Hi Stefano,
Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.

Parsing config from freertos.cfg
(XEN) Dom2 colors: [ 8 ]
(XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
(XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
(XEN) sched_null.c:344: 1 <-- d2v0
(XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c


I also see that according to this article https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
[https://forums.xilinx.com/t5/image/serverpage/image-id/52662iFAD1733707609204?v=v2]<https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030>
Partner: FreeRTOS as a Xen Virtual Machine - Community Forums<https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030>
Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel capable of task scheduling...
forums.xilinx.com
FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after that. Can you please confirm?

Thanks and Best Regards,
Prem
________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com>
Sent: Friday, May 28, 2021 02:51
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com <JOVITAC@xilinx.com>; MUBINUSM@xilinx.com <MUBINUSM@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

+Mubin


Hi Prem,

Did you make sure to build FreeRTOS as "hypervisor guest"? I think
FreeRTOS is built to run at EL3 by default so if that is not changed
there are errors when run on any hypervisor. It is possible to build
FreeRTOS to run on Xen but it requires a couple of manual steps:

- source settings64.sh from Vitis builds
- xsct
- set WORKSPACE .
- setws -switch $WORKSPACE
- app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
- bsp config stdin psu_uart_1
- bsp config stdout psu_uart_1
- bsp config hypervisor_guest true
- Go to application source path and modify the linker script start address to 0x40000000,
- app build -name freertos-hello-world
- Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"

The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
might be able to provide additional info.



On Thu, 27 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Sorry to bother you again.
> I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled my
> guest running with FreeRTOS and it generates a trap.
> Petalinux and VITIS IDE Version is 2020.2.
>
> Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
>
> root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> (XEN) Dom1 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> (XEN) sched_null.c:344: 1 <-- d1v0
> (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
>
> The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
>
> Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN for
> the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Tuesday, May 18, 2021 20:09
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com
> <JOVITAC@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> Great to hear! Thank you for reporting back.
>
>
> On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > Dear Stefano,
> > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> out.
> > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the DomU
> > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> pointed
> > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> >
> > Thanks again i was able get my custom IP also working as an AXI master from PL.
> >
> > Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Tuesday, May 18, 2021 02:35
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com <fnuv@xilinx.com>;
> > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Your configuration looks correct as far as I can tell.
> >
> > You need dtdev because it triggers the SMMU configuration for your PL
> > block. You don't need device_tree because your baremetal application is
> > not going to read it, but it is not going to cause any issues either.
> >
> > When you say "the execution does not complete the loop at line 244", it
> > means that you see the execution entering the loop and then never
> > leaving? Like an infinite loop?
> >
> > If so, then I imagine the issue could be that the application is
> > crashing when trying to access TxBufferPtr, which is pointing to
> > TX_BUFFER_BASE, which is:
> >
> >
> > #ifndef DDR_BASE_ADDR
> > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > DEFAULT SET TO 0x01000000
> > #define MEM_BASE_ADDR 0x01000000
> > #else
> > #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> > #endif
> >
> > #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
> >
> >
> >
> > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > address? Memory starts at 0x40000000 in a Xen DomU.
> >
> > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> >
> >
> > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> only
> > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > I am still unable to find out why??
> > >
> > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file when
> > it
> > > is already added to xen.dtsi?
> > >
> > > Best Regards,
> > > Prem
> > >
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Saturday, May 15, 2021 00:25
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> <fnuv@xilinx.com>;
> > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Hi Prem,
> > >
> > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > deprecated set/way instructions that are known to cause problems in
> > > virtualized environments.
> > >
> > > More recent implementations use the "dc civac" intruction that shouldn't
> > > have any issues.
> > >
> > > Can you double-check that you are calling the 64-bit version implemented
> > > by CIVAC?
> > >
> >>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L41
> 4
> >
> > >
> > >
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > >
> > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > guest
> > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > >
> > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > through
> > > > in the guest config.(from pl.dtsi, which I manually created).
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> > > > Get Outlook for Android
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Sent: Friday 14 May 2021, 23:16
> > > > To: Prem Harikrishnan
> > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > > edgari@xilinx.com;
> > > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > >
> > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > a baremetal application? Can you run it successfully, just printing
> > > > "hello world", if you only assign the UART and nothing else?
> > > >
> > > > I am asking just to verify that you can run the baremetal VM
> > > > successfully, aside from the PL device assignment.
> > > >
> > > >
> > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > looks like there are two related IRQs: 121 and 122.
> > > >
> > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > follow: I cannot see any SATA controller assignment in your
> > > > configuration.
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Dear Brian, Stefano and Julien,
> > > > >
> > > > > Thank you very much for your replies.
> > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > >
> > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i
> > > tried
> > > > it
> > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > master
> > > > and
> > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> manually
> > > tied
> > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I
> > can
> > > > > update you soon with an ILA trace on Vivado.
> > > > >
> > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > xen.dtsi.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this
> > > SATA
> > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my
> > xen
> > > > > configuration.
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini
> <sstabellini@kernel.org>;
> > > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > Prem,
> > > > >
> > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > calculated:
> > > > >
> > > > > 14-10: TBU bits
> > > > > 9-6: Master ID bits
> > > > > 5-0: AXI ID bits
> > > > >
> > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > table which is table 16-4 on page 409.
> > > > >
> > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > which is table 16-13 on page 429.
> > > > >
> > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > > are work arounds. Here's a couple of examples (with example numbers):
> > > > >
> > > > > AXI master -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > -> PS slave port 1
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > -> PS slave port 1
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > -> PS slave port 1
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > >
> > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > > fairly simple but with some caveats.
> > > > >
> > > > > You can also read in the SMMU faults from Xen and use:
> > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > >
> > > > > Let me know if this isn't clear or you have any other questions.
> > > > >
> > > > > Brian
> > > > >
> > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > AXI ID cannot manually be set.
> > > > > > > >
> > > > > > > >
> > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > >
> > > > > > > >
> > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > image and use tftp to boot.
> > > > > > > >
> > > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > translation.
> > > > > > > >
> > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > >
> > > > > > > >
> > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > which i can be useful to my application
> > > > > > > >
> > > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > assignments without SMMU
> > > > > > > > *
> > > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > Xen virtual machines
> > > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > > >
> > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > recommend to contact them directly.
> > > > > > >
> > > > > > > I have CCed Stefano who might be able to help you.
> > > > > >
> > > > > > Hi Julien,
> > > > > >
> > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > >
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > Let me CC a few people that might be able to help.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Hi Prem,

domU guest support in FreeRTOS port has been added since 2020.2 release. It supposed to work seamlessly if you are using code base from 2020.2 release + xsct build instructions specified by Stefano.

Thanks,
Mubin
From: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Sent: Friday, May 28, 2021 8:03 PM
To: Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org; Jovita Castelino <JOVITAC@xilinx.com>; Mubin Usman Sayyed <MUBINUSM@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

Hi Stefano,
Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.

Parsing config from freertos.cfg
(XEN) Dom2 colors: [ 8 ]
(XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
(XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
(XEN) sched_null.c:344: 1 <-- d2v0
(XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c


I also see that according to this article https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
[https://forums.xilinx.com/t5/image/serverpage/image-id/52662iFAD1733707609204?v=v2]<https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030>
Partner: FreeRTOS as a Xen Virtual Machine - Community Forums<https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030>
Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel capable of task scheduling...
forums.xilinx.com
FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after that. Can you please confirm?

Thanks and Best Regards,
Prem
________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
Sent: Friday, May 28, 2021 02:51
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

+Mubin


Hi Prem,

Did you make sure to build FreeRTOS as "hypervisor guest"? I think
FreeRTOS is built to run at EL3 by default so if that is not changed
there are errors when run on any hypervisor. It is possible to build
FreeRTOS to run on Xen but it requires a couple of manual steps:

- source settings64.sh from Vitis builds
- xsct
- set WORKSPACE .
- setws -switch $WORKSPACE
- app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
- bsp config stdin psu_uart_1
- bsp config stdout psu_uart_1
- bsp config hypervisor_guest true
- Go to application source path and modify the linker script start address to 0x40000000,
- app build -name freertos-hello-world
- Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"

The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
might be able to provide additional info.



On Thu, 27 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Sorry to bother you again.
> I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled my
> guest running with FreeRTOS and it generates a trap.
> Petalinux and VITIS IDE Version is 2020.2.
>
> Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
>
> root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> (XEN) Dom1 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> (XEN) sched_null.c:344: 1 <-- d1v0
> (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
>
> The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
>
> Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN for
> the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> Sent: Tuesday, May 18, 2021 20:09
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> Great to hear! Thank you for reporting back.
>
>
> On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > Dear Stefano,
> > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> out.
> > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the DomU
> > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> pointed
> > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> >
> > Thanks again i was able get my custom IP also working as an AXI master from PL.
> >
> > Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > Sent: Tuesday, May 18, 2021 02:35
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com> <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Your configuration looks correct as far as I can tell.
> >
> > You need dtdev because it triggers the SMMU configuration for your PL
> > block. You don't need device_tree because your baremetal application is
> > not going to read it, but it is not going to cause any issues either.
> >
> > When you say "the execution does not complete the loop at line 244", it
> > means that you see the execution entering the loop and then never
> > leaving? Like an infinite loop?
> >
> > If so, then I imagine the issue could be that the application is
> > crashing when trying to access TxBufferPtr, which is pointing to
> > TX_BUFFER_BASE, which is:
> >
> >
> > #ifndef DDR_BASE_ADDR
> > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > DEFAULT SET TO 0x01000000
> > #define MEM_BASE_ADDR 0x01000000
> > #else
> > #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> > #endif
> >
> > #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
> >
> >
> >
> > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > address? Memory starts at 0x40000000 in a Xen DomU.
> >
> > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> >
> >
> > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> only
> > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > I am still unable to find out why??
> > >
> > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file when
> > it
> > > is already added to xen.dtsi?
> > >
> > > Best Regards,
> > > Prem
> > >
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > Sent: Saturday, May 15, 2021 00:25
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Hi Prem,
> > >
> > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > deprecated set/way instructions that are known to cause problems in
> > > virtualized environments.
> > >
> > > More recent implementations use the "dc civac" intruction that shouldn't
> > > have any issues.
> > >
> > > Can you double-check that you are calling the 64-bit version implemented
> > > by CIVAC?
> > >
> >>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L41
> 4
> >
> > >
> > >
> > >
> > >
> > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > >
> > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > guest
> > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > >
> > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > through
> > > > in the guest config.(from pl.dtsi, which I manually created).
> > > >
> > > > Thanks and Best Regards,
> > > > Prem
> > > >
> > > > Get Outlook for Android
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > Sent: Friday 14 May 2021, 23:16
> > > > To: Prem Harikrishnan
> > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>;
> > > edgari@xilinx.com<mailto:edgari@xilinx.com>;
> > > > bwoods288@gmail.com<mailto:bwoods288@gmail.com>; Renato Costa Amorim; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > >
> > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > a baremetal application? Can you run it successfully, just printing
> > > > "hello world", if you only assign the UART and nothing else?
> > > >
> > > > I am asking just to verify that you can run the baremetal VM
> > > > successfully, aside from the PL device assignment.
> > > >
> > > >
> > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > looks like there are two related IRQs: 121 and 122.
> > > >
> > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > follow: I cannot see any SATA controller assignment in your
> > > > configuration.
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Dear Brian, Stefano and Julien,
> > > > >
> > > > > Thank you very much for your replies.
> > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > >
> > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously i
> > > tried
> > > > it
> > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > master
> > > > and
> > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> manually
> > > tied
> > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct? I
> > can
> > > > > update you soon with an ILA trace on Vivado.
> > > > >
> > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > xen.dtsi.
> > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using this
> > > SATA
> > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in my
> > xen
> > > > > configuration.
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>
> > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > Cc: Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>; Stefano Stabellini
> <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>;
> > > > > xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; brian.woods@xilinx.com<mailto:brian.woods@xilinx.com> <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > > > > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>; edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > Prem,
> > > > >
> > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > calculated:
> > > > >
> > > > > 14-10: TBU bits
> > > > > 9-6: Master ID bits
> > > > > 5-0: AXI ID bits
> > > > >
> > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > table which is table 16-4 on page 409.
> > > > >
> > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > which is table 16-13 on page 429.
> > > > >
> > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > > are work arounds. Here's a couple of examples (with example numbers):
> > > > >
> > > > > AXI master -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > >
> > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > -> PS slave port 1
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > >
> > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > -> PS slave port 1
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > >
> > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > -> PS slave port 1
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > >
> > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > > fairly simple but with some caveats.
> > > > >
> > > > > You can also read in the SMMU faults from Xen and use:
> > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > >
> > > > > Let me know if this isn't clear or you have any other questions.
> > > > >
> > > > > Brian
> > > > >
> > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > AXI ID cannot manually be set.
> > > > > > > >
> > > > > > > >
> > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > >
> > > > > > > >
> > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > image and use tftp to boot.
> > > > > > > >
> > > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > translation.
> > > > > > > >
> > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > >
> > > > > > > >
> > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > which i can be useful to my application
> > > > > > > >
> > > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > assignments without SMMU
> > > > > > > > *
> > > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > Xen virtual machines
> > > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > > >
> > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > recommend to contact them directly.
> > > > > > >
> > > > > > > I have CCed Stefano who might be able to help you.
> > > > > >
> > > > > > Hi Julien,
> > > > > >
> > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > >
> > > > > >
> > > > > > Hi Prem,
> > > > > >
> > > > > > Let me CC a few people that might be able to help.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

From the gpa address, it looks like FreeRTOS is trying to access the TTC
timer? If you look at device tree, the corresponding node is
timer@ff110000.

If FreeRTOS is configured to access the TTC timer, then you need to
assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
any chance?

You can do that by adding:

irqs = [ 68, 69, 70 ]
iomem = [ "0xff110,1" ]


On Fri, 28 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added
> printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.
>
> Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) sched_null.c:344: 1 <-- d2v0
> (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c
>
>
> I also see that according to this article
> https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> [52662iFAD1733707609204?v=v2]
> Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time
> operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel
> capable of task scheduling...
> forums.xilinx.com
> FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after
> that. Can you please confirm?
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Friday, May 28, 2021 02:51
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com
> <JOVITAC@xilinx.com>; MUBINUSM@xilinx.com <MUBINUSM@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> +Mubin
>
>
> Hi Prem,
>
> Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> FreeRTOS is built to run at EL3 by default so if that is not changed
> there are errors when run on any hypervisor. It is possible to build
> FreeRTOS to run on Xen but it requires a couple of manual steps:
>
> - source settings64.sh from Vitis builds
> - xsct
> - set WORKSPACE .
> - setws -switch $WORKSPACE
> - app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> - bsp config stdin psu_uart_1
> - bsp config stdout psu_uart_1
> - bsp config hypervisor_guest true
> - Go to application source path and modify the linker script start address to 0x40000000,
> - app build -name freertos-hello-world
> - Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0  <elf path> BOOT.BIN"
>
> The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> might be able to provide additional info.
>
>
>
> On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Sorry to bother you again.
> > I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled
> my
> > guest running with FreeRTOS and it generates a trap.
> > Petalinux and VITIS IDE Version is 2020.2.
> >
> > Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> > (XEN) Dom1 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > (XEN) sched_null.c:344: 1 <-- d1v0
> > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
> >
> > Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN
> for
> > the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Tuesday, May 18, 2021 20:09
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>;
> JOVITAC@xilinx.com
> > <JOVITAC@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > Great to hear! Thank you for reporting back.
> >
> >
> > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > Dear Stefano,
> > > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> > out.
> > > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the
> DomU
> > > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> > pointed
> > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> > >
> > > Thanks again i was able get my custom IP also working as an AXI master from PL.
> > >
> > > Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Tuesday, May 18, 2021 02:35
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> <fnuv@xilinx.com>;
> > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > Your configuration looks correct as far as I can tell.
> > >
> > > You need dtdev because it triggers the SMMU configuration for your PL
> > > block. You don't need device_tree because your baremetal application is
> > > not going to read it, but it is not going to cause any issues either.
> > >
> > > When you say "the execution does not complete the loop at line 244", it
> > > means that you see the execution entering the loop and then never
> > > leaving? Like an infinite loop?
> > >
> > > If so, then I imagine the issue could be that the application is
> > > crashing when trying to access TxBufferPtr, which is pointing to
> > > TX_BUFFER_BASE, which is:
> > >
> > >
> > > #ifndef DDR_BASE_ADDR
> > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > >                  DEFAULT SET TO 0x01000000
> > > #define MEM_BASE_ADDR           0x01000000
> > > #else
> > > #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> > > #endif
> > >
> > > #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> > >
> > >
> > >
> > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > address? Memory starts at 0x40000000 in a Xen DomU.
> > >
> > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > >
> > >
> > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks again for your reply and your suggestion.  I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> > only
> > > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c  and
> > > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > > I am still unable to find out why??
> > > >
> > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file
> when
> > > it
> > > > is already added to xen.dtsi?
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Sent: Saturday, May 15, 2021 00:25
> > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> > <fnuv@xilinx.com>;
> > > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > Hi Prem,
> > > >
> > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > deprecated set/way instructions that are known to cause problems in
> > > > virtualized environments.
> > > >
> > > > More recent implementations use the "dc civac" intruction that shouldn't
> > > > have any issues.
> > > >
> > > > Can you double-check that you are calling the 64-bit version implemented
> > > > by CIVAC?
> > > >
> >>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L4
> 1
> > 4
> > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano, 
> > > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example. 
> > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > >
> > > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > > guest
> > > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c 
> > > > >
> > > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > > through
> > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> > > > > Get Outlook for Android
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > Sent: Friday 14 May 2021, 23:16
> > > > > To: Prem Harikrishnan
> > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > > > edgari@xilinx.com;
> > > > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > >
> > > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > a baremetal application? Can you run it successfully, just printing
> > > > > "hello world", if you only assign the UART and nothing else?
> > > > >
> > > > > I am asking just to verify that you can run the baremetal VM
> > > > > successfully, aside from the PL device assignment.
> > > > >
> > > > >
> > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > > looks like there are two related IRQs: 121 and 122.
> > > > >
> > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > follow: I cannot see any SATA controller assignment in your
> > > > > configuration.
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Dear Brian, Stefano and Julien,
> > > > > >
> > > > > > Thank you very much for your replies.
> > > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > > >
> > > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously
> i
> > > > tried
> > > > > it
> > > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > > master
> > > > > and
> > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> > manually
> > > > tied
> > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct?
> I
> > > can
> > > > > > update you soon with an ILA trace on Vivado.
> > > > > >
> > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > > xen.dtsi.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using
> this
> > > > SATA
> > > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in
> my
> > > xen
> > > > > > configuration.
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini
> > <sstabellini@kernel.org>;
> > > > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > > > Prem,
> > > > > >
> > > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > > calculated:
> > > > > >
> > > > > > 14-10: TBU bits
> > > > > > 9-6: Master ID bits
> > > > > > 5-0: AXI ID bits
> > > > > >
> > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > > table which is table 16-4 on page 409.
> > > > > >
> > > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > > which is table 16-13 on page 429.
> > > > > >
> > > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > > > > > are work arounds.  Here's a couple of examples (with example numbers):
> > > > > >
> > > > > > AXI master -> PS slave port 0
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > >                            -> PS slave port 1
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > >                                 -> PS slave port 1
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > >                                                         -> PS slave port 1
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> > > > > > fairly simple but with some caveats.
> > > > > >
> > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > >
> > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > > image and use tftp to boot.
> > > > > > > > >
> > > > > > > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > > translation.
> > > > > > > > >
> > > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > > which i can be useful to my application
> > > > > > > > >
> > > > > > > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > >     assignments without SMMU
> > > > > > > > >   *
> > > > > > > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > >     Xen virtual machines
> > > > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > > > >
> > > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > > recommend to contact them directly.
> > > > > > > >
> > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > >
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > Let me CC a few people that might be able to help.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Hi Stefano and Mubin,
Thank you very much for your replies.
@Stefano Stabellini<mailto:stefano.stabellini@xilinx.com> Sorry to bother you again, and thank you for pointing this out, the TTC0 is by default enabled for FreeRTOS to generate ticks, although this says only for R5 core, i would have never figured this out. As soon as I added the irqs and iomem and gave a xen passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started working for other peripherals except for my DMA (FreeRTOS guest does not work with interrupt and DMA).

I can see my DMA is enabled because i have LEDs configured to blink for every interrupt I get from the PL. I don't see my interrupts being processed, but when I destroy the guest, I get this output periodically when an interrupt occurs.

(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000

From section 3.6 of the wiki https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
Xen and PL Masters - Xilinx Wiki - Confluence<https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of PL masters must be added to the Linux Dom0 device tree. Automated device tree generation does not generate the stream IDs for the PL masters so that the user must add them into the device tree manually.
xilinx-wiki.atlassian.net
The fault indicates an unidentified stream ID has been received by the SMMU as shown in the GFSR register. The GFSYNR1 register contains the stream ID that was unidentified. A stream ID of 0xE80 was unidentified in the above fault.

But I think i have already added that in my smmu and iommu in xen.dtsi because my baremetal application works well as expected. I have attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and freertos.cfg(not working).
What could I be missing here? Do I have to enable the ttc0 in the device tree?
Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 KHz mean when XEN boots up?
Thanks, I look forward for your reply.

Best Regards,
Prem


________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com>
Sent: Friday, May 28, 2021 20:43
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com <JOVITAC@xilinx.com>; MUBINUSM@xilinx.com <MUBINUSM@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

From the gpa address, it looks like FreeRTOS is trying to access the TTC
timer? If you look at device tree, the corresponding node is
timer@ff110000.

If FreeRTOS is configured to access the TTC timer, then you need to
assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
any chance?

You can do that by adding:

irqs = [ 68, 69, 70 ]
iomem = [ "0xff110,1" ]


On Fri, 28 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added
> printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.
>
> Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) sched_null.c:344: 1 <-- d2v0
> (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c
>
>
> I also see that according to this article
> https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> [52662iFAD1733707609204?v=v2]
> Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time
> operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel
> capable of task scheduling...
> forums.xilinx.com
> FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after
> that. Can you please confirm?
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Friday, May 28, 2021 02:51
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com
> <JOVITAC@xilinx.com>; MUBINUSM@xilinx.com <MUBINUSM@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> +Mubin
>
>
> Hi Prem,
>
> Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> FreeRTOS is built to run at EL3 by default so if that is not changed
> there are errors when run on any hypervisor. It is possible to build
> FreeRTOS to run on Xen but it requires a couple of manual steps:
>
> - source settings64.sh from Vitis builds
> - xsct
> - set WORKSPACE .
> - setws -switch $WORKSPACE
> - app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> - bsp config stdin psu_uart_1
> - bsp config stdout psu_uart_1
> - bsp config hypervisor_guest true
> - Go to application source path and modify the linker script start address to 0x40000000,
> - app build -name freertos-hello-world
> - Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"
>
> The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> might be able to provide additional info.
>
>
>
> On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Sorry to bother you again.
> > I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled
> my
> > guest running with FreeRTOS and it generates a trap.
> > Petalinux and VITIS IDE Version is 2020.2.
> >
> > Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> > (XEN) Dom1 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > (XEN) sched_null.c:344: 1 <-- d1v0
> > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
> >
> > Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN
> for
> > the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Tuesday, May 18, 2021 20:09
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>;
> JOVITAC@xilinx.com
> > <JOVITAC@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Great to hear! Thank you for reporting back.
> >
> >
> > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > Dear Stefano,
> > > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> > out.
> > > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the
> DomU
> > > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> > pointed
> > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> > >
> > > Thanks again i was able get my custom IP also working as an AXI master from PL.
> > >
> > > Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Tuesday, May 18, 2021 02:35
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> <fnuv@xilinx.com>;
> > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Your configuration looks correct as far as I can tell.
> > >
> > > You need dtdev because it triggers the SMMU configuration for your PL
> > > block. You don't need device_tree because your baremetal application is
> > > not going to read it, but it is not going to cause any issues either.
> > >
> > > When you say "the execution does not complete the loop at line 244", it
> > > means that you see the execution entering the loop and then never
> > > leaving? Like an infinite loop?
> > >
> > > If so, then I imagine the issue could be that the application is
> > > crashing when trying to access TxBufferPtr, which is pointing to
> > > TX_BUFFER_BASE, which is:
> > >
> > >
> > > #ifndef DDR_BASE_ADDR
> > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > DEFAULT SET TO 0x01000000
> > > #define MEM_BASE_ADDR 0x01000000
> > > #else
> > > #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> > > #endif
> > >
> > > #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
> > >
> > >
> > >
> > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > address? Memory starts at 0x40000000 in a Xen DomU.
> > >
> > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > >
> > >
> > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> > only
> > > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > > I am still unable to find out why??
> > > >
> > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file
> when
> > > it
> > > > is already added to xen.dtsi?
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Sent: Saturday, May 15, 2021 00:25
> > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> > <fnuv@xilinx.com>;
> > > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > Hi Prem,
> > > >
> > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > deprecated set/way instructions that are known to cause problems in
> > > > virtualized environments.
> > > >
> > > > More recent implementations use the "dc civac" intruction that shouldn't
> > > > have any issues.
> > > >
> > > > Can you double-check that you are calling the 64-bit version implemented
> > > > by CIVAC?
> > > >
> >>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L4
> 1
> > 4
> > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano,
> > > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > >
> > > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > > guest
> > > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > >
> > > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > > through
> > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> > > > > Get Outlook for Android
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > Sent: Friday 14 May 2021, 23:16
> > > > > To: Prem Harikrishnan
> > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > > > edgari@xilinx.com;
> > > > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > >
> > > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > a baremetal application? Can you run it successfully, just printing
> > > > > "hello world", if you only assign the UART and nothing else?
> > > > >
> > > > > I am asking just to verify that you can run the baremetal VM
> > > > > successfully, aside from the PL device assignment.
> > > > >
> > > > >
> > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > > looks like there are two related IRQs: 121 and 122.
> > > > >
> > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > follow: I cannot see any SATA controller assignment in your
> > > > > configuration.
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Dear Brian, Stefano and Julien,
> > > > > >
> > > > > > Thank you very much for your replies.
> > > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > > >
> > > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously
> i
> > > > tried
> > > > > it
> > > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > > master
> > > > > and
> > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> > manually
> > > > tied
> > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct?
> I
> > > can
> > > > > > update you soon with an ILA trace on Vivado.
> > > > > >
> > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > > xen.dtsi.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using
> this
> > > > SATA
> > > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in
> my
> > > xen
> > > > > > configuration.
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini
> > <sstabellini@kernel.org>;
> > > > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>; fnuv@xilinx.com
> > > > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > > Prem,
> > > > > >
> > > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > > calculated:
> > > > > >
> > > > > > 14-10: TBU bits
> > > > > > 9-6: Master ID bits
> > > > > > 5-0: AXI ID bits
> > > > > >
> > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > > table which is table 16-4 on page 409.
> > > > > >
> > > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > > which is table 16-13 on page 429.
> > > > > >
> > > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > > > are work arounds. Here's a couple of examples (with example numbers):
> > > > > >
> > > > > > AXI master -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > > > fairly simple but with some caveats.
> > > > > >
> > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > >
> > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > > image and use tftp to boot.
> > > > > > > > >
> > > > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > > translation.
> > > > > > > > >
> > > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > > which i can be useful to my application
> > > > > > > > >
> > > > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > > assignments without SMMU
> > > > > > > > > *
> > > > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > > Xen virtual machines
> > > > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > > > >
> > > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > > recommend to contact them directly.
> > > > > > > >
> > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > >
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > Let me CC a few people that might be able to help.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

In case of FreeRTOS, you need to use APIs from FreeRTOS port for interrupt handling (https://github.com/Xilinx/embeddedsw/blob/master/ThirdParty/bsp/freertos10_xilinx/examples/freertos_intr_example.c#L192 and line 194). Please make sure that you are taking care of that in DMA interrupt example .

Thanks,
Mubin

From: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Sent: Tuesday, June 1, 2021 2:48 AM
To: Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org; Jovita Castelino <JOVITAC@xilinx.com>; Mubin Usman Sayyed <MUBINUSM@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit


Hi Stefano and Mubin,
Thank you very much for your replies.
@Stefano Stabellini<mailto:stefano.stabellini@xilinx.com> Sorry to bother you again, and thank you for pointing this out, the TTC0 is by default enabled for FreeRTOS to generate ticks, although this says only for R5 core, i would have never figured this out. As soon as I added the irqs and iomem and gave a xen passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started working for other peripherals except for my DMA (FreeRTOS guest does not work with interrupt and DMA).

I can see my DMA is enabled because i have LEDs configured to blink for every interrupt I get from the PL. I don't see my interrupts being processed, but when I destroy the guest, I get this output periodically when an interrupt occurs.

(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000

From section 3.6 of the wiki https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
Xen and PL Masters - Xilinx Wiki - Confluence<https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of PL masters must be added to the Linux Dom0 device tree. Automated device tree generation does not generate the stream IDs for the PL masters so that the user must add them into the device tree manually.
xilinx-wiki.atlassian.net
The fault indicates an unidentified stream ID has been received by the SMMU as shown in the GFSR register. The GFSYNR1 register contains the stream ID that was unidentified. A stream ID of 0xE80 was unidentified in the above fault.

But I think i have already added that in my smmu and iommu in xen.dtsi because my baremetal application works well as expected. I have attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and freertos.cfg(not working).
What could I be missing here? Do I have to enable the ttc0 in the device tree?
Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 KHz mean when XEN boots up?
Thanks, I look forward for your reply.

Best Regards,
Prem


________________________________
From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
Sent: Friday, May 28, 2021 20:43
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

From the gpa address, it looks like FreeRTOS is trying to access the TTC
timer? If you look at device tree, the corresponding node is
timer@ff110000.

If FreeRTOS is configured to access the TTC timer, then you need to
assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
any chance?

You can do that by adding:

irqs = [ 68, 69, 70 ]
iomem = [ "0xff110,1" ]


On Fri, 28 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added
> printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.
>
> Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) sched_null.c:344: 1 <-- d2v0
> (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c
>
>
> I also see that according to this article
> https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> [52662iFAD1733707609204?v=v2]
> Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time
> operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel
> capable of task scheduling...
> forums.xilinx.com
> FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after
> that. Can you please confirm?
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> Sent: Friday, May 28, 2021 02:51
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> +Mubin
>
>
> Hi Prem,
>
> Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> FreeRTOS is built to run at EL3 by default so if that is not changed
> there are errors when run on any hypervisor. It is possible to build
> FreeRTOS to run on Xen but it requires a couple of manual steps:
>
> - source settings64.sh from Vitis builds
> - xsct
> - set WORKSPACE .
> - setws -switch $WORKSPACE
> - app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> - bsp config stdin psu_uart_1
> - bsp config stdout psu_uart_1
> - bsp config hypervisor_guest true
> - Go to application source path and modify the linker script start address to 0x40000000,
> - app build -name freertos-hello-world
> - Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"
>
> The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> might be able to provide additional info.
>
>
>
> On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Sorry to bother you again.
> > I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled
> my
> > guest running with FreeRTOS and it generates a trap.
> > Petalinux and VITIS IDE Version is 2020.2.
> >
> > Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> > (XEN) Dom1 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > (XEN) sched_null.c:344: 1 <-- d1v0
> > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
> >
> > Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN
> for
> > the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > Sent: Tuesday, May 18, 2021 20:09
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>;
> JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Great to hear! Thank you for reporting back.
> >
> >
> > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > Dear Stefano,
> > > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> > out.
> > > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the
> DomU
> > > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> > pointed
> > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> > >
> > > Thanks again i was able get my custom IP also working as an AXI master from PL.
> > >
> > > Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > Sent: Tuesday, May 18, 2021 02:35
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Your configuration looks correct as far as I can tell.
> > >
> > > You need dtdev because it triggers the SMMU configuration for your PL
> > > block. You don't need device_tree because your baremetal application is
> > > not going to read it, but it is not going to cause any issues either.
> > >
> > > When you say "the execution does not complete the loop at line 244", it
> > > means that you see the execution entering the loop and then never
> > > leaving? Like an infinite loop?
> > >
> > > If so, then I imagine the issue could be that the application is
> > > crashing when trying to access TxBufferPtr, which is pointing to
> > > TX_BUFFER_BASE, which is:
> > >
> > >
> > > #ifndef DDR_BASE_ADDR
> > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > DEFAULT SET TO 0x01000000
> > > #define MEM_BASE_ADDR 0x01000000
> > > #else
> > > #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> > > #endif
> > >
> > > #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
> > >
> > >
> > >
> > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > address? Memory starts at 0x40000000 in a Xen DomU.
> > >
> > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > >
> > >
> > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> > only
> > > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > > I am still unable to find out why??
> > > >
> > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file
> when
> > > it
> > > > is already added to xen.dtsi?
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > Sent: Saturday, May 15, 2021 00:25
> > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > Hi Prem,
> > > >
> > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > deprecated set/way instructions that are known to cause problems in
> > > > virtualized environments.
> > > >
> > > > More recent implementations use the "dc civac" intruction that shouldn't
> > > > have any issues.
> > > >
> > > > Can you double-check that you are calling the 64-bit version implemented
> > > > by CIVAC?
> > > >
> >>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L4
> 1
> > 4
> > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano,
> > > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > >
> > > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > > guest
> > > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > >
> > > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > > through
> > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> > > > > Get Outlook for Android
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > Sent: Friday 14 May 2021, 23:16
> > > > > To: Prem Harikrishnan
> > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>;
> > > > edgari@xilinx.com<mailto:edgari@xilinx.com>;
> > > > > bwoods288@gmail.com<mailto:bwoods288@gmail.com>; Renato Costa Amorim; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > >
> > > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > a baremetal application? Can you run it successfully, just printing
> > > > > "hello world", if you only assign the UART and nothing else?
> > > > >
> > > > > I am asking just to verify that you can run the baremetal VM
> > > > > successfully, aside from the PL device assignment.
> > > > >
> > > > >
> > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > > looks like there are two related IRQs: 121 and 122.
> > > > >
> > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > follow: I cannot see any SATA controller assignment in your
> > > > > configuration.
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Dear Brian, Stefano and Julien,
> > > > > >
> > > > > > Thank you very much for your replies.
> > > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > > >
> > > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously
> i
> > > > tried
> > > > > it
> > > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > > master
> > > > > and
> > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> > manually
> > > > tied
> > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct?
> I
> > > can
> > > > > > update you soon with an ILA trace on Vivado.
> > > > > >
> > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > > xen.dtsi.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using
> this
> > > > SATA
> > > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in
> my
> > > xen
> > > > > > configuration.
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>
> > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > > Cc: Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>; Stefano Stabellini
> > <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>;
> > > > > > xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; brian.woods@xilinx.com<mailto:brian.woods@xilinx.com> <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > > > > > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>; edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > > Prem,
> > > > > >
> > > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > > calculated:
> > > > > >
> > > > > > 14-10: TBU bits
> > > > > > 9-6: Master ID bits
> > > > > > 5-0: AXI ID bits
> > > > > >
> > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > > table which is table 16-4 on page 409.
> > > > > >
> > > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > > which is table 16-13 on page 429.
> > > > > >
> > > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > > > are work arounds. Here's a couple of examples (with example numbers):
> > > > > >
> > > > > > AXI master -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > > > fairly simple but with some caveats.
> > > > > >
> > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > >
> > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > > image and use tftp to boot.
> > > > > > > > >
> > > > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > > translation.
> > > > > > > > >
> > > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > > which i can be useful to my application
> > > > > > > > >
> > > > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > > assignments without SMMU
> > > > > > > > > *
> > > > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > > Xen virtual machines
> > > > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > > > >
> > > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > > recommend to contact them directly.
> > > > > > > >
> > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > >
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > Let me CC a few people that might be able to help.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Hi Stefano and Mubin,

Thank you very much for helping me setup up FreeRTOS and baremetal guests, everything works as expected and I also get expected performance in both baremetal and hypervisor guests. Thank you again for your prompt support during the past few weeks.

Best Regrards,
Prem
________________________________
From: Mubin Usman Sayyed <MUBINUSM@xilinx.com>
Sent: Tuesday, June 1, 2021 05:38
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; Jovita Castelino <JOVITAC@xilinx.com>
Subject: RE: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit


In case of FreeRTOS, you need to use APIs from FreeRTOS port for interrupt handling (https://github.com/Xilinx/embeddedsw/blob/master/ThirdParty/bsp/freertos10_xilinx/examples/freertos_intr_example.c#L192 and line 194). Please make sure that you are taking care of that in DMA interrupt example .



Thanks,

Mubin



From: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Sent: Tuesday, June 1, 2021 2:48 AM
To: Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org; Jovita Castelino <JOVITAC@xilinx.com>; Mubin Usman Sayyed <MUBINUSM@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit





Hi Stefano and Mubin,

Thank you very much for your replies.

@Stefano Stabellini<mailto:stefano.stabellini@xilinx.com> Sorry to bother you again, and thank you for pointing this out, the TTC0 is by default enabled for FreeRTOS to generate ticks, although this says only for R5 core, i would have never figured this out. As soon as I added the irqs and iomem and gave a xen passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started working for other peripherals except for my DMA (FreeRTOS guest does not work with interrupt and DMA).



I can see my DMA is enabled because i have LEDs configured to blink for every interrupt I get from the PL. I don't see my interrupts being processed, but when I destroy the guest, I get this output periodically when an interrupt occurs.



(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000

(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000

(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000



From section 3.6 of the wiki https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters

Xen and PL Masters - Xilinx Wiki - Confluence<https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>

3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of PL masters must be added to the Linux Dom0 device tree. Automated device tree generation does not generate the stream IDs for the PL masters so that the user must add them into the device tree manually.

xilinx-wiki.atlassian.net

The fault indicates an unidentified stream ID has been received by the SMMU as shown in the GFSR register. The GFSYNR1 register contains the stream ID that was unidentified. A stream ID of 0xE80 was unidentified in the above fault.



But I think i have already added that in my smmu and iommu in xen.dtsi because my baremetal application works well as expected. I have attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and freertos.cfg(not working).

What could I be missing here? Do I have to enable the ttc0 in the device tree?

Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 KHz mean when XEN boots up?

Thanks, I look forward for your reply.



Best Regards,

Prem





________________________________

From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
Sent: Friday, May 28, 2021 20:43
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit



From the gpa address, it looks like FreeRTOS is trying to access the TTC
timer? If you look at device tree, the corresponding node is
timer@ff110000.

If FreeRTOS is configured to access the TTC timer, then you need to
assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
any chance?

You can do that by adding:

irqs = [ 68, 69, 70 ]
iomem = [ "0xff110,1" ]


On Fri, 28 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added
> printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.
>
> Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) sched_null.c:344: 1 <-- d2v0
> (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c
>
>
> I also see that according to this article
> https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> [52662iFAD1733707609204?v=v2]
> Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time
> operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel
> capable of task scheduling...
> forums.xilinx.com
> FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after
> that. Can you please confirm?
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> Sent: Friday, May 28, 2021 02:51
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> +Mubin
>
>
> Hi Prem,
>
> Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> FreeRTOS is built to run at EL3 by default so if that is not changed
> there are errors when run on any hypervisor. It is possible to build
> FreeRTOS to run on Xen but it requires a couple of manual steps:
>
> - source settings64.sh from Vitis builds
> - xsct
> - set WORKSPACE .
> - setws -switch $WORKSPACE
> - app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> - bsp config stdin psu_uart_1
> - bsp config stdout psu_uart_1
> - bsp config hypervisor_guest true
> - Go to application source path and modify the linker script start address to 0x40000000,
> - app build -name freertos-hello-world
> - Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"
>
> The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> might be able to provide additional info.
>
>
>
> On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Sorry to bother you again.
> > I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled
> my
> > guest running with FreeRTOS and it generates a trap.
> > Petalinux and VITIS IDE Version is 2020.2.
> >
> > Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> > (XEN) Dom1 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > (XEN) sched_null.c:344: 1 <-- d1v0
> > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
> >
> > Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN
> for
> > the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > Sent: Tuesday, May 18, 2021 20:09
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>;
> JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Great to hear! Thank you for reporting back.
> >
> >
> > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > Dear Stefano,
> > > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> > out.
> > > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the
> DomU
> > > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> > pointed
> > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> > >
> > > Thanks again i was able get my custom IP also working as an AXI master from PL.
> > >
> > > Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > Sent: Tuesday, May 18, 2021 02:35
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Your configuration looks correct as far as I can tell.
> > >
> > > You need dtdev because it triggers the SMMU configuration for your PL
> > > block. You don't need device_tree because your baremetal application is
> > > not going to read it, but it is not going to cause any issues either.
> > >
> > > When you say "the execution does not complete the loop at line 244", it
> > > means that you see the execution entering the loop and then never
> > > leaving? Like an infinite loop?
> > >
> > > If so, then I imagine the issue could be that the application is
> > > crashing when trying to access TxBufferPtr, which is pointing to
> > > TX_BUFFER_BASE, which is:
> > >
> > >
> > > #ifndef DDR_BASE_ADDR
> > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > DEFAULT SET TO 0x01000000
> > > #define MEM_BASE_ADDR 0x01000000
> > > #else
> > > #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> > > #endif
> > >
> > > #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
> > >
> > >
> > >
> > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > address? Memory starts at 0x40000000 in a Xen DomU.
> > >
> > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > >
> > >
> > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> > only
> > > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > > I am still unable to find out why??
> > > >
> > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file
> when
> > > it
> > > > is already added to xen.dtsi?
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > Sent: Saturday, May 15, 2021 00:25
> > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > Hi Prem,
> > > >
> > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > deprecated set/way instructions that are known to cause problems in
> > > > virtualized environments.
> > > >
> > > > More recent implementations use the "dc civac" intruction that shouldn't
> > > > have any issues.
> > > >
> > > > Can you double-check that you are calling the 64-bit version implemented
> > > > by CIVAC?
> > > >
> >>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L4
> 1
> > 4
> > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano,
> > > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > >
> > > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > > guest
> > > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > >
> > > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > > through
> > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> > > > > Get Outlook for Android
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > Sent: Friday 14 May 2021, 23:16
> > > > > To: Prem Harikrishnan
> > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>;
> > > > edgari@xilinx.com<mailto:edgari@xilinx.com>;
> > > > > bwoods288@gmail.com<mailto:bwoods288@gmail.com>; Renato Costa Amorim; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > >
> > > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > a baremetal application? Can you run it successfully, just printing
> > > > > "hello world", if you only assign the UART and nothing else?
> > > > >
> > > > > I am asking just to verify that you can run the baremetal VM
> > > > > successfully, aside from the PL device assignment.
> > > > >
> > > > >
> > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > > looks like there are two related IRQs: 121 and 122.
> > > > >
> > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > follow: I cannot see any SATA controller assignment in your
> > > > > configuration.
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Dear Brian, Stefano and Julien,
> > > > > >
> > > > > > Thank you very much for your replies.
> > > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > > >
> > > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously
> i
> > > > tried
> > > > > it
> > > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > > master
> > > > > and
> > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> > manually
> > > > tied
> > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct?
> I
> > > can
> > > > > > update you soon with an ILA trace on Vivado.
> > > > > >
> > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > > xen.dtsi.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using
> this
> > > > SATA
> > > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in
> my
> > > xen
> > > > > > configuration.
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>
> > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > > Cc: Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>; Stefano Stabellini
> > <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>;
> > > > > > xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; brian.woods@xilinx.com<mailto:brian.woods@xilinx.com> <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > > > > > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>; edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > > Prem,
> > > > > >
> > > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > > calculated:
> > > > > >
> > > > > > 14-10: TBU bits
> > > > > > 9-6: Master ID bits
> > > > > > 5-0: AXI ID bits
> > > > > >
> > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > > table which is table 16-4 on page 409.
> > > > > >
> > > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > > which is table 16-13 on page 429.
> > > > > >
> > > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > > > are work arounds. Here's a couple of examples (with example numbers):
> > > > > >
> > > > > > AXI master -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > > > fairly simple but with some caveats.
> > > > > >
> > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > >
> > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > > image and use tftp to boot.
> > > > > > > > >
> > > > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > > translation.
> > > > > > > > >
> > > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > > which i can be useful to my application
> > > > > > > > >
> > > > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > > assignments without SMMU
> > > > > > > > > *
> > > > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > > Xen virtual machines
> > > > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > > > >
> > > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > > recommend to contact them directly.
> > > > > > > >
> > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > >
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > Let me CC a few people that might be able to help.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Hi Stefano,

Is it possible to boot XEN from QSPI? Is there an image builder script already available to generate boot.scr? Please let me know.
Thanks and Best Regards,
Prem

________________________________
From: Xen-users <xen-users-bounces@lists.xenproject.org> on behalf of Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Sent: Tuesday, June 1, 2021 18:07
To: Mubin Usman Sayyed <MUBINUSM@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; Jovita Castelino <JOVITAC@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit

Hi Stefano and Mubin,

Thank you very much for helping me setup up FreeRTOS and baremetal guests, everything works as expected and I also get expected performance in both baremetal and hypervisor guests. Thank you again for your prompt support during the past few weeks.

Best Regrards,
Prem
________________________________
From: Mubin Usman Sayyed <MUBINUSM@xilinx.com>
Sent: Tuesday, June 1, 2021 05:38
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; Jovita Castelino <JOVITAC@xilinx.com>
Subject: RE: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit


In case of FreeRTOS, you need to use APIs from FreeRTOS port for interrupt handling (https://github.com/Xilinx/embeddedsw/blob/master/ThirdParty/bsp/freertos10_xilinx/examples/freertos_intr_example.c#L192 and line 194). Please make sure that you are taking care of that in DMA interrupt example .



Thanks,

Mubin



From: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
Sent: Tuesday, June 1, 2021 2:48 AM
To: Stefano Stabellini <stefanos@xilinx.com>
Cc: xen-users@lists.xenproject.org; Jovita Castelino <JOVITAC@xilinx.com>; Mubin Usman Sayyed <MUBINUSM@xilinx.com>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit





Hi Stefano and Mubin,

Thank you very much for your replies.

@Stefano Stabellini<mailto:stefano.stabellini@xilinx.com> Sorry to bother you again, and thank you for pointing this out, the TTC0 is by default enabled for FreeRTOS to generate ticks, although this says only for R5 core, i would have never figured this out. As soon as I added the irqs and iomem and gave a xen passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started working for other peripherals except for my DMA (FreeRTOS guest does not work with interrupt and DMA).



I can see my DMA is enabled because i have LEDs configured to blink for every interrupt I get from the PL. I don't see my interrupts being processed, but when I destroy the guest, I get this output periodically when an interrupt occurs.



(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000

(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000

(XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
(XEN) smmu: /smmu@fd800000: GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000



From section 3.6 of the wiki https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters

Xen and PL Masters - Xilinx Wiki - Confluence<https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>

3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of PL masters must be added to the Linux Dom0 device tree. Automated device tree generation does not generate the stream IDs for the PL masters so that the user must add them into the device tree manually.

xilinx-wiki.atlassian.net

The fault indicates an unidentified stream ID has been received by the SMMU as shown in the GFSR register. The GFSYNR1 register contains the stream ID that was unidentified. A stream ID of 0xE80 was unidentified in the above fault.



But I think i have already added that in my smmu and iommu in xen.dtsi because my baremetal application works well as expected. I have attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and freertos.cfg(not working).

What could I be missing here? Do I have to enable the ttc0 in the device tree?

Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 KHz mean when XEN boots up?

Thanks, I look forward for your reply.



Best Regards,

Prem





________________________________

From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
Sent: Friday, May 28, 2021 20:43
To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit



From the gpa address, it looks like FreeRTOS is trying to access the TTC
timer? If you look at device tree, the corresponding node is
timer@ff110000.

If FreeRTOS is configured to access the TTC timer, then you need to
assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
any chance?

You can do that by adding:

irqs = [ 68, 69, 70 ]
iomem = [ "0xff110,1" ]


On Fri, 28 May 2021, Prem Harikrishnan wrote:
> Hi Stefano,
> Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added
> printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.
>
> Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
> (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> (XEN) sched_null.c:344: 1 <-- d2v0
> (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c
>
>
> I also see that according to this article
> https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> [52662iFAD1733707609204?v=v2]
> Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading real-time
> operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel
> capable of task scheduling...
> forums.xilinx.com
> FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after
> that. Can you please confirm?
>
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> Sent: Friday, May 28, 2021 02:51
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>; MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com> <MUBINUSM@xilinx.com<mailto:MUBINUSM@xilinx.com>>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> +Mubin
>
>
> Hi Prem,
>
> Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> FreeRTOS is built to run at EL3 by default so if that is not changed
> there are errors when run on any hypervisor. It is possible to build
> FreeRTOS to run on Xen but it requires a couple of manual steps:
>
> - source settings64.sh from Vitis builds
> - xsct
> - set WORKSPACE .
> - setws -switch $WORKSPACE
> - app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> - bsp config stdin psu_uart_1
> - bsp config stdout psu_uart_1
> - bsp config hypervisor_guest true
> - Go to application source path and modify the linker script start address to 0x40000000,
> - app build -name freertos-hello-world
> - Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0 <elf path> BOOT.BIN"
>
> The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> might be able to provide additional info.
>
>
>
> On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Sorry to bother you again.
> > I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now compiled
> my
> > guest running with FreeRTOS and it generates a trap.
> > Petalinux and VITIS IDE Version is 2020.2.
> >
> > Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context fault:
> > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> > (XEN) Dom1 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > (XEN) sched_null.c:344: 1 <-- d1v0
> > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> >
> > The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
> >
> > Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN
> for
> > the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > Sent: Tuesday, May 18, 2021 20:09
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>;
> JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > Great to hear! Thank you for reporting back.
> >
> >
> > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > Dear Stefano,
> > > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> > out.
> > > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the
> DomU
> > > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> > pointed
> > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> > >
> > > Thanks again i was able get my custom IP also working as an AXI master from PL.
> > >
> > > Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > Sent: Tuesday, May 18, 2021 02:35
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > Your configuration looks correct as far as I can tell.
> > >
> > > You need dtdev because it triggers the SMMU configuration for your PL
> > > block. You don't need device_tree because your baremetal application is
> > > not going to read it, but it is not going to cause any issues either.
> > >
> > > When you say "the execution does not complete the loop at line 244", it
> > > means that you see the execution entering the loop and then never
> > > leaving? Like an infinite loop?
> > >
> > > If so, then I imagine the issue could be that the application is
> > > crashing when trying to access TxBufferPtr, which is pointing to
> > > TX_BUFFER_BASE, which is:
> > >
> > >
> > > #ifndef DDR_BASE_ADDR
> > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > DEFAULT SET TO 0x01000000
> > > #define MEM_BASE_ADDR 0x01000000
> > > #else
> > > #define MEM_BASE_ADDR (DDR_BASE_ADDR + 0x1000000)
> > > #endif
> > >
> > > #define TX_BUFFER_BASE (MEM_BASE_ADDR + 0x00100000)
> > >
> > >
> > >
> > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > address? Memory starts at 0x40000000 in a Xen DomU.
> > >
> > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > >
> > >
> > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > Hi Stefano,
> > > > Thanks again for your reply and your suggestion. I did try to replace the Xil_DCacheFlushRange with Xil_DCacheInvalidateRange(works
> > only
> > > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c and
> > > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > > I am still unable to find out why??
> > > >
> > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file
> when
> > > it
> > > > is already added to xen.dtsi?
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > Sent: Saturday, May 15, 2021 00:25
> > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>; Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Stefano
> > > > Stabellini <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>;
> > > > edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>; Renato Costa Amorim <renato.amorim@evoleotech.com<mailto:renato.amorim@evoleotech.com>>;
> > > > JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com> <JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > Hi Prem,
> > > >
> > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > deprecated set/way instructions that are known to cause problems in
> > > > virtualized environments.
> > > >
> > > > More recent implementations use the "dc civac" intruction that shouldn't
> > > > have any issues.
> > > >
> > > > Can you double-check that you are calling the 64-bit version implemented
> > > > by CIVAC?
> > > >
> >>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L4
> 1
> > 4
> > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano,
> > > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example.
> > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > >
> > > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in the
> > > guest
> > > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before the
> > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c
> > > > >
> > > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to pass
> > > > through
> > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > >
> > > > > Thanks and Best Regards,
> > > > > Prem
> > > > >
> > > > > Get Outlook for Android
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > Sent: Friday 14 May 2021, 23:16
> > > > > To: Prem Harikrishnan
> > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>;
> > > > edgari@xilinx.com<mailto:edgari@xilinx.com>;
> > > > > bwoods288@gmail.com<mailto:bwoods288@gmail.com>; Renato Costa Amorim; JOVITAC@xilinx.com<mailto:JOVITAC@xilinx.com>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > >
> > > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > a baremetal application? Can you run it successfully, just printing
> > > > > "hello world", if you only assign the UART and nothing else?
> > > > >
> > > > > I am asking just to verify that you can run the baremetal VM
> > > > > successfully, aside from the PL device assignment.
> > > > >
> > > > >
> > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > > looks like there are two related IRQs: 121 and 122.
> > > > >
> > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > follow: I cannot see any SATA controller assignment in your
> > > > > configuration.
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Dear Brian, Stefano and Julien,
> > > > > >
> > > > > > Thank you very much for your replies.
> > > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > > >
> > > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling mode(previously
> i
> > > > tried
> > > > > it
> > > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only 1
> > > master
> > > > > and
> > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> > manually
> > > > tied
> > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is correct?
> I
> > > can
> > > > > > update you soon with an ILA trace on Vivado.
> > > > > >
> > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually adding
> > > > > xen.dtsi.
> > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using
> this
> > > > SATA
> > > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in
> my
> > > xen
> > > > > > configuration.
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Brian Woods <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>
> > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com<mailto:stefano.stabellini@xilinx.com>>
> > > > > > Cc: Julien Grall <julien@xen.org<mailto:julien@xen.org>>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com<mailto:prem.harikrishnan@evoleotech.com>>; Stefano Stabellini
> > <sstabellini@kernel.org<mailto:sstabellini@kernel.org>>;
> > > > > > xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org> <xen-users@lists.xenproject.org<mailto:xen-users@lists.xenproject.org>>; brian.woods@xilinx.com<mailto:brian.woods@xilinx.com> <brian.woods@xilinx.com<mailto:brian.woods@xilinx.com>>; fnuv@xilinx.com<mailto:fnuv@xilinx.com>
> > > > > > <fnuv@xilinx.com<mailto:fnuv@xilinx.com>>; edgari@xilinx.com<mailto:edgari@xilinx.com> <edgari@xilinx.com<mailto:edgari@xilinx.com>>; bwoods288@gmail.com<mailto:bwoods288@gmail.com> <bwoods288@gmail.com<mailto:bwoods288@gmail.com>>
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > > Prem,
> > > > > >
> > > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > > from my computer. In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > > calculated:
> > > > > >
> > > > > > 14-10: TBU bits
> > > > > > 9-6: Master ID bits
> > > > > > 5-0: AXI ID bits
> > > > > >
> > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > > table which is table 16-4 on page 409.
> > > > > >
> > > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > > which is table 16-13 on page 429.
> > > > > >
> > > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on. There
> > > > > > are work arounds. Here's a couple of examples (with example numbers):
> > > > > >
> > > > > > AXI master -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > >
> > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > -> PS slave port 1
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > > TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > > TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > > >
> > > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > > replaced with "AXI Interconnect" to my knowledge. So the AXI ID bits are
> > > > > > fairly simple but with some caveats.
> > > > > >
> > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > >
> > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > >
> > > > > > Brian
> > > > > >
> > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > > image and use tftp to boot.
> > > > > > > > >
> > > > > > > > > From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > > translation.
> > > > > > > > >
> > > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > > which i can be useful to my application
> > > > > > > > >
> > > > > > > > > * 1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > > assignments without SMMU
> > > > > > > > > *
> > > > > > > > > Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > > Xen virtual machines
> > > > > > > > > * Support for PL Device Passthrough to DomU in Xen
> > > > > > > >
> > > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > > recommend to contact them directly.
> > > > > > > >
> > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > >
> > > > > > > Hi Julien,
> > > > > > >
> > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > >
> > > > > > >
> > > > > > > Hi Prem,
> > > > > > >
> > > > > > > Let me CC a few people that might be able to help.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

We don't have a recipe ready. It is not something we test. However I
think it should be possible.

The first step would be try to generate a BOOT.BIN for QSPI booting and
write it to QSPI. The goal is to get to the u-boot prompt, with
everything booted from QSPI. Then the next step is to add Xen, Linux,
the DTB and Linux ramdisk. (Of course if you are OK with loading Xen and
the other binaries from somewhere else like the SD card then it is going
to work as is.)

You should be able to write each binary to QSPI as separate partitions.

At that point, you are only missing a boot.scr that can load them and
boot. ImageBuilder supports custom u-boot load commands by setting the
LOAD_CMD variable. In this case you'd need to set LOAD_CMD to something
like "sf read", but given that the other command arguments are not
filenames, ImageBuilder won't be able to handle it anyway.

But you can take an ImageBuilder-generated boot.source, substitute the
"tftpb" commands with the appropriate "sf read" commands and call
mkimage to get a boot.scr.

Now you should be able to run boot.scr from the u-boot prompt and load
everything from QSPI.

This is some relevant info:

https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842465/U-Boot+QSPI+Driver


I know it is a bit complex but I think it should work OK.



On Tue, 15 Jun 2021, Prem Harikrishnan wrote:
> Hi Stefano,
>
> Is it possible to boot XEN from QSPI? Is there an image builder script already available to generate boot.scr? Please let me know.
> Thanks and Best Regards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Xen-users <xen-users-bounces@lists.xenproject.org> on behalf of Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Sent: Tuesday, June 1, 2021 18:07
> To: Mubin Usman Sayyed <MUBINUSM@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>
> Cc: xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; Jovita Castelino <JOVITAC@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> Hi Stefano and Mubin,
>
> Thank you very much for helping me setup up FreeRTOS and baremetal guests, everything works as expected and I also get expected performance
> in both baremetal and hypervisor guests. Thank you again for your prompt support during the past few weeks.
>
> Best Regrards,
> Prem
>
> ___________________________________________________________________________________________________________________________________________
> From: Mubin Usman Sayyed <MUBINUSM@xilinx.com>
> Sent: Tuesday, June 1, 2021 05:38
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini <stefanos@xilinx.com>
> Cc: xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; Jovita Castelino <JOVITAC@xilinx.com>
> Subject: RE: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
>
> In case of FreeRTOS,  you need to use APIs from FreeRTOS port for interrupt handling
>  (https://github.com/Xilinx/embeddedsw/blob/master/ThirdParty/bsp/freertos10_xilinx/examples/freertos_intr_example.c#L192 and line 194).
> Please make sure that you are taking care of that in DMA interrupt example .
>
>  
>
> Thanks,
>
> Mubin
>
>  
>
> From: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Sent: Tuesday, June 1, 2021 2:48 AM
> To: Stefano Stabellini <stefanos@xilinx.com>
> Cc: xen-users@lists.xenproject.org; Jovita Castelino <JOVITAC@xilinx.com>; Mubin Usman Sayyed <MUBINUSM@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
>
>  
>
>  
>
> Hi Stefano and Mubin,
>
> Thank you very much for your replies.
>
> @Stefano Stabellini Sorry to bother you again, and thank you for pointing this out, the TTC0 is by default enabled for FreeRTOS to generate
> ticks, although this says only for R5 core, i would have never figured this out. As soon as I added the irqs and iomem and gave a xen
> passthrough in the xen.dtsi as you have suggested, FreeRTOS guest started working for other peripherals except for my DMA (FreeRTOS guest
> does not work with interrupt and DMA).
>
>  
>
> I can see my DMA is enabled because i have LEDs configured to blink for every interrupt I get from the PL. I don't see my interrupts being
> processed, but when I destroy the guest, I get this output periodically when an interrupt occurs.
>
>  
>
> (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
>
> (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
>
> (XEN) smmu: /smmu@fd800000: Unexpected global fault, this could be serious
> (XEN) smmu: /smmu@fd800000:     GFSR 0x80000002, GFSYNR0 0x00000002, GFSYNR1 0x00000e80, GFSYNR2 0x00000000
>
>  
>
> From section 3.6 of the wiki https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters
>
> Xen and PL Masters - Xilinx Wiki - Confluence
>
> 3.4 Stream IDs in the Device Tree. For Xen based systems the stream IDs of PL masters must be added to the Linux Dom0 device tree.
> Automated device tree generation does not generate the stream IDs for the PL masters so that the user must add them into the device tree
> manually.
>
> xilinx-wiki.atlassian.net
>
> The fault indicates an unidentified stream ID has been received by the SMMU as shown in the GFSR register. The GFSYNR1 register contains
> the stream ID that was unidentified. A stream ID of 0xE80 was unidentified in the above fault.
>
>  
>
> But I think i have already added that in my smmu and iommu in xen.dtsi because my baremetal application works well as expected. I have
> attached my xen.dtsi, system.dtsi, baremetal.cfg(working) and freertos.cfg(not working).
>
> What could I be missing here? Do I have to enable the ttc0 in the device tree?
>
> Also what does (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 99990 KHz mean when XEN boots up?
>
> Thanks, I look forward for your reply.
>
>  
>
> Best Regards,
>
> Prem
>
>  
>
>  
>
>
> ________________________________________________________________________________________________________________________________________
>
>
> From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> Sent: Friday, May 28, 2021 20:43
> To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; JOVITAC@xilinx.com
> <JOVITAC@xilinx.com>; MUBINUSM@xilinx.com <MUBINUSM@xilinx.com>
> Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
>
>  
>
> From the gpa address, it looks like FreeRTOS is trying to access the TTC
> timer? If you look at device tree, the corresponding node is
> timer@ff110000.
>
> If FreeRTOS is configured to access the TTC timer, then you need to
> assign it to it. Did you assign the TTC timer to the FreeRTOS domU by
> any chance?
>
> You can do that by adding:
>
> irqs = [ 68, 69, 70 ]
> iomem = [ "0xff110,1" ]
>
>
> On Fri, 28 May 2021, Prem Harikrishnan wrote:
> > Hi Stefano,
> > Thanks for your reply. I am pretty sure I followed the steps you mentioned below to run the FreeRTOS hello world example. I already added
> > printfs and i can see XEN cannot start the scheduler and i get the following out put on XEN.
> >
> > Parsing config from freertos.cfg (XEN) Dom2 colors: [ 8 ]
> > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=2 memflags=0xc0 (0 of 64)
> > (XEN) memory_map:add: dom2 gfn=ff010 mfn=ff010 nr=1 cache=0
> > (XEN) sched_null.c:344: 1 <-- d2v0
> > (XEN) traps.c:1973:d2v0 HSR=0x93820007 pc=0x00000040012b08 gva=0xff11000c gpa=0x000000ff11000c
> >
> >
> > I also see that according to this article
> > https://forums.xilinx.com/t5/Adaptable-Advantage-Blog/Partner-FreeRTOS-as-a-Xen-Virtual-Machine/ba-p/947030
> > [52662iFAD1733707609204?v=v2]
> > Partner: FreeRTOS as a Xen Virtual Machine - Community Forums
> > Contributed by Jeff Kubascik, Embedded Systems Engineer at DornerWorks What is FreeRTOS? The FreeRTOS kernel is a market leading
> real-time
> > operating system for embedded systems. As embedded applications grow in complexity, there is a growing need for a lightweight kernel
> > capable of task scheduling...
> > forums.xilinx.com
> > FreeRTOS does not run out of the box for XEN, although the article is 2 years old, I am not sure if there are any updates on this after
> > that. Can you please confirm?
> >
> > Thanks and Best Regards,
> > Prem
> >
> >__________________________________________________________________________________________________________________________________________
> _
> > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > Sent: Friday, May 28, 2021 02:51
> > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>;
> JOVITAC@xilinx.com
> > <JOVITAC@xilinx.com>; MUBINUSM@xilinx.com <MUBINUSM@xilinx.com>
> > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > +Mubin
> >
> >
> > Hi Prem,
> >
> > Did you make sure to build FreeRTOS as "hypervisor guest"? I think
> > FreeRTOS is built to run at EL3 by default so if that is not changed
> > there are errors when run on any hypervisor. It is possible to build
> > FreeRTOS to run on Xen but it requires a couple of manual steps:
> >
> > - source settings64.sh from Vitis builds
> > - xsct
> > - set WORKSPACE .
> > - setws -switch $WORKSPACE
> > - app create -name freertos-hello-world -template {FreeRTOS Hello World} -proc psu_cortexa53_0 -hw {xsa path} -os freertos10_xilinx
> > - bsp config stdin psu_uart_1
> > - bsp config stdout psu_uart_1
> > - bsp config hypervisor_guest true
> > - Go to application source path and modify the linker script start address to 0x40000000,
> > - app build -name freertos-hello-world
> > - Run command to generate BIN file from executable "aarch64-linux-gnu-objcopy -O binary --gap-fill 0  <elf path> BOOT.BIN"
> >
> > The resulting BOOT.BIN can be started a Xen DomU guest. Mubin, CC'ed,
> > might be able to provide additional info.
> >
> >
> >
> > On Thu, 27 May 2021, Prem Harikrishnan wrote:
> > > Hi Stefano,
> > > Sorry to bother you again.
> > > I am currently trying to run my custom IP which sends periodic dma interrupts to the PS with cache coloring enabled. I have now
> compiled
> > my
> > > guest running with FreeRTOS and it generates a trap.
> > > Petalinux and VITIS IDE Version is 2020.2.
> > >
> > > Please find the log attached. After the trap occurs, for every interrupt XEN prints (XEN) smmu: /smmu@fd800000: Unhandled context
> fault:
> > > fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > >
> > > root@pilbara:/etc/xen# xl create -c apu_freertos.cfg Parsing config from apu_freertos.cfg
> > > (XEN) Dom1 colors: [ 8 ]
> > > (XEN) memory.c:255:d0v0 Could not allocate order=9 extent: id=1 memflags=0xc0 (0 of 64)
> > > (XEN) memory_map:add: dom1 gfn=ff010 mfn=ff010 nr=1 cache=0
> > > (XEN) memory_map:add: dom1 gfn=a0010 mfn=a0010 nr=1 cache=0
> > > (XEN) smmu: /smmu@fd800000: d1: p2maddr 0x000000087f554000
> > > (XEN) sched_null.c:344: 1 <-- d1v0
> > > (XEN) traps.c:1973:d1v0 HSR=0x93820007 pc=0x00000040013188 gva=0xff11000c gpa=0x000000ff11000c
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > > (XEN) smmu: /smmu@fd800000: Unhandled context fault: fsr=0x402, iova=0x41000000, fsynr=0x12, cb=1
> > >
> > > The same application works as expected without FreeRTOS, i.e as a baremetal guest. I look forward for your comment on this.
> > >
> > > Although i see performance degradation, normally i can measure delta time for each interrupt configured to 1ms (without XEN), with XEN
> > for
> > > the same measurement shows 16 ms for each interval (but the performance is not a priority for now).
> > > Thanks and Best Regards,
> > > Prem
> > >
> >>_________________________________________________________________________________________________________________________________________
> _
> > _
> > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > Sent: Tuesday, May 18, 2021 20:09
> > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>;
> > JOVITAC@xilinx.com
> > > <JOVITAC@xilinx.com>
> > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > Great to hear! Thank you for reporting back.
> > >
> > >
> > > On Tue, 18 May 2021, Prem Harikrishnan wrote:
> > > > Dear Stefano,
> > > > Thank you very much for your guidance and support. I finally fixed the issue. I made an error in base memory address like you pointed
> > > out.
> > > > I actually modified the base address correctly to 0x40000000 but the default C code had an additional offset of 0x1100000 , and the
> > DomU
> > > > memory allocated was only 8MB. I either had to change the offset or allocate more memory to the DomU guest which i missed until you
> > > pointed
> > > > out. Additionally i had to set to AxPROT bits to 0x2 in Vivado and the SMID required is 0xE80.
> > > >
> > > > Thanks again i was able get my custom IP also working as an AXI master from PL.
> > > >
> > > > Best Regards,
> > > > Prem
> > > >
> >>>________________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > Sent: Tuesday, May 18, 2021 02:35
> > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>; Stefano
> > > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> > <fnuv@xilinx.com>;
> > > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim <renato.amorim@evoleotech.com>;
> > > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > Your configuration looks correct as far as I can tell.
> > > >
> > > > You need dtdev because it triggers the SMMU configuration for your PL
> > > > block. You don't need device_tree because your baremetal application is
> > > > not going to read it, but it is not going to cause any issues either.
> > > >
> > > > When you say "the execution does not complete the loop at line 244", it
> > > > means that you see the execution entering the loop and then never
> > > > leaving? Like an infinite loop?
> > > >
> > > > If so, then I imagine the issue could be that the application is
> > > > crashing when trying to access TxBufferPtr, which is pointing to
> > > > TX_BUFFER_BASE, which is:
> > > >
> > > >
> > > > #ifndef DDR_BASE_ADDR
> > > > #warning CHECK FOR THE VALID DDR ADDRESS IN XPARAMETERS.H, \
> > > >                  DEFAULT SET TO 0x01000000
> > > > #define MEM_BASE_ADDR           0x01000000
> > > > #else
> > > > #define MEM_BASE_ADDR           (DDR_BASE_ADDR + 0x1000000)
> > > > #endif
> > > >
> > > > #define TX_BUFFER_BASE          (MEM_BASE_ADDR + 0x00100000)
> > > >
> > > >
> > > >
> > > > Can you double check that TX_BUFFER_BASE is pointing to a correct
> > > > address? Memory starts at 0x40000000 in a Xen DomU.
> > > >
> > > > Maybe MEM_BASE_ADDR/DDR_BASE_ADDR is not set correctly?
> > > >
> > > >
> > > > On Mon, 17 May 2021, Prem Harikrishnan wrote:
> > > > > Hi Stefano,
> > > > > Thanks again for your reply and your suggestion.  I did try to replace the Xil_DCacheFlushRange with
> Xil_DCacheInvalidateRange(works
> > > only
> > > > > in EL3), I added printfs to debug, i can see that the execution does not complete the loop at line 244
> > > > > https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c  and
> > > > > Xil_DCacheFlushRange is not reached. Please find attached my xen bootlog.txt.
> > > > > I am still unable to find out why??
> > > > >
> > > > > Can you please atleast verify that my xen.dtsi, and pl.dtb is correct? and do i need dtdev and device_tree in my guest config file
> > when
> > > > it
> > > > > is already added to xen.dtsi?
> > > > >
> > > > > Best Regards,
> > > > > Prem
> > > > >
> > > > >
> >>>>_______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > Sent: Saturday, May 15, 2021 00:25
> > > > > To: Prem Harikrishnan <prem.harikrishnan@evoleotech.com>
> > > > > Cc: Stefano Stabellini <stefano.stabellini@xilinx.com>; Brian Woods <brian.woods@xilinx.com>; Julien Grall <julien@xen.org>;
> Stefano
> > > > > Stabellini <sstabellini@kernel.org>; xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; fnuv@xilinx.com
> > > <fnuv@xilinx.com>;
> > > > > edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>; Renato Costa Amorim
> <renato.amorim@evoleotech.com>;
> > > > > JOVITAC@xilinx.com <JOVITAC@xilinx.com>
> > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > > Hi Prem,
> > > > >
> > > > > Some older versions of Xil_DCacheFlushRange were implemented using
> > > > > deprecated set/way instructions that are known to cause problems in
> > > > > virtualized environments.
> > > > >
> > > > > More recent implementations use the "dc civac" intruction that shouldn't
> > > > > have any issues.
> > > > >
> > > > > Can you double-check that you are calling the 64-bit version implemented
> > > > > by CIVAC?
> > > > >
> >>>>https://github.com/Xilinx/embeddedsw/blob/875dcc7a4dca47ccb18eda0182f3577e598f8917/lib/bsp/standalone/src/arm/ARMv8/64bit/xil_cache.c#L
> 4
> > 1
> > > 4
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > Hi Stefano, 
> > > > > > Thanks for your reply. Yes the axidma application is a baremetal application from the xilinx vitis ide example. 
> > > > > > Also yes, I have already run hello world as DomU guest, with UART1.
> > > > > >
> > > > > > I am using the simple mode using polling and not interrupts(they are disabled in the c code), which is Why I didn't include in
> the
> > > > guest
> > > > > > config file. This works well when I compile it normally (as EL3). When compiled as hypervisor guest, the execution stops before
> the
> > > > > > function xil_dcacheflushrange(). Please find the c code attached
> > > > > > in https://github.com/Xilinx/embeddedsw/blob/master/XilinxProcessorIPLib/drivers/axidma/examples/xaxidma_example_simple_poll.c 
> > > > > >
> > > > > > I found the SATA example similar to my use case, also using DMA and SMMU. I just used it as a reference to create my pl.dtb to
> pass
> > > > > through
> > > > > > in the guest config.(from pl.dtsi, which I manually created).
> > > > > >
> > > > > > Thanks and Best Regards,
> > > > > > Prem
> > > > > >
> > > > > > Get Outlook for Android
> > > > > >
> >>>>>______________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > From: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > > Sent: Friday 14 May 2021, 23:16
> > > > > > To: Prem Harikrishnan
> > > > > > Cc: Brian Woods; Stefano Stabellini; Julien Grall; Stefano Stabellini; xen-users@lists.xenproject.org; fnuv@xilinx.com;
> > > > > edgari@xilinx.com;
> > > > > > bwoods288@gmail.com; Renato Costa Amorim; JOVITAC@xilinx.com
> > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit
> > > > > >
> > > > > > I take that the guest kernel "/bin/xaxidma_example_simple_poll_1.bin" is
> > > > > > a baremetal application? Can you run it successfully, just printing
> > > > > > "hello world", if you only assign the UART and nothing else?
> > > > > >
> > > > > > I am asking just to verify that you can run the baremetal VM
> > > > > > successfully, aside from the PL device assignment.
> > > > > >
> > > > > >
> > > > > > In regards to the PL device assignment, I'll let Brian answer on the
> > > > > > Vivado configuration and SMID calculation. But I noticed that you only
> > > > > > assigned IRQ 54 to the DomU which is the one for UART1. You haven't
> > > > > > assigned any IRQs for axi_dma_0. Looking at pl_from_petalinux.dtsi, it
> > > > > > looks like there are two related IRQs: 121 and 122.
> > > > > >
> > > > > > Finally, why did you mention SATA passthrough in your reply? I don't
> > > > > > follow: I cannot see any SATA controller assignment in your
> > > > > > configuration.
> > > > > >
> > > > > >
> > > > > > On Fri, 14 May 2021, Prem Harikrishnan wrote:
> > > > > > > Dear Brian, Stefano and Julien,
> > > > > > >
> > > > > > > Thank you very much for your replies.
> > > > > > > Hi Brian and Stefano, Can you please help me with the following questions?
> > > > > > >
> > > > > > > @Brian Woods Please find attached my vivado configuration, when i added my AXI DMA now running on simple polling
> mode(previously
> > i
> > > > > tried
> > > > > > it
> > > > > > > with Scatter Gather mode), Run Automation automatically generated, 1 AXI Interconnect and 1 smart connect. Since there is only
> 1
> > > > master
> > > > > > and
> > > > > > > I use the S_AXI _HP0_FPD, I finally get a value of 0xE80 for the stream IDs.
> > > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters According to this link section 6.5 I have
> > > manually
> > > > > tied
> > > > > > > of AxPROT bits to value of 0x2 to create non secure transactions. Can you please confirm that my Vivado configuration is
> correct?
> > I
> > > > can
> > > > > > > update you soon with an ILA trace on Vivado.
> > > > > > >
> > > > > > > @Stefano Stabellini Please find attached my pl.dtsi generated from petalinux and system.dtsi also generated after manually
> adding
> > > > > > xen.dtsi.
> > > > > > > https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/384663561/Building+Xen+Hypervisor+with+Petalinux+2020.1+and+2020.2 Using
> > this
> > > > > SATA
> > > > > > > passthrough, i made my own pl.dtsi for passthrough used in dma.cfg. I could really use your help to identify what is missing in
> > my
> > > > xen
> > > > > > > configuration.
> > > > > > >
> > > > > > > Thanks and Best Regards,
> > > > > > > Prem
> > > > > > >
> >>>>>>_____________________________________________________________________________________________________________________________________
> _
> > _
> > > _
> > > > _
> > > > > _
> > > > > > _
> > > > > > > From: Brian Woods <brian.woods@xilinx.com>
> > > > > > > Sent: Thursday, May 13, 2021 17:13
> > > > > > > To: Stefano Stabellini <stefano.stabellini@xilinx.com>
> > > > > > > Cc: Julien Grall <julien@xen.org>; Prem Harikrishnan <prem.harikrishnan@evoleotech.com>; Stefano Stabellini
> > > <sstabellini@kernel.org>;
> > > > > > > xen-users@lists.xenproject.org <xen-users@lists.xenproject.org>; brian.woods@xilinx.com <brian.woods@xilinx.com>;
> fnuv@xilinx.com
> > > > > > > <fnuv@xilinx.com>; edgari@xilinx.com <edgari@xilinx.com>; bwoods288@gmail.com <bwoods288@gmail.com>
> > > > > > > Subject: Re: Help: Information required for AXI DMA passthrough for PL masters in ZCU102 dev kit  
> > > > > > > Prem,
> > > > > > >
> > > > > > > Sorry for the late email, I've been under the weather and have been away
> > > > > > > from my computer.  In the Zynq SOCs this is how the SMMU IDs in PL are
> > > > > > > calculated:
> > > > > > >
> > > > > > > 14-10: TBU bits
> > > > > > > 9-6: Master ID bits
> > > > > > > 5-0: AXI ID bits
> > > > > > >
> > > > > > > https://www.xilinx.com/support/documentation/user_guides/ug1085-zynq-ultrascale-trm.pdf
> > > > > > > To look up the TBU value, see UG1085 and look for the "System Masters"
> > > > > > > table which is table 16-4 on page 409.
> > > > > > >
> > > > > > > For the master ID, see UG1085 and look for the "Master IDs List" table
> > > > > > > which is table 16-13 on page 429.
> > > > > > >
> > > > > > > The AXI ID bits is a bit trickery since it's going to be out in PL.
> > > > > > > See: https://www.xilinx.com/support/answers/69447.html
> > > > > > > With SmartConnect all the AXI IDs are 0, they aren't passed on.  There
> > > > > > > are work arounds.  Here's a couple of examples (with example numbers):
> > > > > > >
> > > > > > > AXI master -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001
> > > > > > >
> > > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > > >
> > > > > > > AXI master -> SmartConnect -> PS slave port 0
> > > > > > >                            -> PS slave port 1
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000
> > > > > > >
> > > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > > >
> > > > > > > Two AXI masters -> SmartConnect -> PS slave port 0
> > > > > > >                                 -> PS slave port 1
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000000 (both share)
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000000 (both share)
> > > > > > >
> > > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > > >
> > > > > > > Two AXI masters -> sideband -> SmartConnect -> sideband -> PS slave port 0
> > > > > > >                                                         -> PS slave port 1
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000001 (master #1)
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000001 (master #1)
> > > > > > >         TBU bits: 00001, Master ID bits: 0000, AXI ID bits: 0000002 (master #2)
> > > > > > >         TBU bits: 00001, Master ID bits: 0001, AXI ID bits: 0000002 (master #2)
> > > > > > >
> > > > > > > The "sideband -> SmartConenct -> sideband" can also be functionally
> > > > > > > replaced with "AXI Interconnect" to my knowledge.  So the AXI ID bits are
> > > > > > > fairly simple but with some caveats.
> > > > > > >
> > > > > > > You can also read in the SMMU faults from Xen and use:
> > > > > > > https://static.docs.arm.com/ihi0062/dc/IHI0062D_c_system_mmu_architecture_specification.pdf
> > > > > > > to decode them. That should tell you the SMMU IDs as well.
> > > > > > >
> > > > > > > Let me know if this isn't clear or you have any other questions.
> > > > > > >
> > > > > > > Brian
> > > > > > >
> > > > > > > On Mon, May 10, 2021 at 06:49:07PM -0700, Stefano Stabellini wrote:
> > > > > > > > On Mon, 10 May 2021, Julien Grall wrote:
> > > > > > > > > On 26/04/2021 15:41, Prem Harikrishnan wrote:
> > > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > Hi Prem,
> > > > > > > > >
> > > > > > > > > > I am using the latest version of the xilinx mpsoc zcu102 development board
> > > > > > > > > > and i am trying to create a barematal application that uses the AXI DMA on
> > > > > > > > > > the programmable logic, using the scatter gather example from the xilinx
> > > > > > > > > > repository. This works perfectly well for the baremetal application. The
> > > > > > > > > > diagram below represents the information flow. The AXI DMA and the AXI data
> > > > > > > > > > FIFO are in the Programable logic (FPGA). This AXI DMA is out of the box IP
> > > > > > > > > > from Vivado and uses a single channel (Does not use AxUSER) which means the
> > > > > > > > > > AXI ID cannot manually be set.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Now i want to make this as a baremetal hypervisor guest, in future this will
> > > > > > > > > > be replaced by a custom IP with cache colouring enabled. I want to pass
> > > > > > > > > > through the AXI DMA in the PL. (Not to be used by Dom0)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I followed this tutorial Baremetal + DomU
> > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842536/XEN+EL1+Baremetal+DomU>
> > > > > > > > > > guest changed the base address to 0x4000000,(1GB allocated to Dom-0) to
> > > > > > > > > > create my EL1 binary. I used Petalinux 2020.2 to generate my linux, xen
> > > > > > > > > > image and use tftp to boot.
> > > > > > > > > >
> > > > > > > > > >  From what i understand from the xilinx tutorials Xen+and+PL+Masters
> > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842066/Xen+and+PL+Masters>
> > > > > > > > > > section 3 and MPSOC SMMU
> > > > > > > > > > <https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841981/Zynq+UltraScale+MPSOC+SMMU>
> > > > > > > > > > section 4, stream ids are required by the SMMU to get the correct address
> > > > > > > > > > translation.
> > > > > > > > > >
> > > > > > > > > > I am still unable to figure out how to get the iommu property for my AXI DMA
> > > > > > > > > > and how to set the dtdev config for my hypervisor guest.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I can also see some new updates with the XEN 2020.1 and 2020.2 releases
> > > > > > > > > > which i can be useful to my application
> > > > > > > > > >
> > > > > > > > > >   *   1:1 memory mappings for Xen virtual machines, enabling device
> > > > > > > > > >     assignments without SMMU
> > > > > > > > > >   *
> > > > > > > > > >       Static assignment of PL blocks, including bus-mastering blocks, to
> > > > > > > > > >     Xen virtual machines
> > > > > > > > > >   *   Support for PL Device Passthrough to DomU in Xen
> > > > > > > > >
> > > > > > > > > You are usiong a version of Xen that has been customised by Xilinx. So I would
> > > > > > > > > recommend to contact them directly.
> > > > > > > > >
> > > > > > > > > I have CCed Stefano who might be able to help you.
> > > > > > > >
> > > > > > > > Hi Julien,
> > > > > > > >
> > > > > > > > Many thanks for CC'ing me, I missed the email.
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi Prem,
> > > > > > > >
> > > > > > > > Let me CC a few people that might be able to help.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
>