Mailing List Archive

Hello Julien,

> On 30 Oct 2020, at 9:21 am, Julien Grall <julien@xen.org> wrote:
>
> Hi,
>
> On 30/10/2020 08:46, Rahul Singh wrote:
>> Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also.
>
> I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread.
>
> The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen.
>
> However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9).
>
> So can you please clarify?
>
> Cheers,

Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations.

After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit.

>
> --
> Julien Grall

Regards,
Rahul

On 30/10/2020 09:45, Rahul Singh wrote:
> Hello Julien,
>
>> On 30 Oct 2020, at 9:21 am, Julien Grall <julien@xen.org> wrote:
>>
>> Hi,
>>
>> On 30/10/2020 08:46, Rahul Singh wrote:
>>> Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also.
>>
>> I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread.
>>
>> The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen.
>>
>> However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9).
>>
>> So can you please clarify?
>>
>> Cheers,
>
> Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations.
>
> After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit.

Ok. That was definitely not clear from the commit message. Please make
this clearer in the commit message.

Anway, it means we need to do a full review of the code (rather than a
light one) because of the hybrid model.

I am still a bit puzzle to why it would require almost of a restart of
the implementation in order to sync the latest code. Does it imply that
you are mostly using the old code?

Cheers,

--
Julien Grall

Hi, Rahul!

On 10/20/20 6:25 PM, Rahul Singh wrote:
> Add support for ARM architected SMMUv3 implementations. It is based on
> the Linux SMMUv3 driver.
>
> Major differences between the Linux driver are as follows:
> 1. Only Stage-2 translation is supported as compared to the Linux driver
> that supports both Stage-1 and Stage-2 translations.

First of all thank you for the efforts!

I tried the patch with QEMU and would like to know if my understanding correct

that this combination will not work as of now:

(XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
(XEN) Data Abort Trap. Syndrome=0x1940010
(XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
(XEN) 0TH[0x0] = 0x00000000b8468f7f

[snip]

If this is expected then is there any plan to make QEMU work as well?

I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.


We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough

implementation, so it could allow testing different setups and configurations with QEMU.


Thank you in advance,

Oleksandr

[1] https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.auger@redhat.com/

Hello Julien,

> On 30 Oct 2020, at 10:05 am, Julien Grall <julien@xen.org> wrote:
>
>
>
> On 30/10/2020 09:45, Rahul Singh wrote:
>> Hello Julien,
>>> On 30 Oct 2020, at 9:21 am, Julien Grall <julien@xen.org> wrote:
>>>
>>> Hi,
>>>
>>> On 30/10/2020 08:46, Rahul Singh wrote:
>>>> Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also.
>>>
>>> I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread.
>>>
>>> The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen.
>>>
>>> However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9).
>>>
>>> So can you please clarify?
>>>
>>> Cheers,
>> Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations.
>> After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit.
>
> Ok. That was definitely not clear from the commit message. Please make this clearer in the commit message.
>

Ok. I will make this clearer in the commit message.

> Anway, it means we need to do a full review of the code (rather than a light one) because of the hybrid model.
>
> I am still a bit puzzle to why it would require almost of a restart of the implementation in order to sync the latest code. Does it imply that you are mostly using the old code?
>

SMMuv3 code is divided into below parts :

1. Low-level/High level queue manipulation functions.
2. Context descriptor manipulation functions.
3. Stream table manipulation functions.
4. Interrupt handling.
5. Linux IOMMU API functions.
6. Driver initialisation functions( probe/reset ).

Low-level/High-level queue manipulation functions are from the old code, rest is the new code whenever it was possible.

I started with porting the latest code but there are many dependencies for the queue manipulation function so we decided to use the old queue manipulation function.
As the queue manipulation function is a big part of the code it will require a lot of effort and testing to sync with the latest code once the atomic operation is in place to use.

Once atomic operation is available in XEN we have merge the below commit from Linux to XEN to make XEN in sync with Linux code.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/iommu/arm-smmu-v3.c?h=v5.8&id=587e6c10a7ce89a5924fdbeff2ec524fbd6a124b

> Cheers,
>
> --
> Julien Grall

Regards,
Rahul

On 30/10/2020 11:33, Rahul Singh wrote:
> Hello Julien,

Hi,

>
>> On 30 Oct 2020, at 10:05 am, Julien Grall <julien@xen.org> wrote:
>>
>>
>>
>> On 30/10/2020 09:45, Rahul Singh wrote:
>>> Hello Julien,
>>>> On 30 Oct 2020, at 9:21 am, Julien Grall <julien@xen.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 30/10/2020 08:46, Rahul Singh wrote:
>>>>> Ok Yes when I ported the driver I port the command queue operation from the previous commit where atomic operations is not used and rest all the code is from the latest code. I will again make sure that any bug that is fixed in Linux should be fixed in XEN also.
>>>>
>>>> I would like to seek some clarifications on the code because there seem to be conflicting information provided in this thread.
>>>>
>>>> The patch (the baseline commit is provided) and the discussion with Bertrand suggests that you took a snapshot of the code last year and adapted for Xen.
>>>>
>>>> However, here you suggest that you took an hybrid approach where part of the code is based from last year and other part is based from the latest code (I assume v5.9).
>>>>
>>>> So can you please clarify?
>>>>
>>>> Cheers,
>>> Approach I took is to first merge the code from the commit ( Jul 2, 2019 7c288a5b27934281d9ea8b5807bc727268b7001a ) the snapshot before atomic operation is used in SMMUv3 code for command queue operations.
>>> After that I fixed the other code( not related to command queue operations.) from the latest code so that no bug is introduced in XEN because of using the last year commit.
>>
>> Ok. That was definitely not clear from the commit message. Please make this clearer in the commit message.
>>
>
> Ok. I will make this clearer in the commit message.
>
>> Anway, it means we need to do a full review of the code (rather than a light one) because of the hybrid model.
>>
>> I am still a bit puzzle to why it would require almost of a restart of the implementation in order to sync the latest code. Does it imply that you are mostly using the old code?
>>
>
> SMMuv3 code is divided into below parts :
>
> 1. Low-level/High level queue manipulation functions.
> 2. Context descriptor manipulation functions.
> 3. Stream table manipulation functions.
> 4. Interrupt handling.
> 5. Linux IOMMU API functions.
> 6. Driver initialisation functions( probe/reset ).
>
> Low-level/High-level queue manipulation functions are from the old code, rest is the new code whenever it was possible.

Thanks for the details! I think it would be useful to mention that in
the commit message.

> I started with porting the latest code but there are many dependencies for the queue manipulation function so we decided to use the old queue manipulation function.

In general, I would recommend to involve the community as soon as
possible in the development process. This is quite important for big
feature because it allows all the party to agree on the approach without
investing significant effort.

> As the queue manipulation function is a big part of the code it will require a lot of effort and testing to sync with the latest code once the atomic operation is in place to use
Sure, everything has a cost (including maintaining the code). This has
to be a trade-off.

My main concern was the maintenance of the code long term. However, as
you and Bertrand stepped up for maintaining the code, then this should
be less of a concern...

Cheers,

--
Julien Grall

Hello Oleksandr,

> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hi, Rahul!
>
> On 10/20/20 6:25 PM, Rahul Singh wrote:
>> Add support for ARM architected SMMUv3 implementations. It is based on
>> the Linux SMMUv3 driver.
>>
>> Major differences between the Linux driver are as follows:
>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>> that supports both Stage-1 and Stage-2 translations.
>
> First of all thank you for the efforts!
>
> I tried the patch with QEMU and would like to know if my understanding correct
>
> that this combination will not work as of now:
>
> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq

I have limited knowledge about QEMU internals.As what I see from the logs, fault is occurred at early driver initialisation when SMMU driver is trying to probe the HW.

> (XEN) Data Abort Trap. Syndrome=0x1940010
> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>
> [snip]
>
> If this is expected then is there any plan to make QEMU work as well?
>
> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.

Yes as of now only Stage-2 is supported in XEN.If we have any requirement or use case that depends on Stage-1 translation we can support that also in XEN.

>
>
> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>
> implementation, so it could allow testing different setups and configurations with QEMU.
>
>
> Thank you in advance,
>
> Oleksandr
>
> [1] https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.auger@redhat.com/

Regards,
Rahul

Hi,

On 10/30/20 4:47 PM, Rahul Singh wrote:
> Hello Oleksandr,
>
>> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>
>> Hi, Rahul!
>>
>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>> the Linux SMMUv3 driver.
>>>
>>> Major differences between the Linux driver are as follows:
>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>> that supports both Stage-1 and Stage-2 translations.
>> First of all thank you for the efforts!
>>
>> I tried the patch with QEMU and would like to know if my understanding correct
>>
>> that this combination will not work as of now:
>>
>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
> I have limited knowledge about QEMU internals.As what I see from the logs, fault is occurred at early driver initialisation when SMMU driver is trying to probe the HW.
>
>> (XEN) Data Abort Trap. Syndrome=0x1940010
>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>
>> [snip]
>>
>> If this is expected then is there any plan to make QEMU work as well?
>>
>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
> Yes as of now only Stage-2 is supported in XEN.If we have any requirement or use case that depends on Stage-1 translation we can support that also in XEN.
The use case is below: PCI passthrough and various configurations including SR-IOV which is possible with QEMU...
>
>>
>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>
>> implementation, so it could allow testing different setups and configurations with QEMU.
>>
>>
>> Thank you in advance,
>>
>> Oleksandr
>>
>> [1] https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.auger@redhat.com/__;!!GF_29dbcQIUBPA!h-EaE0OnSbXtLBSwIS311alDl7pn8sH7sihgIYqilM5-r-8kCH6USNNlLB3xhbzc6eczUOrhcw$ [patchwork[.]ozlabs[.]org]
> Regards,
> Rahul

Thank you,

Oleksandr

Hi Oleksandr,

> On 30 Oct 2020, at 15:02, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hi,
>
> On 10/30/20 4:47 PM, Rahul Singh wrote:
>> Hello Oleksandr,
>>
>>> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>
>>> Hi, Rahul!
>>>
>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>> the Linux SMMUv3 driver.
>>>>
>>>> Major differences between the Linux driver are as follows:
>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>> that supports both Stage-1 and Stage-2 translations.
>>> First of all thank you for the efforts!
>>>
>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>
>>> that this combination will not work as of now:
>>>
>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>> I have limited knowledge about QEMU internals.As what I see from the logs, fault is occurred at early driver initialisation when SMMU driver is trying to probe the HW.
>>
>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>
>>> [snip]
>>>
>>> If this is expected then is there any plan to make QEMU work as well?
>>>
>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>> Yes as of now only Stage-2 is supported in XEN.If we have any requirement or use case that depends on Stage-1 translation we can support that also in XEN.
> The use case is below: PCI passthrough and various configurations including SR-IOV which is possible with QEMU...

This is currently not in the list of configuration supported or that we have planned on our side to support.

But we would be more then happy to review any changes to enable this :-)

Regards
Bertrand

>>
>>>
>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>
>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>
>>>
>>> Thank you in advance,
>>>
>>> Oleksandr
>>>
>>> [1] https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.auger@redhat.com/__;!!GF_29dbcQIUBPA!h-EaE0OnSbXtLBSwIS311alDl7pn8sH7sihgIYqilM5-r-8kCH6USNNlLB3xhbzc6eczUOrhcw$ [patchwork[.]ozlabs[.]org]
>> Regards,
>> Rahul
>
> Thank you,
>
> Oleksandr

Hi Oleksandr,

On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
> On 10/20/20 6:25 PM, Rahul Singh wrote:
>> Add support for ARM architected SMMUv3 implementations. It is based on
>> the Linux SMMUv3 driver.
>>
>> Major differences between the Linux driver are as follows:
>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>> that supports both Stage-1 and Stage-2 translations.
>
> First of all thank you for the efforts!
>
> I tried the patch with QEMU and would like to know if my understanding correct
>
> that this combination will not work as of now:
>
> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
> (XEN) Data Abort Trap. Syndrome=0x1940010
> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>
> [snip]
>
> If this is expected then is there any plan to make QEMU work as well?
>
> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.

Just for clarication, you are trying to boot Xen on QEMU, right?

You might be able to use the stage-1 page-tables to isolate each device
in Xen. However, I don't think you will be able to share the P2M because
the page-tables layout between stage-1 and stage-2 is different.

>
>
> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>
> implementation, so it could allow testing different setups and configurations with QEMU.

I would recommend to get the SMMU supporting supporting stage-2 page-tables.

Regardless that, I think Xen should be able to say the SMMU is not
supported rather than crashing.

Cheers,

--
Julien Grall

Hi,

On 10/30/20 6:08 PM, Bertrand Marquis wrote:
> Hi Oleksandr,
>
>> On 30 Oct 2020, at 15:02, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>
>> Hi,
>>
>> On 10/30/20 4:47 PM, Rahul Singh wrote:
>>> Hello Oleksandr,
>>>
>>>> On 30 Oct 2020, at 10:44 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>>
>>>> Hi, Rahul!
>>>>
>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>> the Linux SMMUv3 driver.
>>>>>
>>>>> Major differences between the Linux driver are as follows:
>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>> that supports both Stage-1 and Stage-2 translations.
>>>> First of all thank you for the efforts!
>>>>
>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>
>>>> that this combination will not work as of now:
>>>>
>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>> I have limited knowledge about QEMU internals.As what I see from the logs, fault is occurred at early driver initialisation when SMMU driver is trying to probe the HW.
>>>
>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>
>>>> [snip]
>>>>
>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>
>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>> Yes as of now only Stage-2 is supported in XEN.If we have any requirement or use case that depends on Stage-1 translation we can support that also in XEN.
>> The use case is below: PCI passthrough and various configurations including SR-IOV which is possible with QEMU...
> This is currently not in the list of configuration supported or that we have planned on our side to support.
>
> But we would be more then happy to review any changes to enable this :-)

Fair enough

Unfortunately we do not have any HW with SMMUv3 at the moment, so not sure we can

invest time in this at the moment

>
> Regards
> Bertrand

Thank you,

Oleksandr

>
>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>
>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>
>>>>
>>>> Thank you in advance,
>>>>
>>>> Oleksandr
>>>>
>>>> [1] https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/qemu-devel/cover/1524665762-31355-1-git-send-email-eric.auger@redhat.com/__;!!GF_29dbcQIUBPA!h-EaE0OnSbXtLBSwIS311alDl7pn8sH7sihgIYqilM5-r-8kCH6USNNlLB3xhbzc6eczUOrhcw$ [patchwork[.]ozlabs[.]org]
>>> Regards,
>>> Rahul
>> Thank you,
>>
>> Oleksandr
>

Hi, Julien!

On 10/30/20 7:18 PM, Julien Grall wrote:
> Hi Oleksandr,
>
> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>> the Linux SMMUv3 driver.
>>>
>>> Major differences between the Linux driver are as follows:
>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>      that supports both Stage-1 and Stage-2 translations.
>>
>> First of all thank you for the efforts!
>>
>> I tried the patch with QEMU and would like to know if my understanding correct
>>
>> that this combination will not work as of now:
>>
>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>> (XEN) Data Abort Trap. Syndrome=0x1940010
>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>
>> [snip]
>>
>> If this is expected then is there any plan to make QEMU work as well?
>>
>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>
> Just for clarication, you are trying to boot Xen on QEMU, right?
Exactly
>
> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
So, it is even more work then
>
>>
>>
>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>
>> implementation, so it could allow testing different setups and configurations with QEMU.
>
> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
You mean in QEMU?
>
> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
Yes, that would be nice
>
> Cheers,
>
Thank you,

Oleksandr

Hi,

> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>
> Hi, Julien!
>
> On 10/30/20 7:18 PM, Julien Grall wrote:
>> Hi Oleksandr,
>>
>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>> the Linux SMMUv3 driver.
>>>>
>>>> Major differences between the Linux driver are as follows:
>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>> that supports both Stage-1 and Stage-2 translations.
>>>
>>> First of all thank you for the efforts!
>>>
>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>
>>> that this combination will not work as of now:
>>>
>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>
>>> [snip]
>>>
>>> If this is expected then is there any plan to make QEMU work as well?
>>>
>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>
>> Just for clarication, you are trying to boot Xen on QEMU, right?
> Exactly
>>
>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
> So, it is even more work then

Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.

>>
>>>
>>>
>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>
>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>
>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
> You mean in QEMU?

See before.

>>
>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
> Yes, that would be nice

Fully agree and we will look into that.

Anything you could share so that we could quickly reproduce your setup would be more then great.

Regards
Bertrand

>>
>> Cheers,
>>
> Thank you,
>
> Oleksandr

Hi,

On 11/2/20 11:55 AM, Bertrand Marquis wrote:
> Hi,
>
>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>
>> Hi, Julien!
>>
>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>> Hi Oleksandr,
>>>
>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>> the Linux SMMUv3 driver.
>>>>>
>>>>> Major differences between the Linux driver are as follows:
>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>> that supports both Stage-1 and Stage-2 translations.
>>>> First of all thank you for the efforts!
>>>>
>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>
>>>> that this combination will not work as of now:
>>>>
>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>
>>>> [snip]
>>>>
>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>
>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>> Exactly
>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>> So, it is even more work then
> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>
>>>>
>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>
>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>> You mean in QEMU?
> See before.
>
>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>> Yes, that would be nice
> Fully agree and we will look into that.
>
> Anything you could share so that we could quickly reproduce your setup would be more then great.

Nothing special,

qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \

-machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \

-nographic -serial mon:stdio [..snip..]

I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1

>
> Regards
> Bertrand
>
>>> Cheers,
>>>
>> Thank you,
>>
>> Oleksandr

Hello Oleksandr,

> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hi,
>
> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>> Hi,
>>
>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>
>>> Hi, Julien!
>>>
>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>> Hi Oleksandr,
>>>>
>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>> the Linux SMMUv3 driver.
>>>>>>
>>>>>> Major differences between the Linux driver are as follows:
>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>> First of all thank you for the efforts!
>>>>>
>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>
>>>>> that this combination will not work as of now:
>>>>>
>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>
>>>>> [snip]
>>>>>
>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>
>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>> Exactly
>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>> So, it is even more work then
>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>
>>>>>
>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>
>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>> You mean in QEMU?
>> See before.
>>
>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>> Yes, that would be nice
>> Fully agree and we will look into that.
>>
>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>
> Nothing special,
>
> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>
> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>
> -nographic -serial mon:stdio [..snip..]
>
> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1

Thanks for sharing the information. I will also try to reproduce the issue and will let you know the results.

>
>>
>> Regards
>> Bertrand
>>
>>>> Cheers,
>>>>
>>> Thank you,
>>>
>>> Oleksandr

Regards,
Rahul

Hello Oleksandr,

> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hi,
>
> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>> Hi,
>>
>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>
>>> Hi, Julien!
>>>
>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>> Hi Oleksandr,
>>>>
>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>> the Linux SMMUv3 driver.
>>>>>>
>>>>>> Major differences between the Linux driver are as follows:
>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>> First of all thank you for the efforts!
>>>>>
>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>
>>>>> that this combination will not work as of now:
>>>>>
>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>
>>>>> [snip]
>>>>>
>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>
>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>> Exactly
>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>> So, it is even more work then
>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>
>>>>>
>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>
>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>> You mean in QEMU?
>> See before.
>>
>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>> Yes, that would be nice
>> Fully agree and we will look into that.
>>
>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>
> Nothing special,
>
> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>
> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>
> -nographic -serial mon:stdio [..snip..]
>
> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1

I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.


(XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
(XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
(XEN) SMMUv3: /smmuv3@9050000: no translation support!
(XEN) I/O virtualisation disabled

Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.

Please let me know if it also works for you.

>
>>
>> Regards
>> Bertrand
>>
>>>> Cheers,
>>>>
>>> Thank you,
>>>
>>> Oleksandr

Hello, Rahul!

On 11/6/20 2:48 PM, Rahul Singh wrote:
> Hello Oleksandr,
>
>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>
>> Hi,
>>
>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>> Hi,
>>>
>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>>
>>>> Hi, Julien!
>>>>
>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>> Hi Oleksandr,
>>>>>
>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>> the Linux SMMUv3 driver.
>>>>>>>
>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>>> First of all thank you for the efforts!
>>>>>>
>>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>>
>>>>>> that this combination will not work as of now:
>>>>>>
>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>
>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>> Exactly
>>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>>> So, it is even more work then
>>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>>
>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>>
>>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>>> You mean in QEMU?
>>> See before.
>>>
>>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>>> Yes, that would be nice
>>> Fully agree and we will look into that.
>>>
>>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>> Nothing special,
>>
>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>>
>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>
>> -nographic -serial mon:stdio [..snip..]
>>
>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.
>
>
> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
> (XEN) I/O virtualisation disabled
>
> Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.
I do use the option
>
> Please let me know if it also works for you.

Well, I should have reported that earlier that I do not use the staging Xen at the moment,

it is 4.14.0. So, can this be a problem with that Xen version?

Anyways, if it works with the staging then everything looks ok

Thank you,

Oleksandr

>
>>> Regards
>>> Bertrand
>>>
>>>>> Cheers,
>>>>>
>>>> Thank you,
>>>>
>>>> Oleksandr

Hello Oleksandr,

> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hello, Rahul!
>
> On 11/6/20 2:48 PM, Rahul Singh wrote:
>> Hello Oleksandr,
>>
>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>
>>> Hi,
>>>
>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>> Hi,
>>>>
>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>>>
>>>>> Hi, Julien!
>>>>>
>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>> Hi Oleksandr,
>>>>>>
>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>
>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>>>> First of all thank you for the efforts!
>>>>>>>
>>>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>>>
>>>>>>> that this combination will not work as of now:
>>>>>>>
>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>
>>>>>>> [snip]
>>>>>>>
>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>
>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>> Exactly
>>>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>>>> So, it is even more work then
>>>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>>>
>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>>>
>>>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>>>> You mean in QEMU?
>>>> See before.
>>>>
>>>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>>>> Yes, that would be nice
>>>> Fully agree and we will look into that.
>>>>
>>>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>>> Nothing special,
>>>
>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>>>
>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>
>>> -nographic -serial mon:stdio [..snip..]
>>>
>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.
>>
>>
>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>> (XEN) I/O virtualisation disabled
>>
>> Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.
> I do use the option

I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and while launching the QEMU.
I also observed the same error what you observed if I am not using the "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this might be case for you also but anyways you have use the options it might be other issue.

>>
>> Please let me know if it also works for you.
>
> Well, I should have reported that earlier that I do not use the staging Xen at the moment,
>
> it is 4.14.0. So, can this be a problem with that Xen version?

I don’t think so this is the problem with the XEN version.
>
> Anyways, if it works with the staging then everything looks ok
>
> Thank you,
>
> Oleksandr
>
>>
>>>> Regards
>>>> Bertrand
>>>>
>>>>>> Cheers,
>>>>>>
>>>>> Thank you,
>>>>>
>>>>> Oleksandr

Regards,
Rahul

Hi, Rahul!

On 11/6/20 3:58 PM, Rahul Singh wrote:
> Hello Oleksandr,
>
>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>
>> Hello, Rahul!
>>
>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>> Hello Oleksandr,
>>>
>>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>>> Hi,
>>>>>
>>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>>>>
>>>>>> Hi, Julien!
>>>>>>
>>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>>> Hi Oleksandr,
>>>>>>>
>>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>>
>>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>>>>> First of all thank you for the efforts!
>>>>>>>>
>>>>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>>>>
>>>>>>>> that this combination will not work as of now:
>>>>>>>>
>>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>>
>>>>>>>> [snip]
>>>>>>>>
>>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>>
>>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>>> Exactly
>>>>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>>>>> So, it is even more work then
>>>>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>>>>
>>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>>>>
>>>>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>>>>> You mean in QEMU?
>>>>> See before.
>>>>>
>>>>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>>>>> Yes, that would be nice
>>>>> Fully agree and we will look into that.
>>>>>
>>>>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>>>> Nothing special,
>>>>
>>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>>>>
>>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>>
>>>> -nographic -serial mon:stdio [..snip..]
>>>>
>>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.
>>>
>>>
>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
>>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>>> (XEN) I/O virtualisation disabled
>>>
>>> Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.
>> I do use the option
> I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and while launching the QEMU.
> I also observed the same error what you observed if I am not using the "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this might be case for you also but anyways you have use the options it might be other issue.

Hm, probably that was on my side as now I can see:

(XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
(XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
(XEN) SMMUv3: /smmuv3@9050000: no translation support!
(XEN) I/O virtualisation disabled
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Couldn't configure correctly all the IOMMUs.
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

So, sorry for the noise, I might have misconfigured something it seems

When you say "Xen is booting" do you mean you see the same panic?

Thank you,

Oleksandr

>
>>> Please let me know if it also works for you.
>> Well, I should have reported that earlier that I do not use the staging Xen at the moment,
>>
>> it is 4.14.0. So, can this be a problem with that Xen version?
> I don’t think so this is the problem with the XEN version.
>> Anyways, if it works with the staging then everything looks ok
>>
>> Thank you,
>>
>> Oleksandr
>>
>>>>> Regards
>>>>> Bertrand
>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Oleksandr
> Regards,
> Rahul

Hello Oleksandr,

> On 6 Nov 2020, at 2:22 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>
> Hi, Rahul!
>
> On 11/6/20 3:58 PM, Rahul Singh wrote:
>> Hello Oleksandr,
>>
>>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>
>>> Hello, Rahul!
>>>
>>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>>> Hello Oleksandr,
>>>>
>>>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>>>> Hi,
>>>>>>
>>>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi, Julien!
>>>>>>>
>>>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>>>> Hi Oleksandr,
>>>>>>>>
>>>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>>>
>>>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>>>>>> First of all thank you for the efforts!
>>>>>>>>>
>>>>>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>>>>>
>>>>>>>>> that this combination will not work as of now:
>>>>>>>>>
>>>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>>>
>>>>>>>>> [snip]
>>>>>>>>>
>>>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>>>
>>>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>>>> Exactly
>>>>>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>>>>>> So, it is even more work then
>>>>>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>>>>>
>>>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>>>>>
>>>>>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>>>>>> You mean in QEMU?
>>>>>> See before.
>>>>>>
>>>>>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>>>>>> Yes, that would be nice
>>>>>> Fully agree and we will look into that.
>>>>>>
>>>>>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>>>>> Nothing special,
>>>>>
>>>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>>>>>
>>>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>>>
>>>>> -nographic -serial mon:stdio [..snip..]
>>>>>
>>>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.
>>>>
>>>>
>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
>>>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>>>> (XEN) I/O virtualisation disabled
>>>>
>>>> Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.
>>> I do use the option
>> I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and while launching the QEMU.
>> I also observed the same error what you observed if I am not using the "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this might be case for you also but anyways you have use the options it might be other issue.
>
> Hm, probably that was on my side as now I can see:
>
> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
> (XEN) I/O virtualisation disabled
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Couldn't configure correctly all the IOMMUs.
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
>
> So, sorry for the noise, I might have misconfigured something it seems
>
> When you say "Xen is booting" do you mean you see the same panic?

Yes I observe the same.

We have to decide now if for SMMUv3 there is no translation support do we have to print the logs and move forward or as above return error to iommu_setup that will cal panic().

Regards,
Rahul

>
> Thank you,
>
> Oleksandr
>
>>
>>>> Please let me know if it also works for you.
>>> Well, I should have reported that earlier that I do not use the staging Xen at the moment,
>>>
>>> it is 4.14.0. So, can this be a problem with that Xen version?
>> I don’t think so this is the problem with the XEN version.
>>> Anyways, if it works with the staging then everything looks ok
>>>
>>> Thank you,
>>>
>>> Oleksandr
>>>
>>>>>> Regards
>>>>>> Bertrand
>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Oleksandr
>> Regards,
>> Rahul

Hello, Rahul!

On 11/6/20 4:41 PM, Rahul Singh wrote:
> Hello Oleksandr,
>
>> On 6 Nov 2020, at 2:22 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>
>> Hi, Rahul!
>>
>> On 11/6/20 3:58 PM, Rahul Singh wrote:
>>> Hello Oleksandr,
>>>
>>>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>>
>>>> Hello, Rahul!
>>>>
>>>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>>>> Hello Oleksandr,
>>>>>
>>>>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi, Julien!
>>>>>>>>
>>>>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>>>>> Hi Oleksandr,
>>>>>>>>>
>>>>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based on
>>>>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>>>>
>>>>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux driver
>>>>>>>>>>> that supports both Stage-1 and Stage-2 translations.
>>>>>>>>>> First of all thank you for the efforts!
>>>>>>>>>>
>>>>>>>>>> I tried the patch with QEMU and would like to know if my understanding correct
>>>>>>>>>>
>>>>>>>>>> that this combination will not work as of now:
>>>>>>>>>>
>>>>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 0x00000000b8469000
>>>>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>>>>
>>>>>>>>>> [snip]
>>>>>>>>>>
>>>>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>>>>
>>>>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on QEMU side.
>>>>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>>>>> Exactly
>>>>>>>>> You might be able to use the stage-1 page-tables to isolate each device in Xen. However, I don't think you will be able to share the P2M because the page-tables layout between stage-1 and stage-2 is different.
>>>>>>>> So, it is even more work then
>>>>>>> Overall it would make more sense to spend some time adding proper support in Qemu then trying to modify the driver to support Qemu right now.
>>>>>>>
>>>>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI passthrough
>>>>>>>>>>
>>>>>>>>>> implementation, so it could allow testing different setups and configurations with QEMU.
>>>>>>>>> I would recommend to get the SMMU supporting supporting stage-2 page-tables.
>>>>>>>> You mean in QEMU?
>>>>>>> See before.
>>>>>>>
>>>>>>>>> Regardless that, I think Xen should be able to say the SMMU is not supported rather than crashing.
>>>>>>>> Yes, that would be nice
>>>>>>> Fully agree and we will look into that.
>>>>>>>
>>>>>>> Anything you could share so that we could quickly reproduce your setup would be more then great.
>>>>>> Nothing special,
>>>>>>
>>>>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine virt,gic-version=2 \
>>>>>>
>>>>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>>>>
>>>>>> -nographic -serial mon:stdio [..snip..]
>>>>>>
>>>>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch and XEN is able to say SMMU translation is not supported. As XEN supports Stage-2 translation and QEMU supports Stage-1 only.
>>>>>
>>>>>
>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
>>>>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>>>>> (XEN) I/O virtualisation disabled
>>>>>
>>>>> Only difference I observed is that you have to add option "-machine virt,iommu=smmuv3 “ when launching the QEMU.
>>>> I do use the option
>>> I used "-machine virt,iommu=smmuv3 “ option while creating the virt-dtb and while launching the QEMU.
>>> I also observed the same error what you observed if I am not using the "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought this might be case for you also but anyways you have use the options it might be other issue.
>> Hm, probably that was on my side as now I can see:
>>
>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration (false)
>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>> (XEN) I/O virtualisation disabled
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Couldn't configure correctly all the IOMMUs.
>> (XEN) ****************************************
>> (XEN)
>> (XEN) Manual reset required ('noreboot' specified)
>>
>> So, sorry for the noise, I might have misconfigured something it seems
>>
>> When you say "Xen is booting" do you mean you see the same panic?
> Yes I observe the same.
>
> We have to decide now if for SMMUv3 there is no translation support do we have to print the logs and move forward or as above return error to iommu_setup that will cal panic().
I would allow Xen to boot further
>
> Regards,
> Rahul
>
>> Thank you,
>>
>> Oleksandr
>>
>>>>> Please let me know if it also works for you.
>>>> Well, I should have reported that earlier that I do not use the staging Xen at the moment,
>>>>
>>>> it is 4.14.0. So, can this be a problem with that Xen version?
>>> I don’t think so this is the problem with the XEN version.
>>>> Anyways, if it works with the staging then everything looks ok
>>>>
>>>> Thank you,
>>>>
>>>> Oleksandr
>>>>
>>>>>>> Regards
>>>>>>> Bertrand
>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> Oleksandr
>>> Regards,
>>> Rahul