Mailing List Archive

(+ Bertrand, Stefano)

Hi Chuck,

Thanks for the report.

On 26/10/2023 17:17, Chuck Zmudzinski wrote:
> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>> We also have not yet done a thorough analysis of the differences
>> in the kernel boot logs when booting on the bare metal vs. booting
>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>> cause of this problem after a quick look at the logs.
>
> After a more careful look at the logs, this seems to be the error
> causing no display when booting as dom0 on Xen:
>
> *ERROR* Device 14450000.mixer lacks support for IOMMU
>
> A little more context from the logs follows (I did not word wrap
> the log messages to 72 characters because they are easier to read
> without word wrapping).
>
> On bare metal:
>
> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [ 2.535938] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [ 2.536139] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [ 2.536274] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [ 2.536493] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [ 2.536520] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
> ...
> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [ 3.493686] Console: switching to colour frame buffer device 170x48
> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [ 3.521747] exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [ 3.522831] [drm] Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>
> The screen works normally in this case.
>
> On Xen as dom0:
>
> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [ 2.606812] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [ 2.606884] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [ 2.606999] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU
> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [ 2.607044] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [ 2.607162] exynos-drm exynos-drm: adev bind failed: -22
> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [ 2.607183] exynos-dp: probe of 145b0000.dp-controller failed with error -22
>
> There is no display on the screen in this case. The backlight
> does not even come on.
>
> So the error causing no display is probably:
>
> *ERROR* Device 14450000.mixer lacks support for IOMMU
>
> I am new to arm virtualization with Xen. I understand IOMMU on x86
> is needed for PCI passthrough to domU guests, but not for dom0 to
> use such devices.

I believe that the IOMMU would be required on x86 when using dom0 PVH.
PVH is very similar to an Arm guests.

On Arm, we don't require the IOMMU because not all Arm platforms have
all DMA-capable devices protected by an IOMMU. So dom0 will still have
its memory direct mapped (i.e. host physical address = guest physical
address) to allow DMA in dom0 with limited modification.

That said, I thik this is a different situation here (see below).


> So on arm, why is dom0 trying to use IOMMU for
> the exynos-mixer/exynos-drm when bare metal does not use it?

Just to confirm, are you using the same kernel, same config when booting
on baremetal? If so, from looking at the code, I would expect that the
IOMMU is also used on baremetal.

The check failing is:

if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))

I am not quite too sure why the check implies the IOMMU is not
supported. That said, I vaguely recall that Linux will update the DMA
ops when running under Xen. Would you be able to print the two values
returned ("%pS" should give the symbol)?

Anyway, letting dom0 to use the IOMMU is probably a bad idea as even if
dom0 memory is direct mapped, grant mappings are not. So you would end
up to see random crashes.

Right now, if Xen doesn't use the IOMMU (e.g. because it was disabled or
there is no driver), then the device will be assigned to dom0. We
recently had some discussion to hide the IOMMU from dom0. I expect a
patch to be on the ML in the near future.

As a temporary hack, would you be able to compile out the IOMMU driver
from Linux and check if it helps using the GPU?

Looking at the documentation in the Linux tree, I am under the impresion
that the Exynos SMMUs are mainly used to avoid allocating large
contiguous buffer. So in the longer run, it might be good to understand
the performance impact of hiding them from dom0.

>
> I also noted this difference in the logs:
>
> Bare metal:
> 1999-12-31T20:03:21.723483-05:00 devuan-bunsen kernel: [ 0.000000] Normal [mem 0x0000000040000000-0x000000006fffffff]
>
> Xen dom0:
> 1999-12-31T20:01:09.720365-05:00 devuan-bunsen kernel: [ 0.000000] Normal [mem 0x0000000060000000-0x000000008fffffff]
>
> If I am reading those numbers correctly, normal memory starts
> at 1 GB on bare metal, but at 1.5 GB as Xen dom0.
>
> Could that be causing dom0 to try to use IOMMU for
> exynos-mixer/exynos-drm?
>

Cheers,

[1]
https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt

--
Julien Grall

On 10/26/2023 5:24 PM, Julien Grall wrote:
> (+ Bertrand, Stefano)
>
> Hi Chuck,
>
> Thanks for the report.
>
> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>> We also have not yet done a thorough analysis of the differences
>>> in the kernel boot logs when booting on the bare metal vs. booting
>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>> cause of this problem after a quick look at the logs.
>>
>> After a more careful look at the logs, this seems to be the error
>> causing no display when booting as dom0 on Xen:
>>
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>
>> A little more context from the logs follows (I did not word wrap
>> the log messages to 72 characters because they are easier to read
>> without word wrapping).
>>
>> On bare metal:
>>
>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [ 2.535938] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [ 2.536139] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [ 2.536274] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [ 2.536493] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [ 2.536520] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>> ...
>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [ 3.493686] Console: switching to colour frame buffer device 170x48
>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [ 3.521747] exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [ 3.522831] [drm] Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>>
>> The screen works normally in this case.
>>
>> On Xen as dom0:
>>
>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [ 2.606812] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [ 2.606884] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [ 2.606999] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [ 2.607044] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [ 2.607162] exynos-drm exynos-drm: adev bind failed: -22
>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [ 2.607183] exynos-dp: probe of 145b0000.dp-controller failed with error -22
>>
>> There is no display on the screen in this case. The backlight
>> does not even come on.
>>
>> So the error causing no display is probably:
>>
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>
>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>> is needed for PCI passthrough to domU guests, but not for dom0 to
>> use such devices.
>
> I believe that the IOMMU would be required on x86 when using dom0 PVH.
> PVH is very similar to an Arm guests.
>
> On Arm, we don't require the IOMMU because not all Arm platforms have
> all DMA-capable devices protected by an IOMMU. So dom0 will still have
> its memory direct mapped (i.e. host physical address = guest physical
> address) to allow DMA in dom0 with limited modification.
>
> That said, I thik this is a different situation here (see below).
>
>
>> So on arm, why is dom0 trying to use IOMMU for
>> the exynos-mixer/exynos-drm when bare metal does not use it?
>
> Just to confirm, are you using the same kernel, same config when booting
> on baremetal? If so, from looking at the code, I would expect that the
> IOMMU is also used on baremetal.
>
> The check failing is:
>
> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
>
> I am not quite too sure why the check implies the IOMMU is not
> supported. That said, I vaguely recall that Linux will update the DMA
> ops when running under Xen. Would you be able to print the two values
> returned ("%pS" should give the symbol)?

I think maybe those two are referring to the exynos_drm and exynos_mixer
devices. It looks like Linux dom0 was trying to attach the exynos_mixer
but it failed with the error. I will recompile to print these to be sure
what they are if I can't find a way to print them by increasing the log
level.

>
> Anyway, letting dom0 to use the IOMMU is probably a bad idea as even if
> dom0 memory is direct mapped, grant mappings are not. So you would end
> up to see random crashes.

I noticed some possible relevant settings from my Linux config:

CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
# CONFIG_XEN_GRANT_DMA_ALLOC is not set

I also have in my dom0 config:

# CONFIG_XEN_VIRTIO is not set

Also, not even mentioned in my Linux config but option is available in
Linux, but it can be enabled by enabling CONFIG_XEN_VIRTIO in the
configuration stage of the Linux build:

# CONFIG_XEN_GRANT_DMA_IOMMU is not set

Is virtio a safe way to provide IOMMU support for dom0? Maybe I can
enable this and see if it helps, but will it cause random crashes?

>
> Right now, if Xen doesn't use the IOMMU (e.g. because it was disabled or
> there is no driver), then the device will be assigned to dom0. We
> recently had some discussion to hide the IOMMU from dom0. I expect a
> patch to be on the ML in the near future.
>
> As a temporary hack, would you be able to compile out the IOMMU driver
> from Linux and check if it helps using the GPU?

I will try that also. I can start by disabling these:

CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y

and also these:

CONFIG_OF_IOMMU=y
CONFIG_EXYNOS_IOMMU=y

I can also try to determine if removing support for the IOMMU affects
GPU performance on bare metal.

>
> Looking at the documentation in the Linux tree, I am under the impresion
> that the Exynos SMMUs are mainly used to avoid allocating large
> contiguous buffer. So in the longer run, it might be good to understand
> the performance impact of hiding them from dom0.
>
>>
>> I also noted this difference in the logs:
>>
>> Bare metal:
>> 1999-12-31T20:03:21.723483-05:00 devuan-bunsen kernel: [ 0.000000] Normal [mem 0x0000000040000000-0x000000006fffffff]
>>
>> Xen dom0:
>> 1999-12-31T20:01:09.720365-05:00 devuan-bunsen kernel: [ 0.000000] Normal [mem 0x0000000060000000-0x000000008fffffff]
>>
>> If I am reading those numbers correctly, normal memory starts
>> at 1 GB on bare metal, but at 1.5 GB as Xen dom0.
>>
>> Could that be causing dom0 to try to use IOMMU for
>> exynos-mixer/exynos-drm?
>>
>
> Cheers,
>
> [1]
> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>

On 10/26/2023 5:24 PM, Julien Grall wrote:
> (+ Bertrand, Stefano)
>
> Hi Chuck,
>
> Thanks for the report.
>
> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>> We also have not yet done a thorough analysis of the differences
>>> in the kernel boot logs when booting on the bare metal vs. booting
>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>> cause of this problem after a quick look at the logs.
>>
>> After a more careful look at the logs, this seems to be the error
>> causing no display when booting as dom0 on Xen:
>>
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>
>> A little more context from the logs follows (I did not word wrap
>> the log messages to 72 characters because they are easier to read
>> without word wrapping).
>>
>> On bare metal:
>>
>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [ 2.535938] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [ 2.536139] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [ 2.536274] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [ 2.536493] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [ 2.536520] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>> ...
>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [ 3.493686] Console: switching to colour frame buffer device 170x48
>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [ 3.521747] exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [ 3.522831] [drm] Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>>
>> The screen works normally in this case.
>>
>> On Xen as dom0:
>>
>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [ 2.606812] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [ 2.606884] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [ 2.606999] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [ 2.607044] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [ 2.607162] exynos-drm exynos-drm: adev bind failed: -22
>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [ 2.607183] exynos-dp: probe of 145b0000.dp-controller failed with error -22
>>
>> There is no display on the screen in this case. The backlight
>> does not even come on.
>>
>> So the error causing no display is probably:
>>
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>
>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>> is needed for PCI passthrough to domU guests, but not for dom0 to
>> use such devices.
>
> I believe that the IOMMU would be required on x86 when using dom0 PVH.
> PVH is very similar to an Arm guests.
>
> On Arm, we don't require the IOMMU because not all Arm platforms have
> all DMA-capable devices protected by an IOMMU. So dom0 will still have
> its memory direct mapped (i.e. host physical address = guest physical
> address) to allow DMA in dom0 with limited modification.
>
> That said, I thik this is a different situation here (see below).
>
>
>> So on arm, why is dom0 trying to use IOMMU for
>> the exynos-mixer/exynos-drm when bare metal does not use it?
>
> Just to confirm, are you using the same kernel, same config when booting
> on baremetal? If so, from looking at the code, I would expect that the
> IOMMU is also used on baremetal.
>
> The check failing is:
>
> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
>
> I am not quite too sure why the check implies the IOMMU is not
> supported. That said, I vaguely recall that Linux will update the DMA
> ops when running under Xen. Would you be able to print the two values
> returned ("%pS" should give the symbol)?

I got those values:

[ 2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 0xc0d662dc

I presume you know how to interpret those. The failed test is that they are not equal.

For more context of that message from the kernel log:

[ 2.551932] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
[ 2.551981] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
[ 2.552094] [drm] ops(priv->dma_dev): 0xc0d018c0, ops(subdrv_dev): 0xc0d662dc
[ 2.552109] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU
[ 2.552152] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
[ 2.552266] exynos-drm exynos-drm: adev bind failed: -22
[ 2.552288] exynos-dp: probe of 145b0000.dp-controller failed with error -22

Tomorrow I will try the hack of removing support for the IOMMU from Linux and
report the results here.

>
> Anyway, letting dom0 to use the IOMMU is probably a bad idea as even if
> dom0 memory is direct mapped, grant mappings are not. So you would end
> up to see random crashes.
>
> Right now, if Xen doesn't use the IOMMU (e.g. because it was disabled or
> there is no driver), then the device will be assigned to dom0. We
> recently had some discussion to hide the IOMMU from dom0. I expect a
> patch to be on the ML in the near future.
>
> As a temporary hack, would you be able to compile out the IOMMU driver
> from Linux and check if it helps using the GPU?
>
> Looking at the documentation in the Linux tree, I am under the impresion
> that the Exynos SMMUs are mainly used to avoid allocating large
> contiguous buffer. So in the longer run, it might be good to understand
> the performance impact of hiding them from dom0. ...
>
> ...>
> Cheers,
>
> [1]
> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>

On 10/26/2023 5:24 PM, Julien Grall wrote:
> (+ Bertrand, Stefano)
>
> Hi Chuck,
>
> Thanks for the report.
>
> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>> We also have not yet done a thorough analysis of the differences
>>> in the kernel boot logs when booting on the bare metal vs. booting
>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>> cause of this problem after a quick look at the logs.
>>
>> After a more careful look at the logs, this seems to be the error
>> causing no display when booting as dom0 on Xen:
>>
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>
>> A little more context from the logs follows (I did not word wrap
>> the log messages to 72 characters because they are easier to read
>> without word wrapping).
>>
>> On bare metal:
>>
>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [ 2.535938] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [ 2.536139] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [ 2.536274] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [ 2.536493] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [ 2.536520] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>> ...
>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [ 3.493686] Console: switching to colour frame buffer device 170x48
>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [ 3.521747] exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [ 3.522831] [drm] Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>>
>> The screen works normally in this case.
>>
>> On Xen as dom0:
>>
>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [ 2.606812] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [ 2.606884] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [ 2.606999] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU
>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [ 2.607044] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [ 2.607162] exynos-drm exynos-drm: adev bind failed: -22
>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [ 2.607183] exynos-dp: probe of 145b0000.dp-controller failed with error -22
>>
>> There is no display on the screen in this case. The backlight
>> does not even come on.
>>
>> So the error causing no display is probably:
>>
>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>
>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>> is needed for PCI passthrough to domU guests, but not for dom0 to
>> use such devices.
>
> I believe that the IOMMU would be required on x86 when using dom0 PVH.
> PVH is very similar to an Arm guests.
>
> On Arm, we don't require the IOMMU because not all Arm platforms have
> all DMA-capable devices protected by an IOMMU. So dom0 will still have
> its memory direct mapped (i.e. host physical address = guest physical
> address) to allow DMA in dom0 with limited modification.
>
> That said, I thik this is a different situation here (see below).
>
>
>> So on arm, why is dom0 trying to use IOMMU for
>> the exynos-mixer/exynos-drm when bare metal does not use it?
>
> Just to confirm, are you using the same kernel, same config when booting
> on baremetal? If so, from looking at the code, I would expect that the
> IOMMU is also used on baremetal.

Hi Julien,

I didn't see that you asked this. The answer is yes, it is the same
kernel on the bare metal. I also thought from my brief look at the
code that on bare metal the IOMMU is also used, or at least, given
the logs for the bare metal case and the Xen case, I don't see any
evidence this check is not made on bare metal, but apparently the
check passes on bare metal.

>
> The check failing is:
>
> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
>
> I am not quite too sure why the check implies the IOMMU is not
> supported. That said, I vaguely recall that Linux will update the DMA
> ops when running under Xen. Would you be able to print the two values
> returned ("%pS" should give the symbol)?

I got those values now:

get_dma_ops(priv->dma_dev): 0xc0d018c0
get_dma_ops(subdrv_dev): 0xc0d662dc

Kind regards,

Chuck

>>
>
> Cheers,
>
> [1]
> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>

On 10/27/2023 12:06 AM, Chuck Zmudzinski wrote:
> On 10/26/2023 5:24 PM, Julien Grall wrote:
>> (+ Bertrand, Stefano)
>>
>> Hi Chuck,
>>
>> Thanks for the report.
>>
>> On 26/10/2023 17:17, Chuck Zmudzinski wrote:
>>> On 10/25/2023 10:44 PM, Chuck Zmudzinski wrote:
>>>> We also have not yet done a thorough analysis of the differences
>>>> in the kernel boot logs when booting on the bare metal vs. booting
>>>> as dom0 on Xen, but nothing stood out in the logs as an obvious
>>>> cause of this problem after a quick look at the logs.
>>>
>>> After a more careful look at the logs, this seems to be the error
>>> causing no display when booting as dom0 on Xen:
>>>
>>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>>
>>> A little more context from the logs follows (I did not word wrap
>>> the log messages to 72 characters because they are easier to read
>>> without word wrapping).
>>>
>>> On bare metal:
>>>
>>> 1999-12-31T20:03:21.728453-05:00 devuan-bunsen kernel: [ 2.535938] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>>> 1999-12-31T20:03:21.728461-05:00 devuan-bunsen kernel: [ 2.536139] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>>> 1999-12-31T20:03:21.728471-05:00 devuan-bunsen kernel: [ 2.536274] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d97554)
>>> 1999-12-31T20:03:21.728480-05:00 devuan-bunsen kernel: [ 2.536493] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d97278)
>>> 1999-12-31T20:03:21.728491-05:00 devuan-bunsen kernel: [ 2.536520] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d97bd0)
>>> ...
>>> 1999-12-31T20:03:21.729272-05:00 devuan-bunsen kernel: [ 3.493686] Console: switching to colour frame buffer device 170x48
>>> 1999-12-31T20:03:21.729282-05:00 devuan-bunsen kernel: [ 3.521747] exynos-drm exynos-drm: [drm] fb0: exynosdrmfb frame buffer device
>>> 1999-12-31T20:03:21.729292-05:00 devuan-bunsen kernel: [ 3.522831] [drm] Initialized exynos 1.1.0 20180330 for exynos-drm on minor 0
>>>
>>> The screen works normally in this case.
>>>
>>> On Xen as dom0:
>>>
>>> 1999-12-31T20:01:09.722790-05:00 devuan-bunsen kernel: [ 2.606812] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
>>> 1999-12-31T20:01:09.722795-05:00 devuan-bunsen kernel: [ 2.606884] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>>> 1999-12-31T20:01:09.722800-05:00 devuan-bunsen kernel: [ 2.606999] exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU
>>> 1999-12-31T20:01:09.722805-05:00 devuan-bunsen kernel: [ 2.607044] exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22
>>> 1999-12-31T20:01:09.722810-05:00 devuan-bunsen kernel: [ 2.607162] exynos-drm exynos-drm: adev bind failed: -22
>>> 1999-12-31T20:01:09.722815-05:00 devuan-bunsen kernel: [ 2.607183] exynos-dp: probe of 145b0000.dp-controller failed with error -22
>>>
>>> There is no display on the screen in this case. The backlight
>>> does not even come on.
>>>
>>> So the error causing no display is probably:
>>>
>>> *ERROR* Device 14450000.mixer lacks support for IOMMU
>>>
>>> I am new to arm virtualization with Xen. I understand IOMMU on x86
>>> is needed for PCI passthrough to domU guests, but not for dom0 to
>>> use such devices.
>>
>> I believe that the IOMMU would be required on x86 when using dom0 PVH.
>> PVH is very similar to an Arm guests.
>>
>> On Arm, we don't require the IOMMU because not all Arm platforms have
>> all DMA-capable devices protected by an IOMMU. So dom0 will still have
>> its memory direct mapped (i.e. host physical address = guest physical
>> address) to allow DMA in dom0 with limited modification.
>>
>> That said, I thik this is a different situation here (see below).
>>
>>
>>> So on arm, why is dom0 trying to use IOMMU for
>>> the exynos-mixer/exynos-drm when bare metal does not use it?
>>
>> Just to confirm, are you using the same kernel, same config when booting
>> on baremetal? If so, from looking at the code, I would expect that the
>> IOMMU is also used on baremetal.
>
> Hi Julien,
>
> I didn't see that you asked this. The answer is yes, it is the same
> kernel on the bare metal. I also thought from my brief look at the
> code that on bare metal the IOMMU is also used, or at least, given
> the logs for the bare metal case and the Xen case, I don't see any
> evidence this check is not made on bare metal, but apparently the
> check passes on bare metal.
>
>>
>> The check failing is:
>>
>> if (get_dma_ops(priv->dma_dev) != get_dma_ops(subdrv_dev))
>>
>> I am not quite too sure why the check implies the IOMMU is not
>> supported. That said, I vaguely recall that Linux will update the DMA
>> ops when running under Xen. Would you be able to print the two values
>> returned ("%pS" should give the symbol)?
>
> I got those values now:
>
> get_dma_ops(priv->dma_dev): 0xc0d018c0
> get_dma_ops(subdrv_dev): 0xc0d662dc

On bare metal:

get_dma_ops(priv->dma_dev): 0xc0d018c0
get_dma_ops(subdrv_dev): 0xc0d018c0

So the dma_ops of the subdrv_dev is changed by Xen, causing the check
to fail. I assume the subdrv_dev is the exynos_mixer in this case.

>
> Kind regards,
>
> Chuck
>
>>>
>>
>> Cheers,
>>
>> [1]
>> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/Documentation/devicetree/bindings/iommu/samsung%2Csysmmu.txt
>>
>
>

Hi Chuck,

On 27/10/2023 00:26, Chuck Zmudzinski wrote:
> On 10/26/2023 5:24 PM, Julien Grall wrote:
>> Anyway, letting dom0 to use the IOMMU is probably a bad idea as even if
>> dom0 memory is direct mapped, grant mappings are not. So you would end
>> up to see random crashes.
>
> I noticed some possible relevant settings from my Linux config:
>
> CONFIG_XEN_GNTDEV=m
> CONFIG_XEN_GRANT_DEV_ALLOC=m
> # CONFIG_XEN_GRANT_DMA_ALLOC is not set
>
> I also have in my dom0 config:
>
> # CONFIG_XEN_VIRTIO is not set
>
> Also, not even mentioned in my Linux config but option is available in
> Linux, but it can be enabled by enabling CONFIG_XEN_VIRTIO in the
> configuration stage of the Linux build:
>
> # CONFIG_XEN_GRANT_DMA_IOMMU is not set
>
> Is virtio a safe way to provide IOMMU support for dom0? Maybe I can
> enable this and see if it helps, but will it cause random crashes?

None of the options above matter to boot dom0. They are related to PV
functionality on both the backend and frontend side.

Cheers,


--
Julien Grall

Hi Chuck,

On 27/10/2023 04:42, Chuck Zmudzinski wrote:
> On 10/26/2023 5:24 PM, Julien Grall wrote:
>> I am not quite too sure why the check implies the IOMMU is not
>> supported. That said, I vaguely recall that Linux will update the DMA
>> ops when running under Xen. Would you be able to print the two values
>> returned ("%pS" should give the symbol)?
>
> I got those values:
>
> [ 2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 0xc0d662dc
>
> I presume you know how to interpret those. The failed test is that they are not equal.

Unfortunately the values are specific to the kernel build.

From [1], I was expecting that %pS would print something like:

%pS versatile_init+0x0/0x110

Can you use 'nm', gdb or addr2line to find out the associated the symbols?

Cheers,

[1] https://www.kernel.org/doc/Documentation/printk-formats.txt

--
Julien Grall

On 10/27/2023 9:44 AM, Julien Grall wrote:
> Hi Chuck,
>
> On 27/10/2023 04:42, Chuck Zmudzinski wrote:
>> On 10/26/2023 5:24 PM, Julien Grall wrote:
>>> I am not quite too sure why the check implies the IOMMU is not
>>> supported. That said, I vaguely recall that Linux will update the DMA
>>> ops when running under Xen. Would you be able to print the two values
>>> returned ("%pS" should give the symbol)?
>>
>> I got those values:
>>
>> [ 2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 0xc0d662dc
>>
>> I presume you know how to interpret those. The failed test is that they are not equal.
>
> Unfortunately the values are specific to the kernel build.
>
> From [1], I was expecting that %pS would print something like:
>
> %pS versatile_init+0x0/0x110
>
> Can you use 'nm', gdb or addr2line to find out the associated the symbols?

From addr2line and nm, I get this:

0xc0d018c0 is in dma_mapping.c, symbol is iommu_ops

0xc0d662dc is in swiotlb-xen.c, symbol is xen_swiotlb_dma_ops

Details:

user@linux:~/kernelbuild/linux-6.1.59$ addr2line -e vmlinux
0xc0d018c0
dma-mapping.c:?
0xc0d662dc
swiotlb-xen.c:?

user@linux:~/kernelbuild/linux-6.1.59$ nm vmlinux | grep c0d018c0
c0d018c0 d $d
c0d018c0 d iommu_ops
user@linux:~/kernelbuild/linux-6.1.59$ nm vmlinux | grep c0d662dc
c0d662dc d $d
c0d662dc D xen_swiotlb_dma_ops
user@linux:~/kernelbuild/linux-6.1.59

>
> Cheers,
>
> [1] https://www.kernel.org/doc/Documentation/printk-formats.txt
>

On 10/27/2023 10:40 AM, Chuck Zmudzinski wrote:
> On 10/27/2023 9:44 AM, Julien Grall wrote:
>> Hi Chuck,
>>
>> On 27/10/2023 04:42, Chuck Zmudzinski wrote:
>>> On 10/26/2023 5:24 PM, Julien Grall wrote:
>>>> I am not quite too sure why the check implies the IOMMU is not
>>>> supported. That said, I vaguely recall that Linux will update the DMA
>>>> ops when running under Xen. Would you be able to print the two values
>>>> returned ("%pS" should give the symbol)?
>>>
>>> I got those values:
>>>
>>> [ 2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 0xc0d662dc
>>>
>>> I presume you know how to interpret those. The failed test is that they are not equal.
>>
>> Unfortunately the values are specific to the kernel build.
>>
>> From [1], I was expecting that %pS would print something like:
>>
>> %pS versatile_init+0x0/0x110
>>
>> Can you use 'nm', gdb or addr2line to find out the associated the symbols?
>
> From addr2line and nm, I get this:
>
> 0xc0d018c0 is in dma_mapping.c, symbol is iommu_ops

Sorry, that is dma-mapping.c

>
> 0xc0d662dc is in swiotlb-xen.c, symbol is xen_swiotlb_dma_ops
>
> Details:
>
> user@linux:~/kernelbuild/linux-6.1.59$ addr2line -e vmlinux
> 0xc0d018c0
> dma-mapping.c:?
> 0xc0d662dc
> swiotlb-xen.c:?
>
> user@linux:~/kernelbuild/linux-6.1.59$ nm vmlinux | grep c0d018c0
> c0d018c0 d $d
> c0d018c0 d iommu_ops
> user@linux:~/kernelbuild/linux-6.1.59$ nm vmlinux | grep c0d662dc
> c0d662dc d $d
> c0d662dc D xen_swiotlb_dma_ops
> user@linux:~/kernelbuild/linux-6.1.59
>
>>
>> Cheers,
>>
>> [1] https://www.kernel.org/doc/Documentation/printk-formats.txt
>>
>

On 10/27/2023 9:44 AM, Julien Grall wrote:
> Hi Chuck,
>
> On 27/10/2023 04:42, Chuck Zmudzinski wrote:
>> On 10/26/2023 5:24 PM, Julien Grall wrote:
>>> I am not quite too sure why the check implies the IOMMU is not
>>> supported. That said, I vaguely recall that Linux will update the DMA
>>> ops when running under Xen. Would you be able to print the two values
>>> returned ("%pS" should give the symbol)?
>>
>> I got those values:
>>
>> [ 2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 0xc0d662dc
>>
>> I presume you know how to interpret those. The failed test is that they are not equal.
>
> Unfortunately the values are specific to the kernel build.
>
> From [1], I was expecting that %pS would print something like:
>
> %pS versatile_init+0x0/0x110
>
> Can you use 'nm', gdb or addr2line to find out the associated the symbols?

From addr2line and nm, I get this:

0xc0d018c0 is in dma-mapping.c, symbol is iommu_ops

0xc0d662dc is in swiotlb-xen.c, symbol is xen_swiotlb_dma_ops

> As a temporary hack, would you be able to compile out the IOMMU driver
> from Linux and check if it helps using the GPU?

I think I understand now why this might help. Maybe without IOMMU support
in Linux the exynos_drm device might also use xen_swiotlb_dma_ops instead
of iommu_ops so the check would succeed.

So the experiment of compiling IOMMU out of Linux should reveal if the
exynos GPU requires the IOMMU.

>
> Cheers,
>
> [1] https://www.kernel.org/doc/Documentation/printk-formats.txt
>

On 10/26/2023 5:24 PM, Julien Grall wrote:
> As a temporary hack, would you be able to compile out the IOMMU driver
> from Linux and check if it helps using the GPU?

I did a test with most IOMMU support compiled out of
Linux.

Short summary: This does help the GPU on Xen, now dom0 can print
console messages on the built-in display, but X.org did not work
on Xen. The change also caused a new problem on Xen: without IOMMU
support Wifi stopped working on Xen. Everything still works on bare
metal without the IOMMU support in Linux but probably with a
performance hit.

Detailed description of the test and results:

The make *config targets for the Linux build would not allow
allow disabling CONFIG_IOMMU_SUPPORT without also disabling
CONFIG_DRM so I also did not disable these two generic
IOMMU options because it didn't make sense to do a test of
the GPU with a kernel that has DRM disabled:

#
# Generic IOMMU Pagetable Support
#
CONFIG_IOMMU_IO_PGTABLE=y
CONFIG_IOMMU_IO_PGTABLE_LPAE=y

The tested kernel has disabled these IOMMU options:

CONFIG_EXYNOS_IOMMU and several others as shown in the following
diff compared to the kernel I used in the original report:

--- .config.old 2023-10-24 00:41:01.000000000 -0400
+++ .config 2023-10-27 13:15:57.146322091 -0400
@@ -251,8 +251,6 @@

CONFIG_ARM=y
CONFIG_ARM_HAS_GROUP_RELOCS=y
-CONFIG_ARM_DMA_USE_IOMMU=y
-CONFIG_ARM_DMA_IOMMU_ALIGNMENT=8
CONFIG_SYS_SUPPORTS_APM_EMULATION=y
CONFIG_HAVE_PROC_CPU=y
CONFIG_STACKTRACE_SUPPORT=y
@@ -5151,7 +5148,6 @@
# end of Clock Source drivers

# CONFIG_MAILBOX is not set
-CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y

#
@@ -5164,12 +5160,7 @@
# end of Generic IOMMU Pagetable Support

# CONFIG_IOMMU_DEBUGFS is not set
-CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
-# CONFIG_IOMMU_DEFAULT_DMA_LAZY is not set
-# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
-CONFIG_OF_IOMMU=y
-CONFIG_EXYNOS_IOMMU=y
-# CONFIG_EXYNOS_IOMMU_DEBUG is not set
+# CONFIG_EXYNOS_IOMMU is not set
# CONFIG_ARM_SMMU is not set

#
@@ -6558,7 +6549,6 @@
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
CONFIG_DMA_OPS=y
-CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DMA_DECLARE_COHERENT=y

# CONFIG_GREYBUS is not set
-----------------------------

Logs for case without EXYNOS_IOMMU:

With Xen:

DRM logs:

1999-12-31T20:01:02.777765-05:00 devuan-bunsen kernel: [ 2.362733] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
1999-12-31T20:01:02.777768-05:00 devuan-bunsen kernel: [ 2.362755] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d95d34)
1999-12-31T20:01:02.777771-05:00 devuan-bunsen kernel: [ 2.362894] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d96f34)
1999-12-31T20:01:02.777775-05:00 devuan-bunsen kernel: [ 2.363116] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d96c58)
1999-12-31T20:01:02.777939-05:00 devuan-bunsen kernel: [ 2.363141] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d975b0)

Wifi Logs:

1999-12-31T20:01:04.869178-05:00 devuan-bunsen kernel: [ 13.612951] mwifiex_sdio mmc2:0001:1: info: trying to associate to bssid 28:74:f5:3e:6e:88
1999-12-31T20:01:04.889185-05:00 devuan-bunsen kernel: [ 13.632338] mwifiex_sdio mmc2:0001:1: info: associated to bssid xx:xx:xx:xx:xx:xx successfully
1999-12-31T20:01:04.899132-05:00 devuan-bunsen kernel: [ 13.640107] IPv6: ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@...
1999-12-31T20:00:24.635000-05:00 devuan-bunsen kernel: [ 0.000000] Booting Linux on physical CPU 0x0

Bare metal:

DRM logs:

1999-12-31T20:01:07.261447-05:00 devuan-bunsen kernel: [ 2.359431] [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations
1999-12-31T20:01:07.261455-05:00 devuan-bunsen kernel: [ 2.359452] exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d95d34)
1999-12-31T20:01:07.261489-05:00 devuan-bunsen kernel: [ 2.359569] exynos-drm exynos-drm: bound 14450000.mixer (ops 0xc0d96f34)
1999-12-31T20:01:07.261498-05:00 devuan-bunsen kernel: [ 2.359769] exynos-drm exynos-drm: bound 145b0000.dp-controller (ops 0xc0d96c58)
1999-12-31T20:01:07.261759-05:00 devuan-bunsen kernel: [ 2.359793] exynos-drm exynos-drm: bound 14530000.hdmi (ops 0xc0d975b0)

Wifi logs:

1999-12-31T20:01:09.379505-05:00 devuan-bunsen kernel: [ 13.599578] mwifiex_sdio mmc2:0001:1: info: associated to bssid xx:xx:xx:xx:xx:xx successfully
1999-12-31T20:01:09.389524-05:00 devuan-bunsen kernel: [ 13.606470] IPv6: ADDRCONF(NETDEV_CHANGE): mlan0: link becomes ready
1999-12-31T20:01:09.389573-05:00 devuan-bunsen kernel: [ 13.611797] mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
2023-10-27T15:34:40.082622-04:00 devuan-bunsen kernel: [ 25.954828] mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
2023-10-27T15:34:40.375838-04:00 devuan-bunsen kernel: [ 26.238291] mwifiex_sdio mmc2:0001:1: CMD_RESP: cmd 0x23f error, result=0x2
2023-10-27T15:35:11.332585-04:00 devuan-bunsen kernel: [ 57.204105] mwifiex_sdio mmc2:0001:1: mlan0: already connected

Notable observations:

1. Without support for the Exynos IOMMU there is no difference
in the logs for the initialization of the exynos-drm GPU
between Xen and bare metal. Still, on Xen, Xorg did not start
but on bare metal Xorg did start, but on Xen there was an
improvement because now the system is able to display
console messages from dom0 on the primary display.

2. Notice that when Wifi works, the Chromebook learns the correct
date from ntp. It did not get the correct date on Xen because
on Xen it did not successfully connect to the network, even
though it successfully associated with the Wifi access point.

3. It appears the check that was failing in the kernel with IOMMU
support for exynos when running on Xen is skipped in this kernel
on both Xen and the bare metal. That makes sense because the
name of the function (drm_iommu_attach_device) where the failed
test occurred appears to be specific to iommu so it is not used
in a kernel without exynos IOMMU support.

4. The change in the kernel caused Wifi to not connect on Xen, and it
looks like on Xen the Wifi driver dumped some binary data into the
kernel log.

Here is what works without CONFIG_EXYNOS_IOMMU enabled:

Bare metal Xen
GPU Y Y
X.org Y N
Wifi Y N

I have not tried to adjust xorg.conf driver settings to
get X.org to work with Xen. If it does work with a different
Xorg driver I expect it would be with a performance hit.

For comparison with the original report of this problem,
here is what works with CONFIG_EXYNOS_IOMMU enabled:

Bare metal Xen
GPU Y N
X.org Y N
Wifi Y Y

Hi Chuck,

On 27/10/2023 15:40, Chuck Zmudzinski wrote:
> On 10/27/2023 9:44 AM, Julien Grall wrote:
>> Hi Chuck,
>>
>> On 27/10/2023 04:42, Chuck Zmudzinski wrote:
>>> On 10/26/2023 5:24 PM, Julien Grall wrote:
>>>> I am not quite too sure why the check implies the IOMMU is not
>>>> supported. That said, I vaguely recall that Linux will update the DMA
>>>> ops when running under Xen. Would you be able to print the two values
>>>> returned ("%pS" should give the symbol)?
>>>
>>> I got those values:
>>>
>>> [ 2.552094] [drm] dma_ops(priv->dma_dev): 0xc0d018c0, dma_ops(subdrv_dev): 0xc0d662dc
>>>
>>> I presume you know how to interpret those. The failed test is that they are not equal.
>>
>> Unfortunately the values are specific to the kernel build.
>>
>> From [1], I was expecting that %pS would print something like:
>>
>> %pS versatile_init+0x0/0x110
>>
>> Can you use 'nm', gdb or addr2line to find out the associated the symbols?
>
> From addr2line and nm, I get this:
>
> 0xc0d018c0 is in dma_mapping.c, symbol is iommu_ops
>
> 0xc0d662dc is in swiotlb-xen.c, symbol is xen_swiotlb_dma_ops

Thanks! AFAICT, xen_swiotlb_dma_ops will be set by arch_setup_dma_ops()
when Xen swiotlb is detected.

I am not sure who is setting iommu_ops given that arch_setup_dma_ops()
is always overriding the dma_ops.

IIRC xen_swiotlb_dma_ops is only necessary on Arm when you have PV
backend running. As you are still at boot, you could try to remove the
call and see if you can get the IOMMU working.

If it works, then maybe xen_setup_dma_ops() should not override dma_ops
if it is set.

Actually, I am assuming we will have the exact same problem when we
start to support stage-1 IOMMU on Xen.

Cheers,

--
Julien Grall