Mailing List Archive

(+Oleksandr)

On 29/09/2022 18:06, stsp wrote:
> Hi!
>
> I am trying to run xen on an rcar H3
> StarterKit (r8a77951 ES2.0).
> First I created a working linux setup.
> It uses latest linux git code and is
> working well. >
> Then I built xen, also from git,
> at rev 211d8419ef.
> Now when runnign xen, I get this:
>
> ---
>
> (XEN) ipmmu: /soc/iommu@e6740000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@e7740000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@e6570000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@ff8b0000: Failed to init IPMMU (-11)
> (XEN) ipmmu: P2M sharing is not supported in current SoC revision
> (XEN) ipmmu: /soc/iommu@e67b0000: Failed to init IPMMU (-19)
> (XEN) ipmmu: /soc/iommu@ec670000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@fd800000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@ffc80000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@fe6b0000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@fe6f0000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@febd0000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@fe990000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@ec680000: Failed to init IPMMU (-11)
> (XEN) ipmmu: /soc/iommu@e7730000: Failed to init IPMMU (-11)
> (XEN) I/O virtualisation disabled
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Couldn't configure correctly all the IOMMUs.
> (XEN) ****************************************

Adding Oleksandr who wrote the driver and also worked with the Renesas SOCs.

Cheers,

> ---
>
> So I added the "iommu=0" option.
> Now linux kernel starts but hangs
> the entire system somewhere
> around the first sdcard transfer.
> I see xen is saying this on load:
>
> ---
>
> (XEN) Loading d0 kernel from boot module @ 000000007c936000
> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
> (XEN) BANK[0] 0x00000050000000-0x00000070000000 (512MB)
>
> ---
>
> If it does 1:1 mappings then perhaps
> it is supposed to work w/o IOMMU?
> But why would it then insist on me
> adding "iommu=0"?
> And there is a hang and I don't know
> if it is related to IOMMU or not...
>
> Any suggestions?
> Is rcar H3 board known to work well
> with xen?
>
>

--
Julien Grall

On 08.10.22 22:14, Julien Grall wrote:
> (+Oleksandr)


Hello all


>
> On 29/09/2022 18:06, stsp wrote:
>> Hi!
>>
>> I am trying to run xen on an rcar H3
>> StarterKit (r8a77951 ES2.0).
>> First I created a working linux setup.
>> It uses latest linux git code and is
>> working well. >
>> Then I built xen, also from git,
>> at rev 211d8419ef.


I didn’t find the 211d8419ef revision in vanilla Xen.


>> Now when runnign xen, I get this:
>>
>> ---
>>
>> (XEN) ipmmu: /soc/iommu@e6740000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@e7740000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@e6570000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@ff8b0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: P2M sharing is not supported in current SoC revision
>> (XEN) ipmmu: /soc/iommu@e67b0000: Failed to init IPMMU (-19)
>> (XEN) ipmmu: /soc/iommu@ec670000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fd800000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@ffc80000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fe6b0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fe6f0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@febd0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fe990000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@ec680000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@e7730000: Failed to init IPMMU (-11)
>> (XEN) I/O virtualisation disabled
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Couldn't configure correctly all the IOMMUs.
>> (XEN) ****************************************
>
> Adding Oleksandr who wrote the driver and also worked with the Renesas
> SOCs.
>
> Cheers,
>
>> ---
>>
>> So I added the "iommu=0" option.
>> Now linux kernel starts but hangs
>> the entire system somewhere
>> around the first sdcard transfer.
>> I see xen is saying this on load:
>>
>> ---
>>
>> (XEN) Loading d0 kernel from boot module @ 000000007c936000
>> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
>> (XEN) BANK[0] 0x00000050000000-0x00000070000000 (512MB)
>>
>> ---
>>
>> If it does 1:1 mappings then perhaps
>> it is supposed to work w/o IOMMU?


Currently Dom0 on Arm is 1:1 mapped, so yes, DMA devices in Dom0 can
work without IOMMU being enabled in Xen.


>> But why would it then insist on me
>> adding "iommu=0"?


Good question. Well, the IOMMUs initialization failed because that SoCs
revision (r8a77951 ES2.0) does not support P2M sharing so cannot be used
(and this is reported by the driver).
I assume, Xen was built with CONFIG_IPMMU_VMSA option explicitly enabled
(this option is disabled by default) although config IPMMU_VMSA mentions
the following:

      "Say Y here if you are using newest R-Car Gen3 SoCs revisions
      (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports
stage 2
      translation table format and is able to use CPU's P2M table as is."

So, the CONFIG_IPMMU_VMSA just shouldn’t be enabled if target H/W
doesn’t match.

Or, if we indeed want/need to relax the behavior (do not panic and
continue to operate with I/O virtualization disabled) if such a case happens
(I mean when driver honestly reports it cannot work due to objective
reason(s)), the code should be updated (fixed?) a bit. I will let Julien
to clarify, how he thinks it would be more correct.

From the log you provided we see the following:
- Driver for all cache IPMMUs returns -EAGAIN because root IPMMU has
been registered yet.
- Driver for root IPMMU returns -ENODEV because it detects incompatible
SoC revision.

The Arm’s iommu_hardware_setup() after scanning all IOMMU devices and
having num_iommus == 0 makes a decision to return -EAGAIN instead of
-ENODEV as the deferred list is not empty.
And any errors except -ENODEV is treated by the upper layer as a reason
to panic (see start_xen()). This is what you are facing.
With the attached patch Xen can continue to boot in such particular case
with IOMMU disabled. You may want to give it a try.


>> And there is a hang and I don't know
>> if it is related to IOMMU or not...


I don’t think it is related to IOMMU, the IOMMU is neither used by Xen
nor by Dom0 (we do not insert the IOMMU nodes into Dom0’s device tree).


>>
>> Any suggestions?


As you mentioned that "entire system hangs" I assume that Xen "hangs" as
well (not only the Dom0), so the one thing which comes to mind is to
re-check whether the "clk_ignore_unused" property is indeed passed via
xen,dom0-bootargs to Linux.
Otherwise, the SCIF clock which supplies UART H/W used for Xen console
will be disabled by Linux as unused, so the Xen console won't be
functional afterwards and that may create the impression that system hangs.

Also you may want to look at the reference (which I wrote last year) and
adopt it for the StarterKit:
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Salvator-XS


>> Is rcar H3 board known to work well
>> with xen?


Yes. Everything needed for R-Car Gen3 is already in upstream Xen.
Note that using old SoC revision you won’t be able to get DMA devices
working in non 1:1 mapped DomUs.


--
Regards,

Oleksandr Tyshchenko

Hello,

12.10.2022 02:07, Oleksandr Tyshchenko ?????:
> I didn’t find the 211d8419ef revision in vanilla Xen.

Here it is:
https://github.com/xen-project/xen/commit/211d8419ef8d8a237ff914fd8304b8fefc3ff2cc
Seems to have a failed CI.


>>> But why would it then insist on me
>>> adding "iommu=0"?
>
> Good question. Well, the IOMMUs initialization failed because that SoCs
> revision (r8a77951 ES2.0) does not support P2M sharing so cannot be used
> (and this is reported by the driver).
> I assume, Xen was built with CONFIG_IPMMU_VMSA option explicitly enabled
> (this option is disabled by default)

True but it is selected by CONFIG_RCAR3.


> although config IPMMU_VMSA mentions
> the following:
>
>       "Say Y here if you are using newest R-Car Gen3 SoCs revisions
>       (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports
> stage 2
>       translation table format and is able to use CPU's P2M table as is."
>
> So, the CONFIG_IPMMU_VMSA just shouldn’t be enabled if target H/W
> doesn’t match.

platforms/Kconfig should be fixed
then. It forces that option for rcar3.

> Or, if we indeed want/need to relax the behavior (do not panic and
> continue to operate with I/O virtualization disabled) if such a case happens
> (I mean when driver honestly reports it cannot work due to objective
> reason(s)), the code should be updated (fixed?) a bit.

I realize that silently disabling IOMMU
even with 1:1 mapping, may lead to the
security problems, so I won't add any
wishes here. But what always helps is
the verbose error messages. You could
suggest in an error msg to explicitly set
iommu=0 in the config, accept the risk
and continue.

By the way, do you really need the
level2 translation even for identity
mapping?

>>> Any suggestions?
>
> As you mentioned that "entire system hangs" I assume that Xen "hangs" as
> well (not only the Dom0), so the one thing which comes to mind is to
> re-check whether the "clk_ignore_unused" property is indeed passed via
> xen,dom0-bootargs to Linux.

It probably wasn't.
At least not in the cfg file I created.
If xen adds that to "chosen" node
automatically then perhaps...

> Otherwise, the SCIF clock which supplies UART H/W used for Xen console
> will be disabled by Linux as unused, so the Xen console won't be
> functional afterwards and that may create the impression that system hangs.

This is likely to be the case.
When I disabled the rcar3 support
in linux kernel, I got lots of errors
about missing clocks, but no hang.
So it might be that.
But unfortunately your reply came
too late (almost a month late), I
already swapped the board...
I won't be helpful in testing any
patches or suggestions, unfortunately. :(