Mailing List Archive

Armv8A - use of Xen and hypervisor EL2 translation tables to extend bridge address span
Hello,

I have a very specific challenge that I'm looking to solve, and I think
that a Hypervisor such as Xen, potentially with some modifications may
help me get there.

In short, I have a ARM 64-bit v8A device (Cortex-A53) that has a bridge
to custom logic with a limited address span, and I'm looking to address
more range than this brige has capacity for in a way that is transparent
to the operating system.  In the hardware, I have a windowed controller
that I can control at run-time to expose a subset of the address space
at any one time, but never the full range.

My thoughts are that in the OS, I could create a device tree to
instantiate a bridge to the address space that has range for the full
capacity I'm looking for at an unused portion of the IPA address map,
and the use the hypervisor to trap page faults to VTTBR0_EL2-mapped
pages, get the hypervisor to steer the windowed controller (if it is not
already pointing to the correct window being addressed) and then clear
the page fault, and allow the OS to continue.  In this way, the OS would
be none-the-wiser to the existance of the windowed controller, and
software/drivers mapping to this bridge would use the interface
transparently (albeit with a performance hit).

Perhaps I've not explained that too well.  It's quite a specific
challenge, that I would assume would require some customization of Xen. 
Has anything similar to this been done before, or can anyone point me in
the direction of if this is feasible, and where to research next?

Appreciate the help.

regards

Kris
Re: Armv8A - use of Xen and hypervisor EL2 translation tables to extend bridge address span [ In reply to ]
Hi Chris,

> On 17 Jan 2021, at 16:15, Kris Chaplin <kris.chaplin@linux.intel.com> wrote:
>
> Hello,
>
> I have a very specific challenge that I'm looking to solve, and I think that a Hypervisor such as Xen, potentially with some modifications may help me get there.
>
> In short, I have a ARM 64-bit v8A device (Cortex-A53) that has a bridge to custom logic with a limited address span, and I'm looking to address more range than this brige has capacity for in a way that is transparent to the operating system. In the hardware, I have a windowed controller that I can control at run-time to expose a subset of the address space at any one time, but never the full range.
>
> My thoughts are that in the OS, I could create a device tree to instantiate a bridge to the address space that has range for the full capacity I'm looking for at an unused portion of the IPA address map, and the use the hypervisor to trap page faults to VTTBR0_EL2-mapped pages, get the hypervisor to steer the windowed controller (if it is not already pointing to the correct window being addressed) and then clear the page fault, and allow the OS to continue. In this way, the OS would be none-the-wiser to the existance of the windowed controller, and software/drivers mapping to this bridge would use the interface transparently (albeit with a performance hit).

From the theory this is possible for sure.

>
> Perhaps I've not explained that too well. It's quite a specific challenge, that I would assume would require some customization of Xen. Has anything similar to this been done before, or can anyone point me in the direction of if this is feasible, and where to research next?

You will need to implement a driver in Xen to do that and catch some accesses and play with mappings inside Xen.
There is not something doing exactly that in Xen but you could look at what the vPCI driver for Arm under development right now is doing:
https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough

In this the vpci is catching accesses from guest to the pci configuration space and is emulating them.

You might find some other examples of parts of Xen doing things like that in other places (like the GIC driver emulating accesses to part of the interrupt controller).

Hope that can help you start even though it does not fully do exactly the stuff you want.

Cheers
Bertrand

>
> Appreciate the help.
>
> regards
>
> Kris
>
>
Re: Armv8A - use of Xen and hypervisor EL2 translation tables to extend bridge address span [ In reply to ]
Hello Bertrand


> You will need to implement a driver in Xen to do that and catch some accesses and play with mappings inside Xen.
> There is not something doing exactly that in Xen but you could look at what the vPCI driver for Arm under development right now is doing:
> https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough
>
> In this the vpci is catching accesses from guest to the pci configuration space and is emulating them.
>
> You might find some other examples of parts of Xen doing things like that in other places (like the GIC driver emulating accesses to part of the interrupt controller).
>
> Hope that can help you start even though it does not fully do exactly the stuff you want.
>
> Cheers
> Bertrand

Brilliant - thank you.  I'll look into this.  Appreciate the pointers.


Regards

Kris