Mailing List Archive

On 08/16/2012 10:35 AM, Stefano Stabellini wrote:
> Add a doc to describe the Xen ARM device tree bindings
>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> CC: devicetree-discuss@lists.ozlabs.org
> CC: David Vrabel <david.vrabel@citrix.com>
> ---
> Documentation/devicetree/bindings/arm/xen.txt | 22 ++++++++++++++++++++++
> 1 files changed, 22 insertions(+), 0 deletions(-)
> create mode 100644 Documentation/devicetree/bindings/arm/xen.txt
>
> diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt
> new file mode 100644
> index 0000000..ec6d884
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/arm/xen.txt
> @@ -0,0 +1,22 @@
> +* Xen hypervisor device tree bindings
> +
> +Xen ARM virtual platforms shall have the following properties:
> +
> +- compatible:
> + compatible = "xen,xen", "xen,xen-<version>";
> + where <version> is the version of the Xen ABI of the platform.
> +
> +- reg: specifies the base physical address and size of a region in
> + memory where the grant table should be mapped to, using an
> + HYPERVISOR_memory_op hypercall.
> +
> +- interrupts: the interrupt used by Xen to inject event notifications.

You should look at ePAPR 1.1 which defines hypervisor related bindings.
While it is a PPC doc, we should reuse or extend what makes sense.

See section 7.5:

https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf

Rob

> +
> +
> +Example:
> +
> +hypervisor {
> + compatible = "xen,xen", "xen,xen-4.3";
> + reg = <0xb0000000 0x20000>;
> + interrupts = <1 15 0xf08>;
> +};
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Tue, 28 Aug 2012, Rob Herring wrote:
> On 08/16/2012 10:35 AM, Stefano Stabellini wrote:
> > Add a doc to describe the Xen ARM device tree bindings
> >
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > CC: devicetree-discuss@lists.ozlabs.org
> > CC: David Vrabel <david.vrabel@citrix.com>
> > ---
> > Documentation/devicetree/bindings/arm/xen.txt | 22 ++++++++++++++++++++++
> > 1 files changed, 22 insertions(+), 0 deletions(-)
> > create mode 100644 Documentation/devicetree/bindings/arm/xen.txt
> >
> > diff --git a/Documentation/devicetree/bindings/arm/xen.txt b/Documentation/devicetree/bindings/arm/xen.txt
> > new file mode 100644
> > index 0000000..ec6d884
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/arm/xen.txt
> > @@ -0,0 +1,22 @@
> > +* Xen hypervisor device tree bindings
> > +
> > +Xen ARM virtual platforms shall have the following properties:
> > +
> > +- compatible:
> > + compatible = "xen,xen", "xen,xen-<version>";
> > + where <version> is the version of the Xen ABI of the platform.
> > +
> > +- reg: specifies the base physical address and size of a region in
> > + memory where the grant table should be mapped to, using an
> > + HYPERVISOR_memory_op hypercall.
> > +
> > +- interrupts: the interrupt used by Xen to inject event notifications.
>
> You should look at ePAPR 1.1 which defines hypervisor related bindings.
> While it is a PPC doc, we should reuse or extend what makes sense.
>
> See section 7.5:
>
> https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf

Thanks for the link, I wasn't aware of ePAPR.

The hypervisor node defined by ePAPR is not very different from what I
am proposing. Should I try to be compatible with the hypervisor
specification above (as in compatible = "epapr,hypervisor-1.1")?
Or should I just use it as a reference for my own specification?

Personally I would rather avoid full compatibility with ePAPR.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, 12 Sep 2012, Stefano Stabellini wrote:
> On Tue, 28 Aug 2012, Rob Herring wrote:
> > You should look at ePAPR 1.1 which defines hypervisor related bindings.
> > While it is a PPC doc, we should reuse or extend what makes sense.
> >
> > See section 7.5:
> >
> > https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf
>
> Thanks for the link, I wasn't aware of ePAPR.
>
> The hypervisor node defined by ePAPR is not very different from what I
> am proposing. Should I try to be compatible with the hypervisor
> specification above (as in compatible = "epapr,hypervisor-1.1")?
> Or should I just use it as a reference for my own specification?
>
> Personally I would rather avoid full compatibility with ePAPR.

In particular reading section 7.5, these are the required properties of
the ePAPR hypervisor node:

- compatible = "epapr,hypervisor-1.1"
compared to what I am proposing, it is laking information about what
hypervisor we are talking about (xen, kvm, vmware, etc) and the version
of the ABI (xen-4.2).

- hcall-instructions
potentially interesting, but given that for Xen we are quite happy with
HVC, we are not going to add any secondary hypercall mechanisms,
therefore at the moment it would just result in a BUG if the specified
hcall instruction is != HVC. Besides if somebody else wanted to
implemented the Xen hypercall interface in a different way they could
just reimplement the hypercall wrappers, that would be easier than
trying to do it with this property.

- guest-id
we usually make a point in trying not to tell the guest its own domid
because if the VM gets migrated to a different host it acquires a new
domid, therefore it should not rely on it.

- guest-name
we could pass the guest name here, but I don't see how it could be
of any use.



On the other hand, thinking more about what Xen needs in the device
tree, I think we could improve the current spec by clarifying the
meaning of the memory region and interrupt properties we currently
require. I thought about moving them to two separate children node with
an explicit name:

---

* Xen hypervisor device tree bindings

Xen ARM virtual platforms shall have the following properties and
children nodes:

- compatible property:
compatible = "xen,xen", "xen,xen-<version>";
where <version> is the version of the Xen ABI of the platform.

- grant_table child with the following properties:
- name:
name = "grant_table";
- reg: specifies the base physical address and size of a region in
memory where the grant table should be mapped to, using an
HYPERVISOR_memory_op hypercall.

- events child with the following properties:
- name:
name = "events";
- interrupts: the interrupt used by Xen to inject event notifications.


Example:
hypervisor {
compatible = "xen,xen", "xen,xen-4.2";
#address-cells = <1>;
#size-cells = <1>;
#interrupt-cells = <3>;
ranges = <0xb0000000 0xb0000000 0x20000>;

grant_table {
name = "grant_table";
reg = <0xb0000000 0x20000>;
};

events {
name = "events";
interrupts = <1 15 0xf08>;
};
};

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 09/12/2012 01:14 PM, Stefano Stabellini wrote:
> On Wed, 12 Sep 2012, Stefano Stabellini wrote:
>> On Tue, 28 Aug 2012, Rob Herring wrote:
>>> You should look at ePAPR 1.1 which defines hypervisor related bindings.
>>> While it is a PPC doc, we should reuse or extend what makes sense.
>>>
>>> See section 7.5:
>>>
>>> https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf
>>
>> Thanks for the link, I wasn't aware of ePAPR.
>>
>> The hypervisor node defined by ePAPR is not very different from what I
>> am proposing. Should I try to be compatible with the hypervisor
>> specification above (as in compatible = "epapr,hypervisor-1.1")?
>> Or should I just use it as a reference for my own specification?
>>
>> Personally I would rather avoid full compatibility with ePAPR.
>
> In particular reading section 7.5, these are the required properties of
> the ePAPR hypervisor node:
>
> - compatible = "epapr,hypervisor-1.1"
> compared to what I am proposing, it is laking information about what
> hypervisor we are talking about (xen, kvm, vmware, etc) and the version
> of the ABI (xen-4.2).

compatible properties are often changed. If we do deviate on the rest of
the binding, then it needs to be different.

Turns out that powerpc KVM guests use "linux,kvm" and "epapr,hypervisor"
doesn't even appear in the kernel.

We also perhaps have to consider the possibility of Xen on PowerPC. Then
alignment is more important.

> - hcall-instructions
> potentially interesting, but given that for Xen we are quite happy with
> HVC, we are not going to add any secondary hypercall mechanisms,
> therefore at the moment it would just result in a BUG if the specified
> hcall instruction is != HVC. Besides if somebody else wanted to
> implemented the Xen hypercall interface in a different way they could
> just reimplement the hypercall wrappers, that would be easier than
> trying to do it with this property.

It's really about the parameters passed with the HVC. The ePAPR may be a
good model for defining this. Just grepping "hypervisor" under
arch/powerpc, it's pretty clear hypervisor support happened first and
the ePAPR came second. Hopefully we can avoid that for ARM.

> - guest-id
> we usually make a point in trying not to tell the guest its own domid
> because if the VM gets migrated to a different host it acquires a new
> domid, therefore it should not rely on it.
>
> - guest-name
> we could pass the guest name here, but I don't see how it could be
> of any use.
>

I have no issue with these being optional.

>
> On the other hand, thinking more about what Xen needs in the device
> tree, I think we could improve the current spec by clarifying the
> meaning of the memory region and interrupt properties we currently
> require. I thought about moving them to two separate children node with
> an explicit name:
>
> ---
>
> * Xen hypervisor device tree bindings
>

I really think we need to define ARM hypervisor device tree bindings
first, then Xen specific bindings as an addition to that. I worry that
the KVM folks aren't paying attention and then want something different
later on.

> Xen ARM virtual platforms shall have the following properties and
> children nodes:
>
> - compatible property:
> compatible = "xen,xen", "xen,xen-<version>";

"xen,xen" should be last as it is less specific.

> where <version> is the version of the Xen ABI of the platform.
>
> - grant_table child with the following properties:
> - name:
> name = "grant_table";

What's a grant table?

> - reg: specifies the base physical address and size of a region in
> memory where the grant table should be mapped to, using an
> HYPERVISOR_memory_op hypercall.
>
> - events child with the following properties:
> - name:
> name = "events";
> - interrupts: the interrupt used by Xen to inject event notifications.

Why a child node? Just an interrupts property alone should be fine. If
you have cases with different number of interrupts, the compatible
property can distinguish that.

Rob

>
>
> Example:
> hypervisor {
> compatible = "xen,xen", "xen,xen-4.2";
> #address-cells = <1>;
> #size-cells = <1>;
> #interrupt-cells = <3>;
> ranges = <0xb0000000 0xb0000000 0x20000>;
>
> grant_table {
> name = "grant_table";
> reg = <0xb0000000 0x20000>;
> };
>
> events {
> name = "events";
> interrupts = <1 15 0xf08>;
> };
> };
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, Sep 12, 2012 at 07:14:58PM +0100, Stefano Stabellini wrote:
> On Wed, 12 Sep 2012, Stefano Stabellini wrote:
> > On Tue, 28 Aug 2012, Rob Herring wrote:
> > > You should look at ePAPR 1.1 which defines hypervisor related bindings.
> > > While it is a PPC doc, we should reuse or extend what makes sense.
> > >
> > > See section 7.5:
> > >
> > > https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf
> >
> > Thanks for the link, I wasn't aware of ePAPR.
> >
> > The hypervisor node defined by ePAPR is not very different from what I
> > am proposing. Should I try to be compatible with the hypervisor
> > specification above (as in compatible = "epapr,hypervisor-1.1")?
> > Or should I just use it as a reference for my own specification?
> >
> > Personally I would rather avoid full compatibility with ePAPR.
>
> In particular reading section 7.5, these are the required properties of
> the ePAPR hypervisor node:
>
> - compatible = "epapr,hypervisor-1.1"
> compared to what I am proposing, it is laking information about what
> hypervisor we are talking about (xen, kvm, vmware, etc) and the version
> of the ABI (xen-4.2).
>
> - hcall-instructions
> potentially interesting, but given that for Xen we are quite happy with
> HVC, we are not going to add any secondary hypercall mechanisms,
> therefore at the moment it would just result in a BUG if the specified
> hcall instruction is != HVC. Besides if somebody else wanted to
> implemented the Xen hypercall interface in a different way they could
> just reimplement the hypercall wrappers, that would be easier than
> trying to do it with this property.

Some thoughts on this:

We decided that embedding machine instructions into the DT is a fairly
awful idea when discussing how to describe low-level debug UARTs in the
DT. I don't think it's a lot better in this case (never mind issues
like ARM versus Thumb, endianness etc.)

If we are going to attempt to describe the call interface, we should
do it symbolically, allowing the hypervisor interface code in the kernel
to choose (or, if necessary, generate) the right call wrappers.

We will have this issue for descrbing power firmware interfaces for
example: we already know that this functionality might require an SMC
or HVC instruction to call it, depending on the platform.

A hypervisor with only one call ABI could leave this to be implicit,
providing there is a version number property of similar to allow future
changes to be accommodated.


Cheers
---Dave

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, Sep 12, 2012 at 09:34:28PM -0500, Rob Herring wrote:
> On 09/12/2012 01:14 PM, Stefano Stabellini wrote:
> > On Wed, 12 Sep 2012, Stefano Stabellini wrote:
> >> On Tue, 28 Aug 2012, Rob Herring wrote:
> >>> You should look at ePAPR 1.1 which defines hypervisor related bindings.
> >>> While it is a PPC doc, we should reuse or extend what makes sense.
> >>>
> >>> See section 7.5:
> >>>
> >>> https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf
> >>
> >> Thanks for the link, I wasn't aware of ePAPR.
> >>
> >> The hypervisor node defined by ePAPR is not very different from what I
> >> am proposing. Should I try to be compatible with the hypervisor
> >> specification above (as in compatible = "epapr,hypervisor-1.1")?
> >> Or should I just use it as a reference for my own specification?
> >>
> >> Personally I would rather avoid full compatibility with ePAPR.
> >
> > In particular reading section 7.5, these are the required properties of
> > the ePAPR hypervisor node:
> >
> > - compatible = "epapr,hypervisor-1.1"
> > compared to what I am proposing, it is laking information about what
> > hypervisor we are talking about (xen, kvm, vmware, etc) and the version
> > of the ABI (xen-4.2).
>
> compatible properties are often changed. If we do deviate on the rest of
> the binding, then it needs to be different.
>
> Turns out that powerpc KVM guests use "linux,kvm" and "epapr,hypervisor"
> doesn't even appear in the kernel.
>
> We also perhaps have to consider the possibility of Xen on PowerPC. Then
> alignment is more important.
>
> > - hcall-instructions
> > potentially interesting, but given that for Xen we are quite happy with
> > HVC, we are not going to add any secondary hypercall mechanisms,
> > therefore at the moment it would just result in a BUG if the specified
> > hcall instruction is != HVC. Besides if somebody else wanted to
> > implemented the Xen hypercall interface in a different way they could
> > just reimplement the hypercall wrappers, that would be easier than
> > trying to do it with this property.
>
> It's really about the parameters passed with the HVC. The ePAPR may be a
> good model for defining this. Just grepping "hypervisor" under
> arch/powerpc, it's pretty clear hypervisor support happened first and
> the ePAPR came second. Hopefully we can avoid that for ARM.

Do you think it's feasible to standardise on some interoperable ABI for
kvm and Xen? This sounds pretty optimistic, but I'm not aware of all
the technicalities, or what possible third-party hypervisors are out
there.

If we could do it, it would be good. But I have my doubts about the
feasibility and the benefits. If different hypervisors are significantly
imcompatible, then having a common low-level ABI doesn't help all that
much.

Cheers
---Dave

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 13 Sep 2012, Dave Martin wrote:
> On Wed, Sep 12, 2012 at 07:14:58PM +0100, Stefano Stabellini wrote:
> > On Wed, 12 Sep 2012, Stefano Stabellini wrote:
> > - hcall-instructions
> > potentially interesting, but given that for Xen we are quite happy with
> > HVC, we are not going to add any secondary hypercall mechanisms,
> > therefore at the moment it would just result in a BUG if the specified
> > hcall instruction is != HVC. Besides if somebody else wanted to
> > implemented the Xen hypercall interface in a different way they could
> > just reimplement the hypercall wrappers, that would be easier than
> > trying to do it with this property.
>
> Some thoughts on this:
>
> We decided that embedding machine instructions into the DT is a fairly
> awful idea when discussing how to describe low-level debug UARTs in the
> DT. I don't think it's a lot better in this case (never mind issues
> like ARM versus Thumb, endianness etc.)
>
> If we are going to attempt to describe the call interface, we should
> do it symbolically, allowing the hypervisor interface code in the kernel
> to choose (or, if necessary, generate) the right call wrappers.
>
> We will have this issue for descrbing power firmware interfaces for
> example: we already know that this functionality might require an SMC
> or HVC instruction to call it, depending on the platform.
>
> A hypervisor with only one call ABI could leave this to be implicit,
> providing there is a version number property of similar to allow future
> changes to be accommodated.

I completely agree with Dave.
I have no problems adding a symbolic property to say "we are using hvc
with parameters on registers". I just want to avoid having actual
machine instructions (and potentially dealing with executing them) into
the DT.

Maybe we could have a "calling-convention" property that can be "xen"
or something else. When a new hypervisor vendor comes along it can
change the value of "calling-convention" to "foo".

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 13 Sep 2012, Dave Martin wrote:
> Do you think it's feasible to standardise on some interoperable ABI for
> kvm and Xen? This sounds pretty optimistic, but I'm not aware of all
> the technicalities, or what possible third-party hypervisors are out
> there.
>
> If we could do it, it would be good. But I have my doubts about the
> feasibility and the benefits. If different hypervisors are significantly
> imcompatible, then having a common low-level ABI doesn't help all that
> much.

It is not really possible because each hypervisor offers a different set
of hypercalls and has a different calling convention, so there is very
little we can do to unify them.

For example many of the current Xen hypercalls are to deal with the grant
table and event channels, that are Xen specific concepts completely
missing in KVM.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, 13 Sep 2012, Rob Herring wrote:
> On 09/12/2012 01:14 PM, Stefano Stabellini wrote:
> > On Wed, 12 Sep 2012, Stefano Stabellini wrote:
> >> On Tue, 28 Aug 2012, Rob Herring wrote:
> >>> You should look at ePAPR 1.1 which defines hypervisor related bindings.
> >>> While it is a PPC doc, we should reuse or extend what makes sense.
> >>>
> >>> See section 7.5:
> >>>
> >>> https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf
> >>
> >> Thanks for the link, I wasn't aware of ePAPR.
> >>
> >> The hypervisor node defined by ePAPR is not very different from what I
> >> am proposing. Should I try to be compatible with the hypervisor
> >> specification above (as in compatible = "epapr,hypervisor-1.1")?
> >> Or should I just use it as a reference for my own specification?
> >>
> >> Personally I would rather avoid full compatibility with ePAPR.
> >
> > In particular reading section 7.5, these are the required properties of
> > the ePAPR hypervisor node:
> >
> > - compatible = "epapr,hypervisor-1.1"
> > compared to what I am proposing, it is laking information about what
> > hypervisor we are talking about (xen, kvm, vmware, etc) and the version
> > of the ABI (xen-4.2).
>
> compatible properties are often changed. If we do deviate on the rest of
> the binding, then it needs to be different.
>
> Turns out that powerpc KVM guests use "linux,kvm" and "epapr,hypervisor"
> doesn't even appear in the kernel.
>
> We also perhaps have to consider the possibility of Xen on PowerPC. Then
> alignment is more important.

OK. In that case I think that we can just use "xen,xen-4.2".


> > - hcall-instructions
> > potentially interesting, but given that for Xen we are quite happy with
> > HVC, we are not going to add any secondary hypercall mechanisms,
> > therefore at the moment it would just result in a BUG if the specified
> > hcall instruction is != HVC. Besides if somebody else wanted to
> > implemented the Xen hypercall interface in a different way they could
> > just reimplement the hypercall wrappers, that would be easier than
> > trying to do it with this property.
>
> It's really about the parameters passed with the HVC. The ePAPR may be a
> good model for defining this. Just grepping "hypervisor" under
> arch/powerpc, it's pretty clear hypervisor support happened first and
> the ePAPR came second. Hopefully we can avoid that for ARM.

Right. As I wrote in the other email, we could have a new property to
select the calling convention (and therefore the hypercall wrappers) to
be used with the hypervisor.


> > - guest-id
> > we usually make a point in trying not to tell the guest its own domid
> > because if the VM gets migrated to a different host it acquires a new
> > domid, therefore it should not rely on it.
> >
> > - guest-name
> > we could pass the guest name here, but I don't see how it could be
> > of any use.
> >
>
> I have no issue with these being optional.

OK, good.


> > On the other hand, thinking more about what Xen needs in the device
> > tree, I think we could improve the current spec by clarifying the
> > meaning of the memory region and interrupt properties we currently
> > require. I thought about moving them to two separate children node with
> > an explicit name:
> >
> > ---
> >
> > * Xen hypervisor device tree bindings
> >
>
> I really think we need to define ARM hypervisor device tree bindings
> first, then Xen specific bindings as an addition to that. I worry that
> the KVM folks aren't paying attention and then want something different
> later on.

The problem is that there isn't much in common between Xen and KVM at
least from the DT point of view. I am not sure what would go into this
common hypervisor node.
The three key pieces of information that we are currently passing in the
DT (xen-4.2, a memory region, a PPI) are Xen specific.

If one day KVM (or another hypervisor vendor) decides to support the Xen
interface, can't they just have the Xen specific binding with a slightly
different compatible node?
For example:

compatible = "linux,kvm", "xen,xen-4.2"

wouldn't that mean "I am KVM but I can support the Xen interface"?


> > Xen ARM virtual platforms shall have the following properties and
> > children nodes:
> >
> > - compatible property:
> > compatible = "xen,xen", "xen,xen-<version>";
>
> "xen,xen" should be last as it is less specific.

Ah OK, thanks.


> > where <version> is the version of the Xen ABI of the platform.
> >
> > - grant_table child with the following properties:
> > - name:
> > name = "grant_table";
>
> What's a grant table?

A Xen specific mechanism to share pages between guests.


> > - reg: specifies the base physical address and size of a region in
> > memory where the grant table should be mapped to, using an
> > HYPERVISOR_memory_op hypercall.
> >
> > - events child with the following properties:
> > - name:
> > name = "events";
> > - interrupts: the interrupt used by Xen to inject event notifications.
>
> Why a child node? Just an interrupts property alone should be fine. If
> you have cases with different number of interrupts, the compatible
> property can distinguish that.

I see, that's a good point. In that case maybe it doesn't make sense to
move the memory region and the interrupt into two different children
nodes after all. I should probably leave the spec as it was.


> >
> >
> > Example:
> > hypervisor {
> > compatible = "xen,xen", "xen,xen-4.2";
> > #address-cells = <1>;
> > #size-cells = <1>;
> > #interrupt-cells = <3>;
> > ranges = <0xb0000000 0xb0000000 0x20000>;
> >
> > grant_table {
> > name = "grant_table";
> > reg = <0xb0000000 0x20000>;
> > };
> >
> > events {
> > name = "events";
> > interrupts = <1 15 0xf08>;
> > };
> > };
> >
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel