Mailing List Archive

Xen-API update
Here's a quick update on the Xen-API work; I thought that with the large
number of changesets that have gone in today that you might be curious ;-)

We had an internal review of the API last week here at XenSource, and what
you've seen today is the result of that. The goal is to have an interface in
Xen 3.0.5 that we will call the Xen-API 1.0, and that we will maintain in a
backwards-compatible fashion for the long term. With that in mind, some of
our "future hopes" that were still unimplemented have been bumped out, and
where we felt that we needed more flexibility, we've made provision for that
too. I think that API that we've settled on is in good shape, and is
certainly something that we can stand behind, and maintain going forward.

I've still got a few more changes to push through over the next day or two,
and when that's done I'll be marking this API version 0.9, to emphasise the
approaching deadline.

If you've not already done so, now is the time to try out what we've got.
This is the _only_ API for Xen that will be maintained in the long term, so if
there's something that you need for your product or your pet project, now is
the time to shout! We have both Python and C client-side bindings in the
tree, and a number of example scripts to get you started, and I'd be very
interested in your feedback.

For those who've been following along at home, here's a summary of the recent
changes and those coming up in the next couple of days:

o Improved console support: the Console class has gained additional fields
to allow VNC passwords, the X DISPLAY for SDL, and similar configuration to
be passed through, both to QEMU and to the paravirtual frame buffer driver.

o Extended VCPU scheduling and pinning parameters, bringing the Xen-API up
to parity with the legacy protocols.

o Metrics classes: I've split the I/O and memory metrics on VM, host, PIF,
VIF, and VBD out into separate classes from the main data class. In other
words, there is now a host_metrics class as well as the host class. These
statistics have very different access patterns and rates of change,
when compared with the object to which they apply, so it makes to have them
held in a separate object.

o The asynchronous method calls are now supported by Xend. For example,
you may call Async.VM.start which will return immediately with a task handle,
and then you may poll the status of that request asynchronously until it
completes. These calls are not yet in the C bindings (coming soon!) but you
can make these calls using the Python bindings (or xm shell) today.

o True asynchronous event support is on its way, with a simple registration
/ blocking-poll mechanism for notifications.

o VTPM handling has been tidied up (thanks to Stefan Berger for that one)

o Improved storage handling has been added with a new PBD class. This will
give Xend room for configuration details related to storage repositories, and
extends the model to show symmetry between network and storage handling.

o The presence and location of crash-dumps and suspend images may be
interrogated through the API, using the same VDI API as for VM disk images.


The plan for the next couple of days is to finish up getting these changes
into xen-unstable, and then look to flesh out xm and the test programs. At
the same time, there will be a little bit of work to the Xen-CIM providers to
match these recent changes, and then we'll be in an excellent position to
stabilise Xend and the CIM providers for the 3.0.5 release.

All the best,

Ewan.

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Xen-API update [ In reply to ]
This is excellent news.

I'm doing my master thesis this semester about (in short) management and
monitoring of a virtual infrastructure. In this project I also plan to
develop a tool which will calculate the total redundancy level of the VI
and, among other things, also be able to live migrate VMs to give them
more available resources if need be. It is also meant for this tool to
be able to communicate with hartbeat to ensure availability in the
virtual infrastructure if physical nodes do, in worst case, go down
unexpectantly.

I will need to collect statistics of the resource consumption in the
different VMs and the new Xen-api seems like the best approach to do
this. I have been recommended to look at xenmon also for this task and I
would very much like more opinions on this matter. I am aware that I
have the option of enabling statistics outpit from Xen, but also that
this might cause a performance hit on about 10% which is unacceptable
since this tool is ment to be used in production systems.

I hope to hear about experiences and opinions from other people on this
list that might have been in the same situation or had a similar task,
or just opinions and advice in general.

Ingard
Oslo University College
Norway

Ewan Mellor wrote:
> Here's a quick update on the Xen-API work; I thought that with the large
> number of changesets that have gone in today that you might be curious ;-)
>
> We had an internal review of the API last week here at XenSource, and what
> you've seen today is the result of that. The goal is to have an interface in
> Xen 3.0.5 that we will call the Xen-API 1.0, and that we will maintain in a
> backwards-compatible fashion for the long term. With that in mind, some of
> our "future hopes" that were still unimplemented have been bumped out, and
> where we felt that we needed more flexibility, we've made provision for that
> too. I think that API that we've settled on is in good shape, and is
> certainly something that we can stand behind, and maintain going forward.
>
> I've still got a few more changes to push through over the next day or two,
> and when that's done I'll be marking this API version 0.9, to emphasise the
> approaching deadline.
>
> If you've not already done so, now is the time to try out what we've got.
> This is the _only_ API for Xen that will be maintained in the long term, so if
> there's something that you need for your product or your pet project, now is
> the time to shout! We have both Python and C client-side bindings in the
> tree, and a number of example scripts to get you started, and I'd be very
> interested in your feedback.
>
> For those who've been following along at home, here's a summary of the recent
> changes and those coming up in the next couple of days:
>
> o Improved console support: the Console class has gained additional fields
> to allow VNC passwords, the X DISPLAY for SDL, and similar configuration to
> be passed through, both to QEMU and to the paravirtual frame buffer driver.
>
> o Extended VCPU scheduling and pinning parameters, bringing the Xen-API up
> to parity with the legacy protocols.
>
> o Metrics classes: I've split the I/O and memory metrics on VM, host, PIF,
> VIF, and VBD out into separate classes from the main data class. In other
> words, there is now a host_metrics class as well as the host class. These
> statistics have very different access patterns and rates of change,
> when compared with the object to which they apply, so it makes to have them
> held in a separate object.
>
> o The asynchronous method calls are now supported by Xend. For example,
> you may call Async.VM.start which will return immediately with a task handle,
> and then you may poll the status of that request asynchronously until it
> completes. These calls are not yet in the C bindings (coming soon!) but you
> can make these calls using the Python bindings (or xm shell) today.
>
> o True asynchronous event support is on its way, with a simple registration
> / blocking-poll mechanism for notifications.
>
> o VTPM handling has been tidied up (thanks to Stefan Berger for that one)
>
> o Improved storage handling has been added with a new PBD class. This will
> give Xend room for configuration details related to storage repositories, and
> extends the model to show symmetry between network and storage handling.
>
> o The presence and location of crash-dumps and suspend images may be
> interrogated through the API, using the same VDI API as for VM disk images.
>
>
> The plan for the next couple of days is to finish up getting these changes
> into xen-unstable, and then look to flesh out xm and the test programs. At
> the same time, there will be a little bit of work to the Xen-CIM providers to
> match these recent changes, and then we'll be in an excellent position to
> stabilise Xend and the CIM providers for the 3.0.5 release.
>
> All the best,
>
> Ewan.
>
> _______________________________________________
> xen-api mailing list
> xen-api@lists.xensource.com
> http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
>


_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Xen-API update [ In reply to ]
On Tue, Jan 30, 2007 at 11:25:04PM +0000, Ewan Mellor wrote:

> I've still got a few more changes to push through over the next day or two,
> and when that's done I'll be marking this API version 0.9, to emphasise the
> approaching deadline.

Is the specification up to date? Is it complete?

thanks,
john

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Xen-API update [ In reply to ]
I just noticed there is no vm_metrics class available from xendapi.py
which makes it (as far as i understood) inacessible from the api. Is
this work in progress? If so, when is it planned to be included in the
unstable tree?

I saw the documentation is in place already so that is why im asking. I
am pretty new to xen so i might have done something wrong when mirroring
the tree. I used the following command:
"hg clone http://xenbits.xensource.com/xen-unstable.hg"

Is this the correct thing to do to get the most updated copy?

Ingard

Ewan Mellor wrote:
> Here's a quick update on the Xen-API work; I thought that with the large
> number of changesets that have gone in today that you might be curious ;-)
>
> We had an internal review of the API last week here at XenSource, and what
> you've seen today is the result of that. The goal is to have an interface in
> Xen 3.0.5 that we will call the Xen-API 1.0, and that we will maintain in a
> backwards-compatible fashion for the long term. With that in mind, some of
> our "future hopes" that were still unimplemented have been bumped out, and
> where we felt that we needed more flexibility, we've made provision for that
> too. I think that API that we've settled on is in good shape, and is
> certainly something that we can stand behind, and maintain going forward.
>
> I've still got a few more changes to push through over the next day or two,
> and when that's done I'll be marking this API version 0.9, to emphasise the
> approaching deadline.
>
> If you've not already done so, now is the time to try out what we've got.
> This is the _only_ API for Xen that will be maintained in the long term, so if
> there's something that you need for your product or your pet project, now is
> the time to shout! We have both Python and C client-side bindings in the
> tree, and a number of example scripts to get you started, and I'd be very
> interested in your feedback.
>
> For those who've been following along at home, here's a summary of the recent
> changes and those coming up in the next couple of days:
>
> o Improved console support: the Console class has gained additional fields
> to allow VNC passwords, the X DISPLAY for SDL, and similar configuration to
> be passed through, both to QEMU and to the paravirtual frame buffer driver.
>
> o Extended VCPU scheduling and pinning parameters, bringing the Xen-API up
> to parity with the legacy protocols.
>
> o Metrics classes: I've split the I/O and memory metrics on VM, host, PIF,
> VIF, and VBD out into separate classes from the main data class. In other
> words, there is now a host_metrics class as well as the host class. These
> statistics have very different access patterns and rates of change,
> when compared with the object to which they apply, so it makes to have them
> held in a separate object.
>
> o The asynchronous method calls are now supported by Xend. For example,
> you may call Async.VM.start which will return immediately with a task handle,
> and then you may poll the status of that request asynchronously until it
> completes. These calls are not yet in the C bindings (coming soon!) but you
> can make these calls using the Python bindings (or xm shell) today.
>
> o True asynchronous event support is on its way, with a simple registration
> / blocking-poll mechanism for notifications.
>
> o VTPM handling has been tidied up (thanks to Stefan Berger for that one)
>
> o Improved storage handling has been added with a new PBD class. This will
> give Xend room for configuration details related to storage repositories, and
> extends the model to show symmetry between network and storage handling.
>
> o The presence and location of crash-dumps and suspend images may be
> interrogated through the API, using the same VDI API as for VM disk images.
>
>
> The plan for the next couple of days is to finish up getting these changes
> into xen-unstable, and then look to flesh out xm and the test programs. At
> the same time, there will be a little bit of work to the Xen-CIM providers to
> match these recent changes, and then we'll be in an excellent position to
> stabilise Xend and the CIM providers for the 3.0.5 release.
>
> All the best,
>
> Ewan.
>
> _______________________________________________
> xen-api mailing list
> xen-api@lists.xensource.com
> http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
>


_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Xen-API update [ In reply to ]
On Fri, Feb 02, 2007 at 03:22:05PM +0100, Ingard Mevåg wrote:

> I just noticed there is no vm_metrics class available from xendapi.py
> which makes it (as far as i understood) inacessible from the api. Is
> this work in progress? If so, when is it planned to be included in the
> unstable tree?
>
> I saw the documentation is in place already so that is why im asking. I
> am pretty new to xen so i might have done something wrong when mirroring
> the tree. I used the following command:
> "hg clone http://xenbits.xensource.com/xen-unstable.hg"
>
> Is this the correct thing to do to get the most updated copy?

Yes, you've done the right thing.

The VM_metrics class is planned for implementation next week, and I'll also
put together some examples and docs (as you asked for the other day).

Cheers,

Ewan.

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Xen-API update [ In reply to ]
On Thu, Feb 01, 2007 at 04:38:01AM +0000, John Levon wrote:

> On Tue, Jan 30, 2007 at 11:25:04PM +0000, Ewan Mellor wrote:
>
> > I've still got a few more changes to push through over the next day or two,
> > and when that's done I'll be marking this API version 0.9, to emphasise the
> > approaching deadline.
>
> Is the specification up to date? Is it complete?

What's in xen-unstable.hg/docs/xen-api is as up-to-date as it gets, which is
fairly close. I've still got a few more features to put in -- I'll send
another announcement to the list and update the version number when I think
it's "complete".

Cheers,

Ewan.

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Xen-API update [ In reply to ]
Ewan Mellor wrote:
> On Fri, Feb 02, 2007 at 03:22:05PM +0100, Ingard Mevåg wrote:
>
>
>> I just noticed there is no vm_metrics class available from xendapi.py
>> which makes it (as far as i understood) inacessible from the api. Is
>> this work in progress? If so, when is it planned to be included in the
>> unstable tree?
>>
>> I saw the documentation is in place already so that is why im asking. I
>> am pretty new to xen so i might have done something wrong when mirroring
>> the tree. I used the following command:
>> "hg clone http://xenbits.xensource.com/xen-unstable.hg"
>>
>> Is this the correct thing to do to get the most updated copy?
>>
>
> Yes, you've done the right thing.
>
> The VM_metrics class is planned for implementation next week, and I'll also
> put together some examples and docs (as you asked for the other day).
>
> Cheers,
>
> Ewan.
>
Great news :)

I manged to find out most of it by trial and error and reading the
source code :)

Just a quick q, will the following methods, which i have listed below,
be able to give me the same metrics as xentop does?
And also, is the NETTX and NETRX values that is shows in xentop the same
as PIF_metrics i/o or am I missing something?

For each host:
host_metrics:
get_memory_total
get_memory_free

host_cpu:
get_utilisation

PIF_metrics:
get_io_read_kbs
get_io_write_kbs

For each VM:
vm_metrics:
get_VCPUs_utilisation
get_memory_actual

VIF:
get_io_read_kbs
get_io_write_kbs

VBD:
get_io_read_kbs
get_io_write_kbs

thx
ingard

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: [Xen-cim] Xen-API update [ In reply to ]
Ewan Mellor wrote:
> Here's a quick update on the Xen-API work; I thought that with the large
> number of changesets that have gone in today that you might be curious ;-)
>

Great progress - thanks for all the work :-).

> We had an internal review of the API last week here at XenSource, and what
> you've seen today is the result of that. The goal is to have an interface in
> Xen 3.0.5 that we will call the Xen-API 1.0, and that we will maintain in a
> backwards-compatible fashion for the long term. With that in mind, some of
> our "future hopes" that were still unimplemented have been bumped out, and
> where we felt that we needed more flexibility, we've made provision for that
> too. I think that API that we've settled on is in good shape, and is
> certainly something that we can stand behind, and maintain going forward.
>
> I've still got a few more changes to push through over the next day or two,
> and when that's done I'll be marking this API version 0.9, to emphasise the
> approaching deadline.
>
> If you've not already done so, now is the time to try out what we've got.
> This is the _only_ API for Xen that will be maintained in the long term, so if
> there's something that you need for your product or your pet project, now is
> the time to shout! We have both Python and C client-side bindings in the
> tree, and a number of example scripts to get you started, and I'd be very
> interested in your feedback.
>

The host class could use a capabilities field, perhaps a set of strings
describing supported guests - similar to 'xen_caps' field produced by
'xm info'.

We've talked in the past about changing config of running guest vs
changing its stored config. I would like to hotplug memory, cpu, disks,
etc on a running guest but have it use stored config on next activation.
Likewise it would be useful to change the stored config, regardless of
guest state, for application on next activation.

Regards,
Jim


> For those who've been following along at home, here's a summary of the recent
> changes and those coming up in the next couple of days:
>
> o Improved console support: the Console class has gained additional fields
> to allow VNC passwords, the X DISPLAY for SDL, and similar configuration to
> be passed through, both to QEMU and to the paravirtual frame buffer driver.
>
> o Extended VCPU scheduling and pinning parameters, bringing the Xen-API up
> to parity with the legacy protocols.
>
> o Metrics classes: I've split the I/O and memory metrics on VM, host, PIF,
> VIF, and VBD out into separate classes from the main data class. In other
> words, there is now a host_metrics class as well as the host class. These
> statistics have very different access patterns and rates of change,
> when compared with the object to which they apply, so it makes to have them
> held in a separate object.
>
> o The asynchronous method calls are now supported by Xend. For example,
> you may call Async.VM.start which will return immediately with a task handle,
> and then you may poll the status of that request asynchronously until it
> completes. These calls are not yet in the C bindings (coming soon!) but you
> can make these calls using the Python bindings (or xm shell) today.
>
> o True asynchronous event support is on its way, with a simple registration
> / blocking-poll mechanism for notifications.
>
> o VTPM handling has been tidied up (thanks to Stefan Berger for that one)
>
> o Improved storage handling has been added with a new PBD class. This will
> give Xend room for configuration details related to storage repositories, and
> extends the model to show symmetry between network and storage handling.
>
> o The presence and location of crash-dumps and suspend images may be
> interrogated through the API, using the same VDI API as for VM disk images.
>
>
> The plan for the next couple of days is to finish up getting these changes
> into xen-unstable, and then look to flesh out xm and the test programs. At
> the same time, there will be a little bit of work to the Xen-CIM providers to
> match these recent changes, and then we'll be in an excellent position to
> stabilise Xend and the CIM providers for the 3.0.5 release.
>
> All the best,
>
> Ewan.
>
> _______________________________________________
> Xen-cim mailing list
> Xen-cim@lists.xensource.com
> http://lists.xensource.com/xen-cim
>

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Re: [Xen-cim] Xen-API update [ In reply to ]
On Fri, Feb 02, 2007 at 10:26:23AM -0700, Jim Fehlig wrote:
> Ewan Mellor wrote:
> > Here's a quick update on the Xen-API work; I thought that with the large
> > number of changesets that have gone in today that you might be curious ;-)
> >
>
> Great progress - thanks for all the work :-).
>
> > We had an internal review of the API last week here at XenSource, and what
> > you've seen today is the result of that. The goal is to have an interface in
> > Xen 3.0.5 that we will call the Xen-API 1.0, and that we will maintain in a
> > backwards-compatible fashion for the long term. With that in mind, some of
> > our "future hopes" that were still unimplemented have been bumped out, and
> > where we felt that we needed more flexibility, we've made provision for that
> > too. I think that API that we've settled on is in good shape, and is
> > certainly something that we can stand behind, and maintain going forward.
> >
> > I've still got a few more changes to push through over the next day or two,
> > and when that's done I'll be marking this API version 0.9, to emphasise the
> > approaching deadline.
> >
> > If you've not already done so, now is the time to try out what we've got.
> > This is the _only_ API for Xen that will be maintained in the long term, so if
> > there's something that you need for your product or your pet project, now is
> > the time to shout! We have both Python and C client-side bindings in the
> > tree, and a number of example scripts to get you started, and I'd be very
> > interested in your feedback.
> >
>
> The host class could use a capabilities field, perhaps a set of strings
> describing supported guests - similar to 'xen_caps' field produced by
> 'xm info'.

Yeah that's critical info if apps want to sanity check what kernels they
present to a user as bootable on a host.

> We've talked in the past about changing config of running guest vs
> changing its stored config. I would like to hotplug memory, cpu, disks,
> etc on a running guest but have it use stored config on next activation.
> Likewise it would be useful to change the stored config, regardless of
> guest state, for application on next activation.

I think that would be very useful for a number of use cases. I think I'd
previously suggested that for those kind of attributes, we'd want the API to
take an extra 'scope' argument, allowing a bitwise or of two constants
SCOPE_CONFIG or SCOPE_GUEST. Which would let us say affect running guest only,
affect config file only, or affect both guest & config.

Regards,
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

_______________________________________________
xen-api mailing list
xen-api@lists.xensource.com
http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-api
Re: Re: [Xen-cim] Xen-API update [ In reply to ]
On Fri, Feb 02, 2007 at 10:26:23AM -0700, Jim Fehlig wrote:
> Ewan Mellor wrote:
> > Here's a quick update on the Xen-API work; I thought that with the large
> > number of changesets that have gone in today that you might be curious ;-)
> >
>
> Great progress - thanks for all the work :-).
>
> > We had an internal review of the API last week here at XenSource, and what
> > you've seen today is the result of that. The goal is to have an interface in
> > Xen 3.0.5 that we will call the Xen-API 1.0, and that we will maintain in a
> > backwards-compatible fashion for the long term. With that in mind, some of
> > our "future hopes" that were still unimplemented have been bumped out, and
> > where we felt that we needed more flexibility, we've made provision for that
> > too. I think that API that we've settled on is in good shape, and is
> > certainly something that we can stand behind, and maintain going forward.
> >
> > I've still got a few more changes to push through over the next day or two,
> > and when that's done I'll be marking this API version 0.9, to emphasise the
> > approaching deadline.
> >
> > If you've not already done so, now is the time to try out what we've got.
> > This is the _only_ API for Xen that will be maintained in the long term, so if
> > there's something that you need for your product or your pet project, now is
> > the time to shout! We have both Python and C client-side bindings in the
> > tree, and a number of example scripts to get you started, and I'd be very
> > interested in your feedback.
> >
>
> The host class could use a capabilities field, perhaps a set of strings
> describing supported guests - similar to 'xen_caps' field produced by
> 'xm info'.

Yeah that's critical info if apps want to sanity check what kernels they
present to a user as bootable on a host.

> We've talked in the past about changing config of running guest vs
> changing its stored config. I would like to hotplug memory, cpu, disks,
> etc on a running guest but have it use stored config on next activation.
> Likewise it would be useful to change the stored config, regardless of
> guest state, for application on next activation.

I think that would be very useful for a number of use cases. I think I'd
previously suggested that for those kind of attributes, we'd want the API to
take an extra 'scope' argument, allowing a bitwise or of two constants
SCOPE_CONFIG or SCOPE_GUEST. Which would let us say affect running guest only,
affect config file only, or affect both guest & config.

Regards,
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|

_______________________________________________
Xen-cim mailing list
Xen-cim@lists.xensource.com
http://lists.xensource.com/xen-cim