Mailing List Archive

VDI import/export with deltas
Hi,

One of the features of the upcoming xapi 2.0 release will be improved VDI (disk) import/export support[1][2][3]. In particular xapi now has APIs to import/export data in vhd format: both sparse images and deltas. I’ve written up a doc describing how these APIs currently work on the wiki:

http://wiki.xensource.com/wiki/Disk_import/export_APIs

There’s still time before 2.0 to change how these work, so your comments and criticism would be appreciated.

Thanks!
Dave

[1] https://github.com/xapi-project/xapi-project/issues/1
[2] https://github.com/xapi-project/xen-api/issues/1778
[3] https://github.com/xapi-project/vhd-tool/pull/14
_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
What a great news!

Correct me if I'm wrong:
- VM export: export of VM meta-data (config, number of RAM, CPU,
network) + all its VDI
- VDI export: just any VDI
- VDI "diff" export: any VDI, delta since a precise snapshot

Is there a possibility to export a "whole" VM diff (config + all its
VDI since a snapshot)?


Thanks!


Olivier.

On Thu, Aug 28, 2014 at 3:11 PM, Dave Scott <Dave.Scott@citrix.com> wrote:
> Hi,
>
> One of the features of the upcoming xapi 2.0 release will be improved VDI (disk) import/export support[1][2][3]. In particular xapi now has APIs to import/export data in vhd format: both sparse images and deltas. I’ve written up a doc describing how these APIs currently work on the wiki:
>
> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>
> There’s still time before 2.0 to change how these work, so your comments and criticism would be appreciated.
>
> Thanks!
> Dave
>
> [1] https://github.com/xapi-project/xapi-project/issues/1
> [2] https://github.com/xapi-project/xen-api/issues/1778
> [3] https://github.com/xapi-project/vhd-tool/pull/14
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
On 28 Aug 2014, at 14:59, Olivier Lambert <lambert.olivier@gmail.com> wrote:

> What a great news!
>
> Correct me if I'm wrong:
> - VM export: export of VM meta-data (config, number of RAM, CPU,
> network) + all its VDI
> - VDI export: just any VDI
> - VDI "diff" export: any VDI, delta since a precise snapshot

That’s right.

> Is there a possibility to export a "whole" VM diff (config + all its
> VDI since a snapshot)?

That would indeed be nice! I’d like to update the VM export/import code to be able to use .vhd format… adding “whole” VM diff would be easy after that.

Cheers,
Dave

>
>
> Thanks!
>
>
> Olivier.
>
> On Thu, Aug 28, 2014 at 3:11 PM, Dave Scott <Dave.Scott@citrix.com> wrote:
>> Hi,
>>
>> One of the features of the upcoming xapi 2.0 release will be improved VDI (disk) import/export support[1][2][3]. In particular xapi now has APIs to import/export data in vhd format: both sparse images and deltas. I’ve written up a doc describing how these APIs currently work on the wiki:
>>
>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>>
>> There’s still time before 2.0 to change how these work, so your comments and criticism would be appreciated.
>>
>> Thanks!
>> Dave
>>
>> [1] https://github.com/xapi-project/xapi-project/issues/1
>> [2] https://github.com/xapi-project/xen-api/issues/1778
>> [3] https://github.com/xapi-project/vhd-tool/pull/14
>> _______________________________________________
>> Xen-api mailing list
>> Xen-api@lists.xen.org
>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
> http://wiki.xensource.com/wiki/Disk_import/export_APIs

This is great, but I think we're missing the ability to pull a VHD in or push a VHD out.

The OpenStack case needs to call a vm-import giving a URL to pull a VHD from, and a vm-export to push a VHD out to.
In particular I think we need to set specific headers as well, as authentication, so I'd like to propose the following as a hopefully simple extension:

xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN

xe vdi-import uuid=$RESTORE format=vhd source=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN

OpenStack also has a case for using bittorrent to download the VDI from peers rather than the central server, but I don't see how to accomplish that with the current proposal without requiring double the disk space as we'd need to download it then push it into XS.

Thoughts?

Thanks,

Bob

> There's still time before 2.0 to change how these work, so your comments
> and criticism would be appreciated.
>
> Thanks!
> Dave
>
> [1] https://github.com/xapi-project/xapi-project/issues/1
> [2] https://github.com/xapi-project/xen-api/issues/1778
> [3] https://github.com/xapi-project/vhd-tool/pull/14
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
On 28/08/14 15:50, Bob Ball wrote:
>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
> This is great, but I think we're missing the ability to pull a VHD in or push a VHD out.
>
> The OpenStack case needs to call a vm-import giving a URL to pull a VHD from, and a vm-export to push a VHD out to.
> In particular I think we need to set specific headers as well, as authentication, so I'd like to propose the following as a hopefully simple extension:
>
> xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN
>
> xe vdi-import uuid=$RESTORE format=vhd source=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN

Agreed - this would be really useful for vagrant too; currently we have
to download the VHD to the users desktop before pushing it out to the
XenServer. It'd be much more useful to host the VHD file somewhere and
ask the XenServer to download it.

Jon


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
On 28 Aug 2014, at 15:50, Bob Ball <bob.ball@citrix.com> wrote:

>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>
> This is great, but I think we're missing the ability to pull a VHD in or push a VHD out.
>
> The OpenStack case needs to call a vm-import giving a URL to pull a VHD from, and a vm-export to push a VHD out to.
> In particular I think we need to set specific headers as well, as authentication, so I'd like to propose the following as a hopefully simple extension:
>
> xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN
>
> xe vdi-import uuid=$RESTORE format=vhd source=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN

This looks sensible to me; plus I’d like to use URLs more in the storage interface in future.

>
> OpenStack also has a case for using bittorrent to download the VDI from peers rather than the central server, but I don't see how to accomplish that with the current proposal without requiring double the disk space as we'd need to download it then push it into XS.

So at the moment, do you receive the .vhd data by bittorrent in-place?

When I last spoke to John about this we discussed some kind of import/export plugin extension. I guess to do the most efficient thing possible it would have to have access to (at least a subset of) the physical storage, so it can ‘drop in’ a vhd (or qcow or vmdk). Ideally the plugin wouldn’t see anything else (so it couldn’t get confused or make a mistake) and the modified files would be health-checked when the operation is finished.

> Thoughts?

Maybe we could make a safe import/export virtual filesystem with FUSE? Perhaps you could even SMB export a virtual folder for XenCenter users on windows to drag ’n drop files into.

Cheers,
Dave

>
> Thanks,
>
> Bob
>
>> There's still time before 2.0 to change how these work, so your comments
>> and criticism would be appreciated.
>>
>> Thanks!
>> Dave
>>
>> [1] https://github.com/xapi-project/xapi-project/issues/1
>> [2] https://github.com/xapi-project/xen-api/issues/1778
>> [3] https://github.com/xapi-project/vhd-tool/pull/14
>> _______________________________________________
>> Xen-api mailing list
>> Xen-api@lists.xen.org
>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
FYI, we'll also provide a way to do that kind of export in the local
file-system of Xen Orchestra Appliance.

On Thu, Aug 28, 2014 at 6:09 PM, Dave Scott <Dave.Scott@citrix.com> wrote:
>
> On 28 Aug 2014, at 15:50, Bob Ball <bob.ball@citrix.com> wrote:
>
>>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>>
>> This is great, but I think we're missing the ability to pull a VHD in or push a VHD out.
>>
>> The OpenStack case needs to call a vm-import giving a URL to pull a VHD from, and a vm-export to push a VHD out to.
>> In particular I think we need to set specific headers as well, as authentication, so I'd like to propose the following as a hopefully simple extension:
>>
>> xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN
>>
>> xe vdi-import uuid=$RESTORE format=vhd source=http://192.168.0.1:5000/... headers:arbitrary_string=$AUTH_TOKEN
>
> This looks sensible to me; plus I’d like to use URLs more in the storage interface in future.
>
>>
>> OpenStack also has a case for using bittorrent to download the VDI from peers rather than the central server, but I don't see how to accomplish that with the current proposal without requiring double the disk space as we'd need to download it then push it into XS.
>
> So at the moment, do you receive the .vhd data by bittorrent in-place?
>
> When I last spoke to John about this we discussed some kind of import/export plugin extension. I guess to do the most efficient thing possible it would have to have access to (at least a subset of) the physical storage, so it can ‘drop in’ a vhd (or qcow or vmdk). Ideally the plugin wouldn’t see anything else (so it couldn’t get confused or make a mistake) and the modified files would be health-checked when the operation is finished.
>
>> Thoughts?
>
> Maybe we could make a safe import/export virtual filesystem with FUSE? Perhaps you could even SMB export a virtual folder for XenCenter users on windows to drag ’n drop files into.
>
> Cheers,
> Dave
>
>>
>> Thanks,
>>
>> Bob
>>
>>> There's still time before 2.0 to change how these work, so your comments
>>> and criticism would be appreciated.
>>>
>>> Thanks!
>>> Dave
>>>
>>> [1] https://github.com/xapi-project/xapi-project/issues/1
>>> [2] https://github.com/xapi-project/xen-api/issues/1778
>>> [3] https://github.com/xapi-project/vhd-tool/pull/14
>>> _______________________________________________
>>> Xen-api mailing list
>>> Xen-api@lists.xen.org
>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@lists.xen.org
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
Hi,

I'm happy to see a prototype for import/export. It seems that this discussion is generating a lot of traffic - do we plan to have some sort of design session about it? I believe the storage team and the stackers would love to be involved in these sessions.
We also had a discussion which might be releated:
https://lists.xenserver.org/sympa/arc/xs-devel/2013-08/thrd2.html#00059

I think the idea of a filesystem interface is very good, however given that the filesystem must provide a consistent view and transactions, as other processes might manipulate files (GC), I think for the moment, let's try to do a bit narrower interface.

I also wanted to raise the content ID question - would be nice to generate a content id during the export, and mark it as dirty whenever it has changed, so that the client would know that she should not expect a consistent output.

Cheers,
Mate


-----Original Message-----
From: Dave Scott
Sent: Thursday, August 28, 2014 5:10 PM
To: Bob Ball
Cc: xen-api@lists.xenproject.org; Mate Lakat; John Garbutt; Ant Messerli
Subject: Re: VDI import/export with deltas


On 28 Aug 2014, at 15:50, Bob Ball <bob.ball@citrix.com> wrote:

>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>
> This is great, but I think we're missing the ability to pull a VHD in or push a VHD out.
>
> The OpenStack case needs to call a vm-import giving a URL to pull a VHD from, and a vm-export to push a VHD out to.
> In particular I think we need to set specific headers as well, as authentication, so I'd like to propose the following as a hopefully simple extension:
>
> xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/...
> headers:arbitrary_string=$AUTH_TOKEN
>
> xe vdi-import uuid=$RESTORE format=vhd
> source=http://192.168.0.1:5000/...
> headers:arbitrary_string=$AUTH_TOKEN

This looks sensible to me; plus I'd like to use URLs more in the storage interface in future.

>
> OpenStack also has a case for using bittorrent to download the VDI from peers rather than the central server, but I don't see how to accomplish that with the current proposal without requiring double the disk space as we'd need to download it then push it into XS.

So at the moment, do you receive the .vhd data by bittorrent in-place?

When I last spoke to John about this we discussed some kind of import/export plugin extension. I guess to do the most efficient thing possible it would have to have access to (at least a subset of) the physical storage, so it can 'drop in' a vhd (or qcow or vmdk). Ideally the plugin wouldn't see anything else (so it couldn't get confused or make a mistake) and the modified files would be health-checked when the operation is finished.

> Thoughts?

Maybe we could make a safe import/export virtual filesystem with FUSE? Perhaps you could even SMB export a virtual folder for XenCenter users on windows to drag 'n drop files into.

Cheers,
Dave

>
> Thanks,
>
> Bob
>
>> There's still time before 2.0 to change how these work, so your
>> comments and criticism would be appreciated.
>>
>> Thanks!
>> Dave
>>
>> [1] https://github.com/xapi-project/xapi-project/issues/1
>> [2] https://github.com/xapi-project/xen-api/issues/1778
>> [3] https://github.com/xapi-project/vhd-tool/pull/14
>> _______________________________________________
>> Xen-api mailing list
>> Xen-api@lists.xen.org
>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
Hi Mate,

On 29 Aug 2014, at 09:58, Mate Lakat <mate.lakat@citrix.com> wrote:

> Hi,
>
> I'm happy to see a prototype for import/export. It seems that this discussion is generating a lot of traffic - do we plan to have some sort of design session about it? I believe the storage team and the stackers would love to be involved in these sessions.
> We also had a discussion which might be releated:
> https://lists.xenserver.org/sympa/arc/xs-devel/2013-08/thrd2.html#00059

We did, thanks for the pointer.

I’ve also done a survey of import/export code in xapi:

http://wiki.xensource.com/wiki/Disk_import/export

The prototype is basically ‘what we already have’: the code was all written for various other purposes (e.g. storage motion and CloudStack integration) and all I did was to fill in a few missing gaps and add CLI commands on top of existing APIs.


> I think the idea of a filesystem interface is very good, however given that the filesystem must provide a consistent view and transactions, as other processes might manipulate files (GC), I think for the moment, let's try to do a bit narrower interface.

I’m definitely in favour of a narrow interface :-)
At the moment our interface is far too wide and exposes far too much (not interesting but still important) internal details.


> I also wanted to raise the content ID question - would be nice to generate a content id during the export, and mark it as dirty whenever it has changed, so that the client would know that she should not expect a consistent output.

That would be good. One of the things the current code can’t do — but that I’m sure we discussed before — is to support resumable downloads.

Thanks,
Dave

>
> Cheers,
> Mate
>
>
> -----Original Message-----
> From: Dave Scott
> Sent: Thursday, August 28, 2014 5:10 PM
> To: Bob Ball
> Cc: xen-api@lists.xenproject.org; Mate Lakat; John Garbutt; Ant Messerli
> Subject: Re: VDI import/export with deltas
>
>
> On 28 Aug 2014, at 15:50, Bob Ball <bob.ball@citrix.com> wrote:
>
>>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>>
>> This is great, but I think we're missing the ability to pull a VHD in or push a VHD out.
>>
>> The OpenStack case needs to call a vm-import giving a URL to pull a VHD from, and a vm-export to push a VHD out to.
>> In particular I think we need to set specific headers as well, as authentication, so I'd like to propose the following as a hopefully simple extension:
>>
>> xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/...
>> headers:arbitrary_string=$AUTH_TOKEN
>>
>> xe vdi-import uuid=$RESTORE format=vhd
>> source=http://192.168.0.1:5000/...
>> headers:arbitrary_string=$AUTH_TOKEN
>
> This looks sensible to me; plus I'd like to use URLs more in the storage interface in future.
>
>>
>> OpenStack also has a case for using bittorrent to download the VDI from peers rather than the central server, but I don't see how to accomplish that with the current proposal without requiring double the disk space as we'd need to download it then push it into XS.
>
> So at the moment, do you receive the .vhd data by bittorrent in-place?
>
> When I last spoke to John about this we discussed some kind of import/export plugin extension. I guess to do the most efficient thing possible it would have to have access to (at least a subset of) the physical storage, so it can 'drop in' a vhd (or qcow or vmdk). Ideally the plugin wouldn't see anything else (so it couldn't get confused or make a mistake) and the modified files would be health-checked when the operation is finished.
>
>> Thoughts?
>
> Maybe we could make a safe import/export virtual filesystem with FUSE? Perhaps you could even SMB export a virtual folder for XenCenter users on windows to drag 'n drop files into.
>
> Cheers,
> Dave
>
>>
>> Thanks,
>>
>> Bob
>>
>>> There's still time before 2.0 to change how these work, so your
>>> comments and criticism would be appreciated.
>>>
>>> Thanks!
>>> Dave
>>>
>>> [1] https://github.com/xapi-project/xapi-project/issues/1
>>> [2] https://github.com/xapi-project/xen-api/issues/1778
>>> [3] https://github.com/xapi-project/vhd-tool/pull/14
>>> _______________________________________________
>>> Xen-api mailing list
>>> Xen-api@lists.xen.org
>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: VDI import/export with deltas [ In reply to ]
Seems like I forgot to respond to this...

> > OpenStack also has a case for using bittorrent to download the VDI from
> peers rather than the central server, but I don't see how to accomplish that
> with the current proposal without requiring double the disk space as we'd
> need to download it then push it into XS.
>
> So at the moment, do you receive the .vhd data by bittorrent in-place?

We download to a sub-directory so the standard SR scan does not identify an incomplete VHD. Once the download is finished, we rename them (to avoid any conflicts), re-write the parentage information and finally move the VHD files to the base of the SR and perform an sr-scan so it finds the VHDs and creates appropriate VDIs.

Bob


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api