Mailing List Archive

Re: [xs-devel] Backup solutions for XenServer
> On 8 Jan 2015, at 14:26, Lorscheider Santiago <lorscheider.santiago@gmail.com> wrote:
>
> Hi Tobias,
>
> Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will not have much gain.

That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are “thick provisioned”
and hence VDIs take up a lot of space. As Tobias says below, this shows how important
thin provisioning is.

Within the upstream xapi-project (cc:d xen-api) we’ve been looking into thin provisioning
options for block storage. I’ve written two draft designs:

1. thin LVHD:

http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html

This proposes to extend the existing LVM-based SRs, allowing LVs to be resized at runtime
as more space is needed. Every LV could start out small (like a file) and grow as new
blocks are written. The trick is to make sure allocation can still happen when there
is a network partition in the pool and avoid timing out VM I/O. The design proposes to
cache free blocks locally on each host, allocate from there, and replay an allocation
journal against the LVM metadata periodically. [. This is all based on previous work by
Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]

2. OCFS2:

http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html

This proposes to manage OCFS2 clustered filesystem instances and to store VDIs as files
(format flexible) on top. Most of the design so far is to deal with the complexity of
managing the configuration and state of the O2CB cluster.

My gut feeling at the moment is that thin LVHD is simpler to implement and will cause
less “behaviour churn”, since we don’t have to change how HA behaves, how maintenance mode
is used etc.

Hopefully at some point in the future Xapi will be able to use any user-managed existing
filesystem, so a Linux expert power-user could use OCFS2 if they wanted.

Cheers,
Dave Scott

>
> Best Regards,
>
> Cordialmente,
>
> Lorscheider Santiago
> Visite meu blog: www.centralcloud.info
> Twitter: @lsantiagos
> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>
> On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl <tobias.kreidl@nau.edu> wrote:
> Thank you very much, Dave, for the updates about this nice addition.
>
> Here is my one concern, namely that for incremental backups to work properly, you need to continuously keep a base snapshot on the server, hence you will -- even if sparse storage is utilized -- still take up a fair amount of storage as long as you want to maintain the snapshot and backup mechanism.
>
> This led me to thinking, so, what happens if you have SRs with different underlying properties? What kind of space requirements are being realized?
>
> As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and checked how much space was really being used as well as allocated on the SR, as well as in the export files. Please note that here, some extra space savings take place on the NexentaStor NFS mount because of some built-in automatic compression taking place.
>
> For a Linux VM with 8 GB of allocated storage on a VDI and around 1.63 GB of space in use and a Windows VM with 40 GB of allocated storage and around 20 GB in actual use, here are the numbers I came up with:
>
>
> RHEL 7 RHEL 7 Windows 8.1 Windows 8.1
> Function LVM NFS+compression LVM NFS+compression
>
> base used: 8.0 GB 1.1 GB 40.0 GB 20.2 GB
> base allocated: 8.0 GB 8.0 GB 40.0 GB 40.0 GB
>
> pre-snapshot
> virtual-size: 8589923591 8589934592 42049672960 42949672960
> physical-utilisation: 8615100416 5984371200 43041947648 20136346112
>
> snapshot used: 13.6 GB 1.1 GB 59.0 GB 20.2 GB
> snapshot allocated: 16.0 GB 16.0 GB 80.1 GB 80.0 GB
>
> full snapshot
> virtual-size: 8589934592 8589934592 42949672960 42949672960
> physical-virtualisation: 8388608 39944704 8388608 88576
>
> snapshot
> export size (VHD): 5.97 GB 5.97 GB 20.24 GB 20.22 GB
>
> delta snapshot used: 13.6 GB 1.1 GB 59.1 GB 0.3 GB
> delta snap allocated: 24.0 GB 24.0 GB 120.1 GB 120.0 GB
>
> delta snapshot
> virtual-size: 8589934592 8589934592 42949672960 42949672960
> physical-virtualistion: 8388608 20992 8388608 88576
>
> delta snapshot
> export size: 0.126 GB 0.044 GB 0.078 GB 0.074 GB
>
>
> Not surprising is that the exports all came our roughly the same (they are, after all, creating VHD files in all cases), with minor difference due to some activity between when the exports were created. However, what is particularly interesting in the NFS-based SR is that the initial snapshot seems to have taken up virtually no discernible space at all. Why? Evidently because it creates it's own initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI) params=all" on the VDI yields details on the space utilization/allocation. This is actually quite interesting, as I had not tried this with an already thinly-provisioned (as opposed to sparely populated) storage before.
>
> Noting how an NFS-based SR is so much more space efficient, it does however leave the issue that if create both a full and delta snapshot, you will no longer be able to storage Xenmotion the VM until you bring the snapshot count back down to two or lower. One option would be to immediately clean this up after the latest delta vdi-export has taken place; alternatively, you could just defer this until a storage Xenmotion or other action is called for that requires prior cleanup.
>
> This brings us back to the LVM case. The huge difference here is that you need readily three times the size of the VDI to be allocated and at least double the size to be able to retain a base copy and triple the size to hold in addition a delta snapshot. Plus, unlike a NFS-based SR, you cannot over-commit your storage allocation on the SR. it would hence be of great benefit in space savings if one could do one of the following: (1) store a thinly-provisioned and compressed version of the initial snapshot similar to what NexentaStor does with the NFS VM, (2) in creating an incremental snapshot, access somehow an off-line VDI file containing the base, (3) had the means to efficiently temporarily pull in an offlined VDI so you'd not have to keep the base snapshot on the SR constantly, (4) be able to store the base snapshot on a different SR than the VDI you want to snapshot, or (5) some other clever, unnamed mechanism.
>
> The NAU VMbackup mechanism we have used in-house for years is efficient mainly because it creates a full snapshot on the fly for the purpose of a full backup and once completed, deletes it. Hence, you never need more additional space at any given time than that of a copy of the largest storage associated with an particular VM. The disadvantage is that this is of course a sequential operation and hence takes quite some time if you have a lot of VMs. However, being able to snapshot and keep a spare image would still take up a large amount of extra space. While having the means to very efficiently create incremental (delta) snapshots and back them up, this still leaves the large storage requirement issue open for LVI-based storage, should one want to retain baseline snapshots over longer periods of time.
>
> To me, this is an indication how thin provisioning (in contrast to just sparse storage) can make a huge difference. It would be really interesting to see what other options could be implemented to help address the LVM limitations outlined above.
>
> Thanks for taking the time to look over these thoughts, and I most certainly welcome feedback, in particular if I have overlooked something blatant.
>
> -=Tobias
>
>
> On 1/5/2015 8:09 AM, Dave Scott wrote:
>>> On 31 Dec 2014, at 11:27, Lorscheider Santiago <lorscheider.santiago@gmail.com>
>>> wrote:
>>>
>>> Hi Dave!
>>>
>>> Very good xapi the project site. Thanks for the tip!
>>> http://xapi-project.github.io/
>>>
>>>
>>> About recuros of "export and import only the blocks Which have changed between two snapshots" this is a sensational news! It's been times that I miss this feature, which brings tremendous agility to backup. From what you wrote, similar operation to VMware CBT.
>>>
>>> By your tests, you confirmed that this resource is in the build 90383c. We can not test since the Release Candidate build is in 90239c. Although he had already noticed changes in the snapshot in earlier builds the 90239c.
>>>
>>> Congratulations for the work and thanks for sharing this information with us!
>>>
>> I’m glad you liked the new site! Credit for the site itself and much of the content should also go to: (extracted from git history in no particular order)
>>
>> Euan Harris
>> John Else
>> Jonathan Davies
>> Rob Hoes
>> Si Beaumont
>>
>> If you find any problems with it or have suggestions for improvement (or new content), feel free to make pull requests or make issues on the tracker:
>>
>>
>> https://github.com/xapi-project/xapi-project.github.io
>>
>>
>> Cheers,
>> Dave
>>
>>
>>> Cordialmente,
>>>
>>> Lorscheider Santiago
>>> Visite meu blog:
>>> www.centralcloud.info
>>>
>>> Twitter: @lsantiagos
>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>>
>>> On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott
>>> <Dave.Scott@citrix.com>
>>> wrote:
>>>
>>>
>>>> On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@nau.edu>
>>>> wrote:
>>>>
>>>> Hi Dave,
>>>>
>>>> This looks very interesting. Since this is in the xapi-project does this mean it is intended for some future release? Is it in the current Creedence pre-release build?
>>>>
>>> I believe the code for this is present in the recent creedence builds. Just to check I installed build number ‘90383c’ and did a bit of smoke testing— the xe commands listed in the xapi-project page ran ok and produced the correct output.
>>>
>>> HTH,
>>> Dave
>>>
>>>
>>>> Duane
>>>>
>>>> -----Original Message-----
>>>> From:
>>>> xs-devel-request@lists.xenserver.org [mailto:xs-devel-request@lists.xenserver.org
>>>> ] On Behalf Of Dave Scott
>>>> Sent: Sunday, December 28, 2014 4:05 AM
>>>> To:
>>>> xs-devel@lists.xenserver.org
>>>>
>>>> Subject: Re: [xs-devel] Backup solutions for XenServer
>>>>
>>>> Hi,
>>>>
>>>>
>>>>> On 27 Dec 2014, at 12:07, Lorscheider Santiago <lorscheider.santiago@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Duane,
>>>>>
>>>>> I do not mean backup tools but the improvements that XenServer 6.5 in your api to facilitate the work of backup tools.
>>>>>
>>>> I've written up some of the recent XenAPI improvements here:
>>>>
>>>>
>>>> http://xapi-project.github.io/features/snapshots/snapshots.html
>>>>
>>>>
>>>> The new APIs allow you to
>>>>
>>>> - export and import disks in .vhd format (previously we only supported raw). This means the files are sparse.
>>>> - export and import only the blocks which have changed between two snapshots. This allows you to avoid re-copying the same data again and again.
>>>>
>>>> Let me know what you think.
>>>>
>>>> Cheers,
>>>> Dave
>>>>
>>>>
>>>>> Cordialmente,
>>>>>
>>>>> Lorscheider Santiago
>>>>> Visite meu blog:
>>>>> www.centralcloud.info
>>>>>
>>>>> Twitter: @lsantiagos
>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>>>>
>>>>> On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
>>>>> <Duane.Booher@nau.edu>
>>>>> wrote:
>>>>> Hi Lorscheider,
>>>>>
>>>>>
>>>>>
>>>>> We use
>>>>> https://github.com/NAUbackup
>>>>> for all our production Xenserver VM backups. It is freely available if you would like to give it a try.
>>>>>
>>>>>
>>>>>
>>>>> Duane
>>>>>
>>>>>
>>>>>
>>>>> From:
>>>>> xs-devel-request@lists.xenserver.org [mailto:xs-devel-request@lists.xenserver.org
>>>>> ] On Behalf Of Lorscheider Santiago
>>>>> Sent: Friday, December 19, 2014 3:33 AM
>>>>> To: xs-devel
>>>>> Subject: [xs-devel] Backup solutions for XenServer
>>>>>
>>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>>
>>>>>
>>>>> Looking backup solutions available on the market, solutions with good support XenServer are limited. The changes made in creedence api will make it easier and more attractive than more backup siluçoes have support XenServer?
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Lorscheider Santiago
>>>>> Visite meu blog:
>>>>> www.centralcloud.info
>>>>>
>>>>> Twitter: @lsantiagos
>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>>>>
>>>>>
>>>>>
>
>
>


_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: [xs-devel] Backup solutions for XenServer [ In reply to ]
Hello, David:

Thanks very much for that feedback. Simple often means good with an
easier implementation and fewer complexities to worry about, so as
suggested some time ago on the Xen Project Web site
https://xenorg.uservoice.com/forums/172169-xen-development?filter=hot&page=1
the thin provisioning of LVM does appear to be a good contender. The
running out of space issue is, of course, endemic in any storage
mechanism, so it's more a question of how to prevent corruption and
recover in that event. As with ext file systems, perhaps keeping a
certain percentage of the volume reserved for emergency issues could
help while refusing additional write commits at some point when all
storage capacity is exhausted (going into "RO" mode). Of course, being
able to copy out and recover lost space would be a useful means of being
able to keep VMs running by doing storage Xenmotions periodically and
recovering the space through some means from the original disk (if need
be though a re-initialization process or an LVM resize operation). With
thin provisioning, it's less of an issue for some scenarios like a
XenDesktop environment containing a large number of Windows clients
because Microsoft comes out with monthly patches and it is way more
efficient to just apply those to your golden images and reprovision all
your VMs. You're then back to your base storage.

It would also be nice to se VHDX adopted for Windows VMs at some point.
And while on the topic of storage, where do things stand with Ceph? The
flexibility looks tremendous, but my understanding is that there are
still performance issues. It has the capability of natively supporting
several different storage options, addressing your comment about plug-in
support for virtually any sort of file system one might want. It would
also be interesting to get an update on both NFSv4 (which works under
Creedence, just isn't officially supported) as well as with potential
pNFS support. Some vendors like NetApp showed early on that NFS is still
a very viable option for many storage needs and our own tests of iSCSI
vs. NFS were convincing enough that with an LACP bond, we have a very
solid NFS connection to our NexentaStor SDS and with thin provisioning
plus on-the-fly compression/decompression, get anywhere from a 20:1 to
50:1 reduction in utilized storage. And as mentioned already, this is
with the equal I/O performance of iSCSI. It's hard to beat that,
especially if you can convince those who hold the purse strings that
you've just increased your storage capacity by a factor of 20 or more
without needing any additional disks.

Again, your efforts in this area are most appreciated and thank you for
continuing to investigate and address these points. Citrix has been
really great about being open to suggestions and feedback from the user
community, so many kudos to you and your colleagues for lending your ears.

-=Tobias



On 1/8/2015 7:57 AM, Dave Scott wrote:
>> On 8 Jan 2015, at 14:26, Lorscheider Santiago <lorscheider.santiago@gmail.com> wrote:
>>
>> Hi Tobias,
>>
>> Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will not have much gain.
> That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are “thick provisioned”
> and hence VDIs take up a lot of space. As Tobias says below, this shows how important
> thin provisioning is.
>
> Within the upstream xapi-project (cc:d xen-api) we’ve been looking into thin provisioning
> options for block storage. I’ve written two draft designs:
>
> 1. thin LVHD:
>
> http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html
>
> This proposes to extend the existing LVM-based SRs, allowing LVs to be resized at runtime
> as more space is needed. Every LV could start out small (like a file) and grow as new
> blocks are written. The trick is to make sure allocation can still happen when there
> is a network partition in the pool and avoid timing out VM I/O. The design proposes to
> cache free blocks locally on each host, allocate from there, and replay an allocation
> journal against the LVM metadata periodically. [. This is all based on previous work by
> Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]
>
> 2. OCFS2:
>
> http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html
>
> This proposes to manage OCFS2 clustered filesystem instances and to store VDIs as files
> (format flexible) on top. Most of the design so far is to deal with the complexity of
> managing the configuration and state of the O2CB cluster.
>
> My gut feeling at the moment is that thin LVHD is simpler to implement and will cause
> less “behaviour churn”, since we don’t have to change how HA behaves, how maintenance mode
> is used etc.
>
> Hopefully at some point in the future Xapi will be able to use any user-managed existing
> filesystem, so a Linux expert power-user could use OCFS2 if they wanted.
>
> Cheers,
> Dave Scott
>
>> Best Regards,
>>
>> Cordialmente,
>>
>> Lorscheider Santiago
>> Visite meu blog: www.centralcloud.info
>> Twitter: @lsantiagos
>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>
>> On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl <tobias.kreidl@nau.edu> wrote:
>> Thank you very much, Dave, for the updates about this nice addition.
>>
>> Here is my one concern, namely that for incremental backups to work properly, you need to continuously keep a base snapshot on the server, hence you will -- even if sparse storage is utilized -- still take up a fair amount of storage as long as you want to maintain the snapshot and backup mechanism.
>>
>> This led me to thinking, so, what happens if you have SRs with different underlying properties? What kind of space requirements are being realized?
>>
>> As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and checked how much space was really being used as well as allocated on the SR, as well as in the export files. Please note that here, some extra space savings take place on the NexentaStor NFS mount because of some built-in automatic compression taking place.
>>
>> For a Linux VM with 8 GB of allocated storage on a VDI and around 1.63 GB of space in use and a Windows VM with 40 GB of allocated storage and around 20 GB in actual use, here are the numbers I came up with:
>>
>>
>> RHEL 7 RHEL 7 Windows 8.1 Windows 8.1
>> Function LVM NFS+compression LVM NFS+compression
>>
>> base used: 8.0 GB 1.1 GB 40.0 GB 20.2 GB
>> base allocated: 8.0 GB 8.0 GB 40.0 GB 40.0 GB
>>
>> pre-snapshot
>> virtual-size: 8589923591 8589934592 42049672960 42949672960
>> physical-utilisation: 8615100416 5984371200 43041947648 20136346112
>>
>> snapshot used: 13.6 GB 1.1 GB 59.0 GB 20.2 GB
>> snapshot allocated: 16.0 GB 16.0 GB 80.1 GB 80.0 GB
>>
>> full snapshot
>> virtual-size: 8589934592 8589934592 42949672960 42949672960
>> physical-virtualisation: 8388608 39944704 8388608 88576
>>
>> snapshot
>> export size (VHD): 5.97 GB 5.97 GB 20.24 GB 20.22 GB
>>
>> delta snapshot used: 13.6 GB 1.1 GB 59.1 GB 0.3 GB
>> delta snap allocated: 24.0 GB 24.0 GB 120.1 GB 120.0 GB
>>
>> delta snapshot
>> virtual-size: 8589934592 8589934592 42949672960 42949672960
>> physical-virtualistion: 8388608 20992 8388608 88576
>>
>> delta snapshot
>> export size: 0.126 GB 0.044 GB 0.078 GB 0.074 GB
>>
>>
>> Not surprising is that the exports all came our roughly the same (they are, after all, creating VHD files in all cases), with minor difference due to some activity between when the exports were created. However, what is particularly interesting in the NFS-based SR is that the initial snapshot seems to have taken up virtually no discernible space at all. Why? Evidently because it creates it's own initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI) params=all" on the VDI yields details on the space utilization/allocation. This is actually quite interesting, as I had not tried this with an already thinly-provisioned (as opposed to sparely populated) storage before.
>>
>> Noting how an NFS-based SR is so much more space efficient, it does however leave the issue that if create both a full and delta snapshot, you will no longer be able to storage Xenmotion the VM until you bring the snapshot count back down to two or lower. One option would be to immediately clean this up after the latest delta vdi-export has taken place; alternatively, you could just defer this until a storage Xenmotion or other action is called for that requires prior cleanup.
>>
>> This brings us back to the LVM case. The huge difference here is that you need readily three times the size of the VDI to be allocated and at least double the size to be able to retain a base copy and triple the size to hold in addition a delta snapshot. Plus, unlike a NFS-based SR, you cannot over-commit your storage allocation on the SR. it would hence be of great benefit in space savings if one could do one of the following: (1) store a thinly-provisioned and compressed version of the initial snapshot similar to what NexentaStor does with the NFS VM, (2) in creating an incremental snapshot, access somehow an off-line VDI file containing the base, (3) had the means to efficiently temporarily pull in an offlined VDI so you'd not have to keep the base snapshot on the SR constantly, (4) be able to store the base snapshot on a different SR than the VDI you want to snapshot, or (5) some other clever, unnamed mechanism.
>>
>> The NAU VMbackup mechanism we have used in-house for years is efficient mainly because it creates a full snapshot on the fly for the purpose of a full backup and once completed, deletes it. Hence, you never need more additional space at any given time than that of a copy of the largest storage associated with an particular VM. The disadvantage is that this is of course a sequential operation and hence takes quite some time if you have a lot of VMs. However, being able to snapshot and keep a spare image would still take up a large amount of extra space. While having the means to very efficiently create incremental (delta) snapshots and back them up, this still leaves the large storage requirement issue open for LVM-based storage, should one want to retain baseline snapshots over longer periods of time.
>>
>> To me, this is an indication how thin provisioning (in contrast to just sparse storage) can make a huge difference. It would be really interesting to see what other options could be implemented to help address the LVM limitations outlined above.
>>
>> Thanks for taking the time to look over these thoughts, and I most certainly welcome feedback, in particular if I have overlooked something blatant.
>>
>> -=Tobias
>>
>>
>> On 1/5/2015 8:09 AM, Dave Scott wrote:
>>>> On 31 Dec 2014, at 11:27, Lorscheider Santiago <lorscheider.santiago@gmail.com>
>>>> wrote:
>>>>
>>>> Hi Dave!
>>>>
>>>> Very good xapi the project site. Thanks for the tip!
>>>> http://xapi-project.github.io/
>>>>
>>>>
>>>> About recuros of "export and import only the blocks Which have changed between two snapshots" this is a sensational news! It's been times that I miss this feature, which brings tremendous agility to backup. From what you wrote, similar operation to VMware CBT.
>>>>
>>>> By your tests, you confirmed that this resource is in the build 90383c. We can not test since the Release Candidate build is in 90239c. Although he had already noticed changes in the snapshot in earlier builds the 90239c.
>>>>
>>>> Congratulations for the work and thanks for sharing this information with us!
>>>>
>>> I’m glad you liked the new site! Credit for the site itself and much of the content should also go to: (extracted from git history in no particular order)
>>>
>>> Euan Harris
>>> John Else
>>> Jonathan Davies
>>> Rob Hoes
>>> Si Beaumont
>>>
>>> If you find any problems with it or have suggestions for improvement (or new content), feel free to make pull requests or make issues on the tracker:
>>>
>>>
>>> https://github.com/xapi-project/xapi-project.github.io
>>>
>>>
>>> Cheers,
>>> Dave
>>>
>>>
>>>> Cordialmente,
>>>>
>>>> Lorscheider Santiago
>>>> Visite meu blog:
>>>> www.centralcloud.info
>>>>
>>>> Twitter: @lsantiagos
>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>>>
>>>> On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott
>>>> <Dave.Scott@citrix.com>
>>>> wrote:
>>>>
>>>>
>>>>> On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@nau.edu>
>>>>> wrote:
>>>>>
>>>>> Hi Dave,
>>>>>
>>>>> This looks very interesting. Since this is in the xapi-project does this mean it is intended for some future release? Is it in the current Creedence pre-release build?
>>>>>
>>>> I believe the code for this is present in the recent creedence builds. Just to check I installed build number ‘90383c’ and did a bit of smoke testing— the xe commands listed in the xapi-project page ran ok and produced the correct output.
>>>>
>>>> HTH,
>>>> Dave
>>>>
>>>>
>>>>> Duane
>>>>>
>>>>> -----Original Message-----
>>>>> From:
>>>>> xs-devel-request@lists.xenserver.org [mailto:xs-devel-request@lists.xenserver.org
>>>>> ] On Behalf Of Dave Scott
>>>>> Sent: Sunday, December 28, 2014 4:05 AM
>>>>> To:
>>>>> xs-devel@lists.xenserver.org
>>>>>
>>>>> Subject: Re: [xs-devel] Backup solutions for XenServer
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>> On 27 Dec 2014, at 12:07, Lorscheider Santiago <lorscheider.santiago@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Duane,
>>>>>>
>>>>>> I do not mean backup tools but the improvements that XenServer 6.5 in your api to facilitate the work of backup tools.
>>>>>>
>>>>> I've written up some of the recent XenAPI improvements here:
>>>>>
>>>>>
>>>>> http://xapi-project.github.io/features/snapshots/snapshots.html
>>>>>
>>>>>
>>>>> The new APIs allow you to
>>>>>
>>>>> - export and import disks in .vhd format (previously we only supported raw). This means the files are sparse.
>>>>> - export and import only the blocks which have changed between two snapshots. This allows you to avoid re-copying the same data again and again.
>>>>>
>>>>> Let me know what you think.
>>>>>
>>>>> Cheers,
>>>>> Dave
>>>>>
>>>>>
>>>>>> Cordialmente,
>>>>>>
>>>>>> Lorscheider Santiago
>>>>>> Visite meu blog:
>>>>>> www.centralcloud.info
>>>>>>
>>>>>> Twitter: @lsantiagos
>>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>>>>>
>>>>>> On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
>>>>>> <Duane.Booher@nau.edu>
>>>>>> wrote:
>>>>>> Hi Lorscheider,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We use
>>>>>> https://github.com/NAUbackup
>>>>>> for all our production Xenserver VM backups. It is freely available if you would like to give it a try.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Duane
>>>>>>
>>>>>>
>>>>>>
>>>>>> From:
>>>>>> xs-devel-request@lists.xenserver.org [mailto:xs-devel-request@lists.xenserver.org
>>>>>> ] On Behalf Of Lorscheider Santiago
>>>>>> Sent: Friday, December 19, 2014 3:33 AM
>>>>>> To: xs-devel
>>>>>> Subject: [xs-devel] Backup solutions for XenServer
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Looking backup solutions available on the market, solutions with good support XenServer are limited. The changes made in creedence api will make it easier and more attractive than more backup siluçoes have support XenServer?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Lorscheider Santiago
>>>>>> Visite meu blog:
>>>>>> www.centralcloud.info
>>>>>>
>>>>>> Twitter: @lsantiagos
>>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio ambiente
>>>>>>
>>>>>>
>>>>>>
>>
>>




_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: [xs-devel] Backup solutions for XenServer [ In reply to ]
As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a
thinlvm utility that seems to take care of supporting thinly-provisioned
LVM volumes. There is an associated snapshot design based on LVM thin
provisioning that even supports the ability to do snapshots of
snapshots, etc. down the chain.

Unfortunately, from a cursory look, it doesn't look like a back port to
CentOS 5 would be that easy or even possible, but that it is integrated
into both CentOS 6 and CentOS 7 gives some hope for a possibly
standardized support of it in a future XenServer release, doesn't it?
The other big question would be how readily something like this could be
integrated into XenServer.

-=Tobias

On 1/8/2015 9:12 AM, Tobias Kreidl wrote:
> Hello, David:
>
> Thanks very much for that feedback. Simple often means good with an
> easier implementation and fewer complexities to worry about, so as
> suggested some time ago on the Xen Project Web site
> https://xenorg.uservoice.com/forums/172169-xen-development?filter=hot&page=1
> the thin provisioning of LVM does appear to be a good contender. The
> running out of space issue is, of course, endemic in any storage
> mechanism, so it's more a question of how to prevent corruption and
> recover in that event. As with ext file systems, perhaps keeping a
> certain percentage of the volume reserved for emergency issues could
> help while refusing additional write commits at some point when all
> storage capacity is exhausted (going into "RO" mode). Of course, being
> able to copy out and recover lost space would be a useful means of
> being able to keep VMs running by doing storage Xenmotions
> periodically and recovering the space through some means from the
> original disk (if need be though a re-initialization process or an LVM
> resize operation). With thin provisioning, it's less of an issue for
> some scenarios like a XenDesktop environment containing a large number
> of Windows clients because Microsoft comes out with monthly patches
> and it is way more efficient to just apply those to your golden images
> and reprovision all your VMs. You're then back to your base storage.
>
> It would also be nice to se VHDX adopted for Windows VMs at some
> point. And while on the topic of storage, where do things stand with
> Ceph? The flexibility looks tremendous, but my understanding is that
> there are still performance issues. It has the capability of natively
> supporting several different storage options, addressing your comment
> about plug-in support for virtually any sort of file system one might
> want. It would also be interesting to get an update on both NFSv4
> (which works under Creedence, just isn't officially supported) as well
> as with potential pNFS support. Some vendors like NetApp showed early
> on that NFS is still a very viable option for many storage needs and
> our own tests of iSCSI vs. NFS were convincing enough that with an
> LACP bond, we have a very solid NFS connection to our NexentaStor SDS
> and with thin provisioning plus on-the-fly compression/decompression,
> get anywhere from a 20:1 to 50:1 reduction in utilized storage. And as
> mentioned already, this is with the equal I/O performance of iSCSI.
> It's hard to beat that, especially if you can convince those who hold
> the purse strings that you've just increased your storage capacity by
> a factor of 20 or more without needing any additional disks.
>
> Again, your efforts in this area are most appreciated and thank you
> for continuing to investigate and address these points. Citrix has
> been really great about being open to suggestions and feedback from
> the user community, so many kudos to you and your colleagues for
> lending your ears.
>
> -=Tobias
>
>
>
> On 1/8/2015 7:57 AM, Dave Scott wrote:
>>> On 8 Jan 2015, at 14:26, Lorscheider Santiago
>>> <lorscheider.santiago@gmail.com> wrote:
>>>
>>> Hi Tobias,
>>>
>>> Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will
>>> not have much gain.
>> That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are
>> “thick provisioned”
>> and hence VDIs take up a lot of space. As Tobias says below, this
>> shows how important
>> thin provisioning is.
>>
>> Within the upstream xapi-project (cc:d xen-api) we’ve been looking
>> into thin provisioning
>> options for block storage. I’ve written two draft designs:
>>
>> 1. thin LVHD:
>>
>> http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html
>>
>> This proposes to extend the existing LVM-based SRs, allowing LVs to
>> be resized at runtime
>> as more space is needed. Every LV could start out small (like a file)
>> and grow as new
>> blocks are written. The trick is to make sure allocation can still
>> happen when there
>> is a network partition in the pool and avoid timing out VM I/O. The
>> design proposes to
>> cache free blocks locally on each host, allocate from there, and
>> replay an allocation
>> journal against the LVM metadata periodically. [. This is all based on
>> previous work by
>> Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]
>>
>> 2. OCFS2:
>>
>> http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html
>>
>> This proposes to manage OCFS2 clustered filesystem instances and to
>> store VDIs as files
>> (format flexible) on top. Most of the design so far is to deal with
>> the complexity of
>> managing the configuration and state of the O2CB cluster.
>>
>> My gut feeling at the moment is that thin LVHD is simpler to
>> implement and will cause
>> less “behaviour churn”, since we don’t have to change how HA behaves,
>> how maintenance mode
>> is used etc.
>>
>> Hopefully at some point in the future Xapi will be able to use any
>> user-managed existing
>> filesystem, so a Linux expert power-user could use OCFS2 if they wanted.
>>
>> Cheers,
>> Dave Scott
>>
>>> Best Regards,
>>>
>>> Cordialmente,
>>>
>>> Lorscheider Santiago
>>> Visite meu blog: www.centralcloud.info
>>> Twitter: @lsantiagos
>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o
>>> meio ambiente
>>>
>>> On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl
>>> <tobias.kreidl@nau.edu> wrote:
>>> Thank you very much, Dave, for the updates about this nice addition.
>>>
>>> Here is my one concern, namely that for incremental backups to work
>>> properly, you need to continuously keep a base snapshot on the
>>> server, hence you will -- even if sparse storage is utilized --
>>> still take up a fair amount of storage as long as you want to
>>> maintain the snapshot and backup mechanism.
>>>
>>> This led me to thinking, so, what happens if you have SRs with
>>> different underlying properties? What kind of space requirements are
>>> being realized?
>>>
>>> As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and
>>> checked how much space was really being used as well as allocated on
>>> the SR, as well as in the export files. Please note that here, some
>>> extra space savings take place on the NexentaStor NFS mount because
>>> of some built-in automatic compression taking place.
>>>
>>> For a Linux VM with 8 GB of allocated storage on a VDI and around
>>> 1.63 GB of space in use and a Windows VM with 40 GB of allocated
>>> storage and around 20 GB in actual use, here are the numbers I came
>>> up with:
>>>
>>>
>>> RHEL 7 RHEL 7 Windows 8.1
>>> Windows 8.1
>>> Function LVM NFS+compression LVM
>>> NFS+compression
>>>
>>> base used: 8.0 GB 1.1 GB 40.0 GB
>>> 20.2 GB
>>> base allocated: 8.0 GB 8.0 GB 40.0 GB
>>> 40.0 GB
>>>
>>> pre-snapshot
>>> virtual-size: 8589923591 8589934592 42049672960
>>> 42949672960
>>> physical-utilisation: 8615100416 5984371200 43041947648
>>> 20136346112
>>>
>>> snapshot used: 13.6 GB 1.1 GB 59.0 GB
>>> 20.2 GB
>>> snapshot allocated: 16.0 GB 16.0 GB 80.1 GB
>>> 80.0 GB
>>>
>>> full snapshot
>>> virtual-size: 8589934592 8589934592 42949672960
>>> 42949672960
>>> physical-virtualisation: 8388608 39944704 8388608 88576
>>>
>>> snapshot
>>> export size (VHD): 5.97 GB 5.97 GB 20.24 GB
>>> 20.22 GB
>>>
>>> delta snapshot used: 13.6 GB 1.1 GB 59.1 GB
>>> 0.3 GB
>>> delta snap allocated: 24.0 GB 24.0 GB 120.1 GB
>>> 120.0 GB
>>>
>>> delta snapshot
>>> virtual-size: 8589934592 8589934592 42949672960
>>> 42949672960
>>> physical-virtualistion: 8388608 20992 8388608 88576
>>>
>>> delta snapshot
>>> export size: 0.126 GB 0.044 GB 0.078 GB
>>> 0.074 GB
>>>
>>>
>>> Not surprising is that the exports all came our roughly the same
>>> (they are, after all, creating VHD files in all cases), with minor
>>> difference due to some activity between when the exports were
>>> created. However, what is particularly interesting in the NFS-based
>>> SR is that the initial snapshot seems to have taken up virtually no
>>> discernible space at all. Why? Evidently because it creates it's own
>>> initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI)
>>> params=all" on the VDI yields details on the space
>>> utilization/allocation. This is actually quite interesting, as I had
>>> not tried this with an already thinly-provisioned (as opposed to
>>> sparely populated) storage before.
>>>
>>> Noting how an NFS-based SR is so much more space efficient, it does
>>> however leave the issue that if create both a full and delta
>>> snapshot, you will no longer be able to storage Xenmotion the VM
>>> until you bring the snapshot count back down to two or lower. One
>>> option would be to immediately clean this up after the latest delta
>>> vdi-export has taken place; alternatively, you could just defer this
>>> until a storage Xenmotion or other action is called for that
>>> requires prior cleanup.
>>>
>>> This brings us back to the LVM case. The huge difference here is
>>> that you need readily three times the size of the VDI to be
>>> allocated and at least double the size to be able to retain a base
>>> copy and triple the size to hold in addition a delta snapshot.
>>> Plus, unlike a NFS-based SR, you cannot over-commit your storage
>>> allocation on the SR. it would hence be of great benefit in space
>>> savings if one could do one of the following: (1) store a
>>> thinly-provisioned and compressed version of the initial snapshot
>>> similar to what NexentaStor does with the NFS VM, (2) in creating an
>>> incremental snapshot, access somehow an off-line VDI file containing
>>> the base, (3) had the means to efficiently temporarily pull in an
>>> offlined VDI so you'd not have to keep the base snapshot on the SR
>>> constantly, (4) be able to store the base snapshot on a different SR
>>> than the VDI you want to snapshot, or (5) some other clever, unnamed
>>> mechanism.
>>>
>>> The NAU VMbackup mechanism we have used in-house for years is
>>> efficient mainly because it creates a full snapshot on the fly for
>>> the purpose of a full backup and once completed, deletes it. Hence,
>>> you never need more additional space at any given time than that of
>>> a copy of the largest storage associated with an particular VM. The
>>> disadvantage is that this is of course a sequential operation and
>>> hence takes quite some time if you have a lot of VMs. However, being
>>> able to snapshot and keep a spare image would still take up a large
>>> amount of extra space. While having the means to very efficiently
>>> create incremental (delta) snapshots and back them up, this still
>>> leaves the large storage requirement issue open for LVM-based
>>> storage, should one want to retain baseline snapshots over longer
>>> periods of time.
>>>
>>> To me, this is an indication how thin provisioning (in contrast to
>>> just sparse storage) can make a huge difference. It would be really
>>> interesting to see what other options could be implemented to help
>>> address the LVM limitations outlined above.
>>>
>>> Thanks for taking the time to look over these thoughts, and I most
>>> certainly welcome feedback, in particular if I have overlooked
>>> something blatant.
>>>
>>> -=Tobias
>>>
>>>
>>> On 1/5/2015 8:09 AM, Dave Scott wrote:
>>>>> On 31 Dec 2014, at 11:27, Lorscheider Santiago
>>>>> <lorscheider.santiago@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Dave!
>>>>>
>>>>> Very good xapi the project site. Thanks for the tip!
>>>>> http://xapi-project.github.io/
>>>>>
>>>>>
>>>>> About recuros of "export and import only the blocks Which have
>>>>> changed between two snapshots" this is a sensational news! It's
>>>>> been times that I miss this feature, which brings tremendous
>>>>> agility to backup. From what you wrote, similar operation to
>>>>> VMware CBT.
>>>>>
>>>>> By your tests, you confirmed that this resource is in the build
>>>>> 90383c. We can not test since the Release Candidate build is in
>>>>> 90239c. Although he had already noticed changes in the snapshot in
>>>>> earlier builds the 90239c.
>>>>>
>>>>> Congratulations for the work and thanks for sharing this
>>>>> information with us!
>>>>>
>>>> I’m glad you liked the new site! Credit for the site itself and
>>>> much of the content should also go to: (extracted from git history
>>>> in no particular order)
>>>>
>>>> Euan Harris
>>>> John Else
>>>> Jonathan Davies
>>>> Rob Hoes
>>>> Si Beaumont
>>>>
>>>> If you find any problems with it or have suggestions for
>>>> improvement (or new content), feel free to make pull requests or
>>>> make issues on the tracker:
>>>>
>>>>
>>>> https://github.com/xapi-project/xapi-project.github.io
>>>>
>>>>
>>>> Cheers,
>>>> Dave
>>>>
>>>>
>>>>> Cordialmente,
>>>>>
>>>>> Lorscheider Santiago
>>>>> Visite meu blog:
>>>>> www.centralcloud.info
>>>>>
>>>>> Twitter: @lsantiagos
>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com
>>>>> o meio ambiente
>>>>>
>>>>> On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott
>>>>> <Dave.Scott@citrix.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>> On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@nau.edu>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Dave,
>>>>>>
>>>>>> This looks very interesting. Since this is in the xapi-project
>>>>>> does this mean it is intended for some future release? Is it in
>>>>>> the current Creedence pre-release build?
>>>>>>
>>>>> I believe the code for this is present in the recent creedence
>>>>> builds. Just to check I installed build number ‘90383c’ and did a
>>>>> bit of smoke testing— the xe commands listed in the xapi-project
>>>>> page ran ok and produced the correct output.
>>>>>
>>>>> HTH,
>>>>> Dave
>>>>>
>>>>>
>>>>>> Duane
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From:
>>>>>> xs-devel-request@lists.xenserver.org
>>>>>> [mailto:xs-devel-request@lists.xenserver.org
>>>>>> ] On Behalf Of Dave Scott
>>>>>> Sent: Sunday, December 28, 2014 4:05 AM
>>>>>> To:
>>>>>> xs-devel@lists.xenserver.org
>>>>>>
>>>>>> Subject: Re: [xs-devel] Backup solutions for XenServer
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>> On 27 Dec 2014, at 12:07, Lorscheider Santiago
>>>>>>> <lorscheider.santiago@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Duane,
>>>>>>>
>>>>>>> I do not mean backup tools but the improvements that XenServer
>>>>>>> 6.5 in your api to facilitate the work of backup tools.
>>>>>>>
>>>>>> I've written up some of the recent XenAPI improvements here:
>>>>>>
>>>>>>
>>>>>> http://xapi-project.github.io/features/snapshots/snapshots.html
>>>>>>
>>>>>>
>>>>>> The new APIs allow you to
>>>>>>
>>>>>> - export and import disks in .vhd format (previously we only
>>>>>> supported raw). This means the files are sparse.
>>>>>> - export and import only the blocks which have changed between
>>>>>> two snapshots. This allows you to avoid re-copying the same data
>>>>>> again and again.
>>>>>>
>>>>>> Let me know what you think.
>>>>>>
>>>>>> Cheers,
>>>>>> Dave
>>>>>>
>>>>>>
>>>>>>> Cordialmente,
>>>>>>>
>>>>>>> Lorscheider Santiago
>>>>>>> Visite meu blog:
>>>>>>> www.centralcloud.info
>>>>>>>
>>>>>>> Twitter: @lsantiagos
>>>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso
>>>>>>> com o meio ambiente
>>>>>>>
>>>>>>> On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
>>>>>>> <Duane.Booher@nau.edu>
>>>>>>> wrote:
>>>>>>> Hi Lorscheider,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We use
>>>>>>> https://github.com/NAUbackup
>>>>>>> for all our production Xenserver VM backups. It is freely
>>>>>>> available if you would like to give it a try.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Duane
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> From:
>>>>>>> xs-devel-request@lists.xenserver.org
>>>>>>> [mailto:xs-devel-request@lists.xenserver.org
>>>>>>> ] On Behalf Of Lorscheider Santiago
>>>>>>> Sent: Friday, December 19, 2014 3:33 AM
>>>>>>> To: xs-devel
>>>>>>> Subject: [xs-devel] Backup solutions for XenServer
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Looking backup solutions available on the market, solutions with
>>>>>>> good support XenServer are limited. The changes made in
>>>>>>> creedence api will make it easier and more attractive than more
>>>>>>> backup siluçoes have support XenServer?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Lorscheider Santiago
>>>>>>> Visite meu blog:
>>>>>>> www.centralcloud.info
>>>>>>>
>>>>>>> Twitter: @lsantiagos
>>>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso
>>>>>>> com o meio ambiente
>>>>>>>
>>>>>>>
>>>>>>>
>>>
>>>
>
>
>



_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api