Mailing List Archive

Re: [xs-devel] lvmthin
On Fri, Jan 09, 2015 at 03:59:26PM -0700, Tobias Kreidl wrote:
> So sorry, I meant of course to write that the utility is lvmthin (not
> thinlvm).
>

Hello,

I *think* the lvmthin stuff is currently designed for single-host systems,
where the storage is used by a single dedicated host only. That's the usual model for LVM aswell.

If multiple hosts access the same LVM volumes one needs to make sure the LVM metadata changes are properly synchronized across all hosts,
so all the hosts sharing the same physical LUNs/PVs/VGs/LVs have exactly the same idea of the LVM settings, and I believe XAPI does this for normal (thick) LVM volumes.

But it gets much more difficult with thin volumes, because the "metadata" can change on *every* write IO (when one needs to allocate more blocks from the thin pool?).

Note I haven't looked at the internals of LVM thinprovisioning, but I assume it'll be more difficult to get working in shared-LUN/multi-host environments compared to normal thick-LVM.

I guess there would have to be some "free-blocks-per-host-for-thin-volumes", so each host would allocate new blocks from its dedicated pool, without the risk of corrupting the LVM PVs or the volumes.. and avoiding the need of syncing metadata on every write IO?


Just random thoughts :)

-- Pasi

> On 1/9/2015 3:54 PM, Tobias Kreidl wrote:
>
> As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a
> thinlvm lvmthin utility that seems to take care of supporting
> thinly-provisioned LVM volumes. There is an associated snapshot design
> based on LVM thin provisioning that even supports the ability to do
> snapshots of snapshots, etc. down the chain.
>
> Unfortunately, from a cursory look, it doesn't look like a back port to
> CentOS 5 would be that easy or even possible, but that it is integrated
> into both CentOS 6 and CentOS 7 gives some hope for a possibly
> standardized support of it in a future XenServer release, doesn't it?
> The other big question would be how readily something like this could be
> integrated into XenServer.
>
> -=Tobias

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: [xs-devel] lvmthin [ In reply to ]
So sorry, I meant of course to write that the utility is *lvmthin*
(_not_ thinlvm).

On 1/9/2015 3:54 PM, Tobias Kreidl wrote:
> As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a
> thinlvm *lvmthin* utility that seems to take care of supporting
> thinly-provisioned LVM volumes. There is an associated snapshot design
> based on LVM thin provisioning that even supports the ability to do
> snapshots of snapshots, etc. down the chain.
>
> Unfortunately, from a cursory look, it doesn't look like a back port
> to CentOS 5 would be that easy or even possible, but that it is
> integrated into both CentOS 6 and CentOS 7 gives some hope for a
> possibly standardized support of it in a future XenServer release,
> doesn't it? The other big question would be how readily something like
> this could be integrated into XenServer.
>
> -=Tobias
>
Re: [xs-devel] lvmthin [ In reply to ]
> On 12 Jan 2015, at 09:29, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>
> On Fri, Jan 09, 2015 at 03:59:26PM -0700, Tobias Kreidl wrote:
>> So sorry, I meant of course to write that the utility is lvmthin (not
>> thinlvm).
>>
>
> Hello,
>
> I *think* the lvmthin stuff is currently designed for single-host systems,
> where the storage is used by a single dedicated host only. That's the usual model for LVM aswell.

That’s right.

>
> If multiple hosts access the same LVM volumes one needs to make sure the LVM metadata changes are properly synchronized across all hosts,
> so all the hosts sharing the same physical LUNs/PVs/VGs/LVs have exactly the same idea of the LVM settings, and I believe XAPI does this for normal (thick) LVM volumes.
>
> But it gets much more difficult with thin volumes, because the "metadata" can change on *every* write IO (when one needs to allocate more blocks from the thin pool?).
>
> Note I haven't looked at the internals of LVM thinprovisioning, but I assume it'll be more difficult to get working in shared-LUN/multi-host environments compared to normal thick-LVM.
>
> I guess there would have to be some "free-blocks-per-host-for-thin-volumes", so each host would allocate new blocks from its dedicated pool, without the risk of corrupting the LVM PVs or the volumes.. and avoiding the need of syncing metadata on every write IO?

LVM’s thin provisioning uses the device mapper “dm-thin” target, which works as you describe. I think for local, non-shared LVM it looks really good. I’ve got a very experimental storage plugin which can use it (ezlvm[1]). Also I think Fedora 21 can boot from thin-provisioned LVM which would let you do things like snapshot (and revert) the whole dom0 filesystem.

I’ve thought a little about how we could use it in future for shared LVM. Potentially we could make the “thin pool” work for per-host allocation by a ballooning-like technique. If every host saw the full set of free blocks, but most of the blocks were masked off by a fake disk (like the balloon driver’s balloon) then each host would be able to allocate locally from the same block address space. This would make it easier to move volumes between hosts. I think each host would need its own private copy of the LVM metadata, which would function like the journal in this “thinlvhd” design. When I tried to make this work a while ago I hit a problem when I tried to modify the “balloon disk”: when I reloaded the thin pool metadata and resumed the device mapper device it had clearly cached some of the data structure in memory because it immediately corrupted itself. This is probably fixable but requires more experimentation :)

So for the shared case I think we should work on “thinlvhd” (i.e. thin provisioning via tapdisk) first but play with dm-thin in the background :)

Cheers,
Dave

[1] https://github.com/xapi-project/ezlvm


>
>
> Just random thoughts :)
>
> -- Pasi
>
>> On 1/9/2015 3:54 PM, Tobias Kreidl wrote:
>>
>> As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a
>> thinlvm lvmthin utility that seems to take care of supporting
>> thinly-provisioned LVM volumes. There is an associated snapshot design
>> based on LVM thin provisioning that even supports the ability to do
>> snapshots of snapshots, etc. down the chain.
>>
>> Unfortunately, from a cursory look, it doesn't look like a back port to
>> CentOS 5 would be that easy or even possible, but that it is integrated
>> into both CentOS 6 and CentOS 7 gives some hope for a possibly
>> standardized support of it in a future XenServer release, doesn't it?
>> The other big question would be how readily something like this could be
>> integrated into XenServer.
>>
>> -=Tobias
>

_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
Re: [xs-devel] lvmthin [ In reply to ]
Thanks much, Dave and Pasi, for your feedback. It certainly makes
sense to proceed in that order, viz. dealing with thin provisioning of
VHD before pursuing a pool-wide lvmthin implementation. Thanks very much
for taking this into consideration!
-=Tobias

On 1/12/2015 2:57 AM, Dave Scott wrote:
>> On 12 Jan 2015, at 09:29, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>>
>> On Fri, Jan 09, 2015 at 03:59:26PM -0700, Tobias Kreidl wrote:
>>> So sorry, I meant of course to write that the utility is lvmthin (not
>>> thinlvm).
>>>
>> Hello,
>>
>> I *think* the lvmthin stuff is currently designed for single-host systems,
>> where the storage is used by a single dedicated host only. That's the usual model for LVM aswell.
> That’s right.
>
>> If multiple hosts access the same LVM volumes one needs to make sure the LVM metadata changes are properly synchronized across all hosts,
>> so all the hosts sharing the same physical LUNs/PVs/VGs/LVs have exactly the same idea of the LVM settings, and I believe XAPI does this for normal (thick) LVM volumes.
>>
>> But it gets much more difficult with thin volumes, because the "metadata" can change on *every* write IO (when one needs to allocate more blocks from the thin pool?).
>>
>> Note I haven't looked at the internals of LVM thinprovisioning, but I assume it'll be more difficult to get working in shared-LUN/multi-host environments compared to normal thick-LVM.
>>
>> I guess there would have to be some "free-blocks-per-host-for-thin-volumes", so each host would allocate new blocks from its dedicated pool, without the risk of corrupting the LVM PVs or the volumes.. and avoiding the need of syncing metadata on every write IO?
> LVM’s thin provisioning uses the device mapper “dm-thin” target, which works as you describe. I think for local, non-shared LVM it looks really good. I’ve got a very experimental storage plugin which can use it (ezlvm[1]). Also I think Fedora 21 can boot from thin-provisioned LVM which would let you do things like snapshot (and revert) the whole dom0 filesystem.
>
> I’ve thought a little about how we could use it in future for shared LVM. Potentially we could make the “thin pool” work for per-host allocation by a ballooning-like technique. If every host saw the full set of free blocks, but most of the blocks were masked off by a fake disk (like the balloon driver’s balloon) then each host would be able to allocate locally from the same block address space. This would make it easier to move volumes between hosts. I think each host would need its own private copy of the LVM metadata, which would function like the journal in this “thinlvhd” design. When I tried to make this work a while ago I hit a problem when I tried to modify the “balloon disk”: when I reloaded the thin pool metadata and resumed the device mapper device it had clearly cached some of the data structure in memory because it immediately corrupted itself. This is probably fixable but requires more experimentation :)
>
> So for the shared case I think we should work on “thinlvhd” (i.e. thin provisioning via tapdisk) first but play with dm-thin in the background :)
>
> Cheers,
> Dave
>
> [1] https://github.com/xapi-project/ezlvm
>
>
>>
>> Just random thoughts :)
>>
>> -- Pasi
>>
>>> On 1/9/2015 3:54 PM, Tobias Kreidl wrote:
>>>
>>> As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a
>>> thinlvm lvmthin utility that seems to take care of supporting
>>> thinly-provisioned LVM volumes. There is an associated snapshot design
>>> based on LVM thin provisioning that even supports the ability to do
>>> snapshots of snapshots, etc. down the chain.
>>>
>>> Unfortunately, from a cursory look, it doesn't look like a back port to
>>> CentOS 5 would be that easy or even possible, but that it is integrated
>>> into both CentOS 6 and CentOS 7 gives some hope for a possibly
>>> standardized support of it in a future XenServer release, doesn't it?
>>> The other big question would be how readily something like this could be
>>> integrated into XenServer.
>>>
>>> -=Tobias




_______________________________________________
Xen-api mailing list
Xen-api@lists.xen.org
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api