Mailing List Archive

Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd
Please suggest the Best Way to snapshot Secondary DRBD Volume(Upper) on
LVM/ZFS(Backing) using protocol A i.e Minor Data Loss(few Seconds) is OK
but data should remain crash consistent like Database.Also Metadata is
external in secondary. So Do i need to snapshot external metadata as well.
if yes Why ?


On Above few Queries are detailed below:


1)As per valuable suggestions by community that normal ZFS/LVM Snapshots
would do as long as it is upto date? How to know that in realtime uptodate
Status as drdbadm status command doesnot appear to be realtime in nature as
it takes time to reflect actual Status for example when primary goes down ,
only after few seconds only it shows as connecting


2) Community has again suggested for internal only metadata to have
simultaneous/atomic snapshot of both data and metadata .If the primary is
having external metadata , it makes sense to have external metadata in
secondary as well. But Why do we need to preserve the metadata via snapshot
at the first place as it is believed that once you rollback the using the
snapshot drbd would get confused and would attempt resynchronisation of the
entire device again any way


3) Do i need to suspend-io first before taking the snapshot and then check
up to date Status mandatorily ?


4) Last Query is regarding metadata, Does Metadata contains actual data in
the form of Activity log/extents.Then what is drbdadm apply-al is
for?Depending on the reply of this query we would be able to get crash
consistent data (via backing disk snapshot clone ofcourse) even if we loose
the metadata


Thanks and Regards,

Chitvan Chhabra

+919911208768
Re: Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd [ In reply to ]
All Suggestions are most welcome.

On Fri, 29 Jul 2022 at 16:58, Chitvan Chhabra <chitvan1988@gmail.com> wrote:

> Please suggest the Best Way to snapshot Secondary DRBD Volume(Upper) on
> LVM/ZFS(Backing) using protocol A i.e Minor Data Loss(few Seconds) is OK
> but data should remain crash consistent like Database.Also Metadata is
> external in secondary. So Do i need to snapshot external metadata as well.
> if yes Why ?
>
>
> On Above few Queries are detailed below:
>
>
> 1)As per valuable suggestions by community that normal ZFS/LVM Snapshots
> would do as long as it is upto date? How to know that in realtime uptodate
> Status as drdbadm status command doesnot appear to be realtime in nature as
> it takes time to reflect actual Status for example when primary goes down ,
> only after few seconds only it shows as connecting
>
>
> 2) Community has again suggested for internal only metadata to have
> simultaneous/atomic snapshot of both data and metadata .If the primary is
> having external metadata , it makes sense to have external metadata in
> secondary as well. But Why do we need to preserve the metadata via snapshot
> at the first place as it is believed that once you rollback the using the
> snapshot drbd would get confused and would attempt resynchronisation of the
> entire device again any way
>
>
> 3) Do i need to suspend-io first before taking the snapshot and then check
> up to date Status mandatorily ?
>
>
> 4) Last Query is regarding metadata, Does Metadata contains actual data in
> the form of Activity log/extents.Then what is drbdadm apply-al is
> for?Depending on the reply of this query we would be able to get crash
> consistent data (via backing disk snapshot clone ofcourse) even if we loose
> the metadata
>
>
> Thanks and Regards,
>
> Chitvan Chhabra
>
> +919911208768
>
Re: Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd [ In reply to ]
On Mon, Aug 1, 2022 at 4:38 AM Chitvan Chhabra <chitvan1988@gmail.com> wrote:
>
> All Suggestions are most welcome.
>
> On Fri, 29 Jul 2022 at 16:58, Chitvan Chhabra <chitvan1988@gmail.com> wrote:
>>
>> Please suggest the Best Way to snapshot Secondary DRBD Volume(Upper) on LVM/ZFS(Backing) using protocol A i.e Minor Data Loss(few Seconds) is OK but data should remain crash consistent like Database.Also Metadata is external in secondary. So Do i need to snapshot external metadata as well. if yes Why ?
>>
>>
>> On Above few Queries are detailed below:
>>
>>
>> 1)As per valuable suggestions by community that normal ZFS/LVM Snapshots would do as long as it is upto date? How to know that in realtime uptodate Status as drdbadm status command doesnot appear to be realtime in nature as it takes time to reflect actual Status for example when primary goes down , only after few seconds only it shows as connecting

I've used ZFS volumes to back my DRBD resources for years, relying on
zfs-auto-snapshot and the Copy On Write nature of ZFS to keep things
recoverable in case of some sort of corruption or human error.

I'm not sure how to answer your "realtime uptodate" question. I've
never run into data loss corner cases with nodes going down,
reconnecting, etc.

>>
>>
>> 2) Community has again suggested for internal only metadata to have simultaneous/atomic snapshot of both data and metadata .If the primary is having external metadata , it makes sense to have external metadata in secondary as well. But Why do we need to preserve the metadata via snapshot at the first place as it is believed that once you rollback the using the snapshot drbd would get confused and would attempt resynchronisation of the entire device again any way
>>

I've never played with external metadata, so don't have any real world
experience to answer this one. Internal aways seems like a much
better solution to me.

>>
>> 3) Do i need to suspend-io first before taking the snapshot and then check up to date Status mandatorily ?

I guess that depends on your volume management. If ZFS volumes, I'm
going to say no that's not necessary, at least I've never gotten
burned by not doing a suspend-io. LVM, I'm not sure.

>>
>>
>> 4) Last Query is regarding metadata, Does Metadata contains actual data in the form of Activity log/extents.Then what is drbdadm apply-al is for?Depending on the reply of this query we would be able to get crash consistent data (via backing disk snapshot clone ofcourse) even if we loose the metadata
>>

No idea. Someone else will have to chime in to answer that one.

>>
>> Thanks and Regards,
>>
>> Chitvan Chhabra
>>
>> +919911208768
>
> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user@lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
_______________________________________________
Star us on GITHUB: https://github.com/LINBIT
drbd-user mailing list
drbd-user@lists.linbit.com
https://lists.linbit.com/mailman/listinfo/drbd-user
Re: Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd [ In reply to ]
>
> But Why do we need to preserve the metadata via snapshot at the first
> place as it is believed that once you rollback the using the snapshot drbd
> would get confused and would attempt resynchronisation of the entire device
> again any way
>

Consider the following scenario, Two nodes A and B. A is the Primary and B
is the Secondary. You create a zfs snapshot (both data and drbd metadata)
at 08:00 am on both nodes. At 08:30 am you realise that a serious
corruption has taken place and you urgently need to rollback *both* nodes
from the snapshot created at 08:00 am. You execute a zfs rollback on both
nodes while the drbd resource is down of course. Before bringing the drbd
resource up on both nodes, you must decide which way the replication must
take place (e.g A -> B or B -> A). Once you decide, bring the resource up.
If all goes well, drbd should bring up the resource on both nodes *without*
needing to do a full sync but rather just a small increment instead, as the
metadata is consistent on both nodes (as it was at the time the snapshot
was taken). So it's important to snapshot the drbd metadata on both nodes,
if you want to prevent a full sync.


> 3) Do i need to suspend-io first before taking the snapshot and then check
> up to date Status mandatorily ?
>

Ideally yes but that depends if the layers above drbd supports that
functionality. For example, I'm using qemu VMs on top of drbd/zvol. QEMU
can suspend i/o before issuing a qemu based snapshot (via guest tools)
which then it will propagate at the layers below (e.g drbd -> zfs). If the
layers above drbd cannot handle this, and you could simply take a snapshot
at the layer below drbd (zfs in this case), then that would have the same
effect as when removing the power from the physical machine (e.g the data
would still be consistent due to zfs transaction based nature, but you may
or may have have not lost the last few writes issued by the layers above).
Re: Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd [ In reply to ]
Many Thanks for valuable suggestions and sharing your experiences.

On Mon, 1 Aug 2022 at 22:26, GM <gianni.milo22@gmail.com> wrote:

> But Why do we need to preserve the metadata via snapshot at the first
>> place as it is believed that once you rollback the using the snapshot drbd
>> would get confused and would attempt resynchronisation of the entire device
>> again any way
>>
>
> Consider the following scenario, Two nodes A and B. A is the Primary and B
> is the Secondary. You create a zfs snapshot (both data and drbd metadata)
> at 08:00 am on both nodes. At 08:30 am you realise that a serious
> corruption has taken place and you urgently need to rollback *both* nodes
> from the snapshot created at 08:00 am. You execute a zfs rollback on both
> nodes while the drbd resource is down of course. Before bringing the drbd
> resource up on both nodes, you must decide which way the replication must
> take place (e.g A -> B or B -> A). Once you decide, bring the resource up.
> If all goes well, drbd should bring up the resource on both nodes *without*
> needing to do a full sync but rather just a small increment instead, as the
> metadata is consistent on both nodes (as it was at the time the snapshot
> was taken). So it's important to snapshot the drbd metadata on both nodes,
> if you want to prevent a full sync.
>
>
>> 3) Do i need to suspend-io first before taking the snapshot and then
>> check up to date Status mandatorily ?
>>
>
> Ideally yes but that depends if the layers above drbd supports that
> functionality. For example, I'm using qemu VMs on top of drbd/zvol. QEMU
> can suspend i/o before issuing a qemu based snapshot (via guest tools)
> which then it will propagate at the layers below (e.g drbd -> zfs). If the
> layers above drbd cannot handle this, and you could simply take a snapshot
> at the layer below drbd (zfs in this case), then that would have the same
> effect as when removing the power from the physical machine (e.g the data
> would still be consistent due to zfs transaction based nature, but you may
> or may have have not lost the last few writes issued by the layers above).
>
> _______________________________________________
> Star us on GITHUB: https://github.com/LINBIT
> drbd-user mailing list
> drbd-user@lists.linbit.com
> https://lists.linbit.com/mailman/listinfo/drbd-user
>
Re: Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd [ In reply to ]
Hello,
Yeah, I agree with what others in this forum have said. My own experiences have been about the same. As long as you have a well-synchronized clock on your storage nodes and your cron/fcron/whatever scheduler is working correctly, you will do very well with internal metadata and zfs-auto-snapshot with little fuss.

One thing though, I have only used the A protocol when I need max performance and don't care about the data all that much. For 99% of my use cases, I've used the C protocol. The A protocol can result in significant data loss in many cases when your system crashes while the storage is in heavy use. Maybe you could consider using C instead of A if your performance is enough and leave A for specific cases where you know it is ok. Also, in many cases where C has too much latency and A is too risky, you can use the B protocol which is the middleground between the two.



--

Thank you,

David
https://www.jaxport.com/?utm_source=emailsig&utm_medium=email&utm_campaign=emailbranding"] David Bruzos Systems Administrator
Jacksonville Port Authority (JAXPORT)
t. &#43;1 (904) 357-3069 | m. &#43;1 (904) 625-0969 David.Bruzos@jaxport.com | https://www.jaxport.com/?utm_source=emailsig&utm_medium=email&utm_campaign=emailbranding"]JAXPORT.com 2831 Talleyrand Ave, Jacksonville, Florida 32206
http://www.linkedin.com/company/334262"] https://twitter.com/jaxport"] https://www.facebook.com/JacksonvillePortAuthority"] http://instagram.com/jax_port"] https://www.youtube.com/user/JAXPORT"]
http://eepurl.com/gjqoY5"] SUBSCRIBE TO OUR E-NEWSLETTER >> JAXPORT REPORT
________________________________________________________________________________________________

Please note that under Florida&#39;s public records law (F.S. 668.6076), most written communications
to or from the Jacksonville Port Authority are public records, available to the public and media
upon request. Your email communications may therefore be subject to public disclosure. If you have
received this email in error, please notify the sender by return email and delete immediately
without forwarding to others.
Re: Regarding Best way for snapshotting with ZFS/LVM as backing disk for drbd [ In reply to ]
Many Thanks for your valuable insights.

On Wed, 17 Aug 2022 at 00:48, David Bruzos <david.bruzos@jaxport.com> wrote:

> Hello,
> Yeah, I agree with what others in this forum have said. My own experiences
> have been about the same. As long as you have a well-synchronized clock on
> your storage nodes and your cron/fcron/whatever scheduler is working
> correctly, you will do very well with internal metadata and
> zfs-auto-snapshot with little fuss.
>
> One thing though, I have only used the A protocol when I need max
> performance and don't care about the data all that much. For 99% of my use
> cases, I've used the C protocol. The A protocol can result in significant
> data loss in many cases when your system crashes while the storage is in
> heavy use. Maybe you could consider using C instead of A if your
> performance is enough and leave A for specific cases where you know it is
> ok.
> Also, in many cases where C has too much latency and A is too risky, you
> can use the B protocol which is the middleground between the two.
>
> --
>
> Thank you,
>
> David
>
> [image: Jacksonville Port Authority (JAXPORT)]
> <https://www.jaxport.com/?utm_source=emailsig&utm_medium=email&utm_campaign=emailbranding>
> David Bruzos
> Systems Administrator
> Jacksonville Port Authority (JAXPORT)
> t. +1 (904) 357-3069 | m. +1 (904) 625-0969
> David.Bruzos@jaxport.com | JAXPORT.com
> <https://www.jaxport.com/?utm_source=emailsig&utm_medium=email&utm_campaign=emailbranding>
> 2831 Talleyrand Ave, Jacksonville, Florida 32206
> [image: Follow JAXPORT on LinkedIn]
> <http://www.linkedin.com/company/334262> [image: Follow JAXPORT on
> Twitter] <https://twitter.com/jaxport> [image: Like JAXPORT on
> Facebook] <https://www.facebook.com/JacksonvillePortAuthority> [image:
> Follow JAXPORT on Instagram] <http://instagram.com/jax_port> [image:
> Subscribe to JAXPORT on YouTube] <https://www.youtube.com/user/JAXPORT>
>
> SUBSCRIBE TO OUR E-NEWSLETTER >> JAXPORT REPORT
> <http://eepurl.com/gjqoY5>
>
>
> ________________________________________________________________________________________________
>
> Please note that under Florida's public records law (F.S. 668.6076), most
> written communications
> to or from the Jacksonville Port Authority are public records, available
> to the public and media
> upon request. Your email communications may therefore be subject to public
> disclosure. If you have
> received this email in error, please notify the sender by return email and
> delete immediately
> without forwarding to others.
>