Mailing List Archive

Dale wrote:
> Howdy,
>
> I thought I had done this before but it appears I have not.  I want to
> replace a 8TB drive with a 10TB drive and then remove the 8TB drive when
> done but these drives are encrypted with cryptsetup on top of LVM.  I
> looked through my how tos and can't find one for encrypted drives.  I
> did find them for regular LVM and have done that in the past.  While I
> have fresh backups, I don't want to test them if not needed.  Does this
> work just like with LVM or does the encryption change how this is done
> or even if it can be done?
>
> I googled and can't really find a good how to.  Does anyone have a link
> to a 'how to' to do this with encrypted stuff, have notes with the
> sequence of commands that I can go by or is this not doable when it is
> encrypted? 
>
> Thanks.
>
> Dale
>
> :-)  :-) 
>


Since my previous post, I've continued to look for a way to do this on
google, youtube and any place else I can think of.  So far, I have yet
to find a way.  Given no one has replied with info, I'm assuming that no
one here knows or can find a way either.  Since this involves LVM which
I know has this ability, it appears that the encryption part complicates
matters but it should be doable, I have two drives in the current setup
so I at least added a drive once before.  So, I updated my backups,
stopped any changes to the affected LVM and set out to test my theory. 
This is a list of commands in sequence that are from my notes, you may
recall my memory isn't great.  lol 

To replace drive on encrypted LVM. 
pvcreate /dev/sd<*new* drive or partition>
vgextend <group name> /dev/sd<*new* pvdevice above>
pvmove /dev/sd<drive being *removed*>  This takes a while. ;-)

vgreduce crypt /dev/sd<drive being *removed*>
pvremove /dev/sd<drive being *removed*>

To add extra/remaining space on replacement drive if any
vgdisplay <name of vg> and get unused space or PE available if using
--extents option or use 100% option
lvresize -l +100%FREE /dev/<pv name>/vg name> EXAMPLE
lvresize -l +100%FREE /dev/crypt/crypt

If LUKS, resize LUKS with
cryptsetup resize <name of encrypted>
or close and reopen.

resize2fs -p /dev/mapper/<name of lv which should be same as mount
point>  EXAMPLE
resize2fs -p /dev/mapper/crypt

Once this was done, I have a previous 8TB and a larger 10TB drive which
gives me 17TBs of space now.  I plan to buy a 12 or 14TB drive to
replace existing 8TB.  I should be done for a while. 

Now to replace my /home drive which is also close to full.  It's not
encrypted tho. The biggest difference in this and plain LVM, resizing
with cryptsetup or close and reopen.  Keep in mind, while I did all
this, LUKS, cryptsetup, whatever was open.  I'm not sure about this
being done while closed.  I did find one mention that it had to be open
at certain points. 

By the way, if someone wants to take this and make a how to out of it
somewhere, I think it would be great.  This is doable.  I even rebooted
and even tho the drives changed SATA ports, it worked fine.  So, I guess
I did it right, even if no one thought it could be done.  lol 

I really hope this helps someone else.  My brain hurts a little.  :/

Dale

:-)  :-) 

On Wed, Sep 14, 2022 at 1:40 AM Dale <rdalek1967@gmail.com> wrote:
>
> Now to replace my /home drive which is also close to full. It's not
> encrypted tho. The biggest difference in this and plain LVM, resizing
> with cryptsetup or close and reopen. Keep in mind, while I did all
> this, LUKS, cryptsetup, whatever was open. I'm not sure about this
> being done while closed. I did find one mention that it had to be open
> at certain points.
>

If you want to do operations on a DM volume, then it needs to exist,
which in the cryptsetup world means "open."

Most of the details of how to do an operation like moving around
encrypted volumes depends on the layers you're using (LUKS, LVM,
mdadm, whatever), and how they're stacked. If you're running LVM on
top of LUKS then you create new LUKS volumes and add them to LVM and
move stuff around. If you're running LUKS on top of LVM then you add
unencrypted volumes to LVM, extend the LV underlying LUKS, and then
resize the LUKS volume. Last comes resizing whatever filesystems are
on top if desired.

When you want to get advice around these kinds of operations it is
usually necessary to detail exactly how things are set up, because to
Linux they're just layers and you can stack them however you want. A
script that adds a hard drive to your setup will look different if
you're running mdadm on the bottom vs LUKS vs LVM and so on.

Plus if you do things in the wrong order there is actually a chance
that it will work and leave you with half your filesystem on an
encrypted drive and half of it on an unencrypted one, or whatever. DM
doesn't care - it just shuffles around blocks and transforms them as
told, and as long as you stack things up the same way every time it
should still work. It just won't necessarily be doing what you want
it to...

--
Rich

Am Wed, Sep 14, 2022 at 12:40:21AM -0500 schrieb Dale:

> Now to replace my /home drive which is also close to full.  It's not
> encrypted tho. The biggest difference in this and plain LVM, resizing
> with cryptsetup or close and reopen.  Keep in mind, while I did all
> this, LUKS, cryptsetup, whatever was open.

I recently learnt that you can encrypt an unencrypted devices on-the-fly.
Kinda neat. It is also resilient to hard interruptions like power-offs.
See man cryptsetup-reencrypt.

But if you want to replace the drive anyways, it may be faster to set up a
new device and rsync the files in the last step. I always use these kinds of
opportunities to weed out old cruft on my filesystem.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Just because I wrote a word wrong doesn’t mean that I am imcomentent.

Rich Freeman wrote:
> On Wed, Sep 14, 2022 at 1:40 AM Dale <rdalek1967@gmail.com> wrote:
>> Now to replace my /home drive which is also close to full. It's not
>> encrypted tho. The biggest difference in this and plain LVM, resizing
>> with cryptsetup or close and reopen. Keep in mind, while I did all
>> this, LUKS, cryptsetup, whatever was open. I'm not sure about this
>> being done while closed. I did find one mention that it had to be open
>> at certain points.
>>
> If you want to do operations on a DM volume, then it needs to exist,
> which in the cryptsetup world means "open."
>
> Most of the details of how to do an operation like moving around
> encrypted volumes depends on the layers you're using (LUKS, LVM,
> mdadm, whatever), and how they're stacked. If you're running LVM on
> top of LUKS then you create new LUKS volumes and add them to LVM and
> move stuff around. If you're running LUKS on top of LVM then you add
> unencrypted volumes to LVM, extend the LV underlying LUKS, and then
> resize the LUKS volume. Last comes resizing whatever filesystems are
> on top if desired.
>
> When you want to get advice around these kinds of operations it is
> usually necessary to detail exactly how things are set up, because to
> Linux they're just layers and you can stack them however you want. A
> script that adds a hard drive to your setup will look different if
> you're running mdadm on the bottom vs LUKS vs LVM and so on.
>
> Plus if you do things in the wrong order there is actually a chance
> that it will work and leave you with half your filesystem on an
> encrypted drive and half of it on an unencrypted one, or whatever. DM
> doesn't care - it just shuffles around blocks and transforms them as
> told, and as long as you stack things up the same way every time it
> should still work. It just won't necessarily be doing what you want
> it to...
>


I see the point but wasn't aware there was more than one way to do it
with cryptsetup.  It seems there is several options for this.  I was
pretty sure LVM was on bottom and mentioned it in my original post. 
After reading your post, I got to wondering, did I do this the right
way?  So, I started looking to see how to tell for sure.  I used several
LVM type commands but didn't see anything that I recognized anyway. 
Keep in mind, I'm not real sure what I'm looking for either. Then I ran
lsblk -f and found a clue that I've never noticed before. 


sdd                                                                                                    

??sdd1              LVM2_member LVM2 001         
pVnP2i-sj48-3co9-nJpa-9tQr-08pa-9JqASR               
  ??crypt-crypt     crypto_LUKS 2                
6e884aae-9377-49ef-a602-e13cba89a377                 
    ??crypt         ext4        1.0      crypt   
76653316-329f-4747-8fed-fc9b1723bd14      3.5T    79%
/home/dale/Desktop/Crypt


I know that is going to be line wrapped and mess up things but the part
I noticed was the drive partition "sdd1" and "LVM2 member".  On top of
that is crypto.  So, LVM is on bottom.  If that is the case, my pvmove
command should be moving what I think you call "raw data", doesn't
matter if it is encrypted or not, right?  Either way, other than
resizing the files system, I was dealing with LVM layers.  Just in case
it matters, could I have done everything but the file system resize
while it was closed?  It seems it is basically encrypted on the layer
just below the file system to me. 

Your post helped me to have a better idea of what I got here.  By all
means, point out if I got anything wrong.  Biggest thing, it seems to
have worked.  Still, I need more drive space.  :/

Thanks.

Dale

:-)  :-) 

Am Wed, Sep 14, 2022 at 08:55:26AM -0500 schrieb Dale:

> I see the point but wasn't aware there was more than one way to do it
> with cryptsetup.  It seems there is several options for this.  I was
> pretty sure LVM was on bottom and mentioned it in my original post. 

Indeed you did and it confused me at first. Then I gave it some thought and
concluded: why not?

You do it like so:
Block device --,
Block device --+-- LVM --- LUKS --- File system
Block device --'

> After reading your post, I got to wondering, did I do this the right
> way?

Your advantage: only one LUKS header to take care of. That means no extra
crypt management when adding or removing disks, except for resizing the
crypt volume. And there is only a single place of storage for your keys (in
case you ever need to change them).

I’m not sure whether it’s the right™ way. It is *one* way. Perhaps there are
drawbacks that I can’t think of right now.


I would typically have done:
Block device --- LUKS --,
Block device --- LUKS --+-- LVM --- File system
Block device --- LUKS --'

That’s how my NAS works at the moment (with ZFS instead of LVM + filesystem).
But that’s because ZFS didn’t have built-in encryption when I set it up some
years ago. These days I would do:

Block device --,
Block device --+-- ZFS
Block device --'

That’s it. :D Encryption, disk arrays and file system all in one shop.

> So, I started looking to see how to tell for sure.  I used several
> LVM type commands but didn't see anything that I recognized anyway. 
> Keep in mind, I'm not real sure what I'm looking for either. Then I ran
> lsblk -f and found a clue that I've never noticed before. 
>
>
> sdd                                                                                                    
>
> ??sdd1              LVM2_member LVM2 001         
> pVnP2i-sj48-3co9-nJpa-9tQr-08pa-9JqASR               
>   ??crypt-crypt     crypto_LUKS 2                
> 6e884aae-9377-49ef-a602-e13cba89a377                 
>     ??crypt         ext4        1.0      crypt   
> 76653316-329f-4747-8fed-fc9b1723bd14      3.5T    79%
> /home/dale/Desktop/Crypt
>
>
> I know that is going to be line wrapped and mess up things

You could have redacted the long UUIDs which aren’t relavant anyways. I write my mail in mutt and vim, thus I can rewrap paragraphs individually and at will. That way I can paint ASCII art, paste over-long console output or write one-line paragraphs like this one. ;-)

> but the part I noticed was the drive partition "sdd1" and "LVM2 member". 
> On top of that is crypto.  So, LVM is on bottom.  If that is the case, my
> pvmove command should be moving what I think you call "raw data", doesn't
> matter if it is encrypted or not, right?

Yup. This kind of layering is one of the big beauty of Linux for me. It’s
all interchangable and layer X doesn’t care what layer X+1 is doing and vice
versa.

> Just in case it matters, could I have done everything but the file system
> resize while it was closed?  It seems it is basically encrypted on the
> layer just below the file system to me. 

I think so, yes.


PS.: All your LVM threads made me embrace LVM on my PC when I recently
switched it from SATA to NVMe. And because after many years of ignorance, I
finally had an actual use case: my laptop’s root partition became too small
and I had to give it some space from the data partition. In my early Gentoo
years I didn’t use an initrd and didn’t want to, so LVM was never an option.
But when I set up the (then brand-new) laptop, I used Sakaki’s howto for
full-disk encryption, which used an initrd + LVM anyways. This saved the
SSD from a full reformat and rewrite.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

The longer it rains, the better the prospect of nicer weather.

Frank Steinmetzger wrote:
> Am Wed, Sep 14, 2022 at 08:55:26AM -0500 schrieb Dale:
>
>> I see the point but wasn't aware there was more than one way to do it
>> with cryptsetup.  It seems there is several options for this.  I was
>> pretty sure LVM was on bottom and mentioned it in my original post. 
> Indeed you did and it confused me at first. Then I gave it some thought and
> concluded: why not?
>
> You do it like so:
> Block device --,
> Block device --+-- LVM --- LUKS --- File system
> Block device --'
>
>> After reading your post, I got to wondering, did I do this the right
>> way?
> Your advantage: only one LUKS header to take care of. That means no extra
> crypt management when adding or removing disks, except for resizing the
> crypt volume. And there is only a single place of storage for your keys (in
> case you ever need to change them).
>
> I’m not sure whether it’s the right™ way. It is *one* way. Perhaps there are
> drawbacks that I can’t think of right now.

By right way, I was talking about the way I moved things around. It
seems to be working so I guess I did it the right way, for this setup
anyway.  I might have broke things if it was set up some other way. 


>
> I would typically have done:
> Block device --- LUKS --,
> Block device --- LUKS --+-- LVM --- File system
> Block device --- LUKS --'
>
> That’s how my NAS works at the moment (with ZFS instead of LVM + filesystem).
> But that’s because ZFS didn’t have built-in encryption when I set it up some
> years ago. These days I would do:
>
> Block device --,
> Block device --+-- ZFS
> Block device --'
>
> That’s it. :D Encryption, disk arrays and file system all in one shop.
>
>> So, I started looking to see how to tell for sure.  I used several
>> LVM type commands but didn't see anything that I recognized anyway. 
>> Keep in mind, I'm not real sure what I'm looking for either. Then I ran
>> lsblk -f and found a clue that I've never noticed before. 
>>
>>
>> sdd                                                                                                    
>>
>> ??sdd1              LVM2_member LVM2 001         
>> pVnP2i-sj48-3co9-nJpa-9tQr-08pa-9JqASR               
>>   ??crypt-crypt     crypto_LUKS 2                
>> 6e884aae-9377-49ef-a602-e13cba89a377                 
>>     ??crypt         ext4        1.0      crypt   
>> 76653316-329f-4747-8fed-fc9b1723bd14      3.5T    79%
>> /home/dale/Desktop/Crypt
>>
>>
>> I know that is going to be line wrapped and mess up things
> You could have redacted the long UUIDs which aren’t relavant anyways. I write my mail in mutt and vim, thus I can rewrap paragraphs individually and at will. That way I can paint ASCII art, paste over-long console output or write one-line paragraphs like this one. ;-)

I started to but thought it would cause yet another issue.  ROFL 

>> but the part I noticed was the drive partition "sdd1" and "LVM2 member". 
>> On top of that is crypto.  So, LVM is on bottom.  If that is the case, my
>> pvmove command should be moving what I think you call "raw data", doesn't
>> matter if it is encrypted or not, right?
> Yup. This kind of layering is one of the big beauty of Linux for me. It’s
> all interchangable and layer X doesn’t care what layer X+1 is doing and vice
> versa.
>
>> Just in case it matters, could I have done everything but the file system
>> resize while it was closed?  It seems it is basically encrypted on the
>> layer just below the file system to me. 
> I think so, yes.
>
>
> PS.: All your LVM threads made me embrace LVM on my PC when I recently
> switched it from SATA to NVMe. And because after many years of ignorance, I
> finally had an actual use case: my laptop’s root partition became too small
> and I had to give it some space from the data partition. In my early Gentoo
> years I didn’t use an initrd and didn’t want to, so LVM was never an option.
> But when I set up the (then brand-new) laptop, I used Sakaki’s howto for
> full-disk encryption, which used an initrd + LVM anyways. This saved the
> SSD from a full reformat and rewrite.
>


I have to say, before I started using LVM, it was very hard to change
storage up.  As it is, /boot and / are on a traditional partition, no
LVM.  Everything else is on LVM.  At the time, I was avoiding the init
thingy.  Since I have /usr separate, and /var, I ended up having to use
one anyway but dracut came along.  Only gave me a problem once so
overall, pretty good. 

I'm not sure who got me started on LVM.  Could be Alan who isn't on this
list anymore, Neil or someone else.  It's been a long long time ago.  I
tell you what tho, it has made things a LOT easier.  I've read about
btrfs(sp?).  It could be the next big thing but I'll likely stick with
LVM as long as it works for me.  ZFS seems to be getting popular too.
I'm not sure who is winning the race between them tho.  I think they all
each have their use case. 

My biggest thing, I couldn't find hardly anything to do that with it
encrypted and what I did find was not very clear.  I couldn't imagine it
not being doable tho.  After all, this is Linux.  :-) 

I'm just glad it worked out and if anyone runs up on this thread, this
additional info will help someone else.  Now, people can see what the
setup is.  See if it matches what they have or not. 

Dale

:-)  :-) 

P. S.  My second move is almost 90% done with pvmove. 

Frank Steinmetzger wrote:
> Am Wed, Sep 14, 2022 at 12:40:21AM -0500 schrieb Dale:
>
>> Now to replace my /home drive which is also close to full.  It's not
>> encrypted tho. The biggest difference in this and plain LVM, resizing
>> with cryptsetup or close and reopen.  Keep in mind, while I did all
>> this, LUKS, cryptsetup, whatever was open.
> I recently learnt that you can encrypt an unencrypted devices on-the-fly.
> Kinda neat. It is also resilient to hard interruptions like power-offs.
> See man cryptsetup-reencrypt.
>
> But if you want to replace the drive anyways, it may be faster to set up a
> new device and rsync the files in the last step. I always use these kinds of
> opportunities to weed out old cruft on my filesystem.
>


The reason I left /home itself plain, when I leave, I lock the encrypted
parts.  I need the rest of /home plain so I can leave downloads and such
running.  If all of /home was encrypted, then I'd have to stop my
downloads or just store them elsewhere.  The things I want to keep
secure, they are in the encrypted parts. 

I don't keep a lot of cruft around. The only exception, I need to clean
out bookmarks in Seamonkey.  I have a tool that helps in Firefox but
sadly, Seamonkey has few working addon type tools anymore.  Other than
email, I'd like to switch to something else.  Wouldn't mind switching
anyway but I just don't like the options. 

Dale

:-)  :-) 

>By the way, if someone wants to take this and make a how to out of it
>somewhere, I think it would be great. This is doable. I even rebooted
>and even tho the drives changed SATA ports, it worked fine. So, I guess
>I did it right, even if no one thought it could be done. lol
>
>I really hope this helps someone else. My brain hurts a little. :/
>
>Dale
>
>:-) :-)

If you want, you can slap it into your personal page on the wiki and then it will show up in searches when people go looking for how to do things with LVM and/or cryptsetup.

LMP

On 14/09/2022 14:55, Dale wrote:
> I see the point but wasn't aware there was more than one way to do it
> with cryptsetup.  It seems there is several options for this.  I was
> pretty sure LVM was on bottom and mentioned it in my original post.
> After reading your post, I got to wondering, did I do this the right
> way?  So, I started looking to see how to tell for sure.  I used several
> LVM type commands but didn't see anything that I recognized anyway.
> Keep in mind, I'm not real sure what I'm looking for either. Then I ran
> lsblk -f and found a clue that I've never noticed before.

https://raid.wiki.kernel.org/index.php/Asking_for_help#lsdrv

Try running this utility. Note that it's written in Python 2, and
unfortunately doesn't work with Python 3 ...

It'll tell you EXACTLY what you've got, provided it knows about the sort
of layers. (It certainly knows about raid and LVM, probably LUKS too ...)

Cheers,
Wol

Laurence Perkins wrote:
>> By the way, if someone wants to take this and make a how to out of it
>> somewhere, I think it would be great. This is doable. I even rebooted
>> and even tho the drives changed SATA ports, it worked fine. So, I guess
>> I did it right, even if no one thought it could be done. lol
>>
>> I really hope this helps someone else. My brain hurts a little. :/
>>
>> Dale
>>
>> :-) :-)
> If you want, you can slap it into your personal page on the wiki and then it will show up in searches when people go looking for how to do things with LVM and/or cryptsetup.
>
> LMP


I don't have a personal page.  Not that I know of anyway.  Plus, I
haven't wrote a how to in ages. 

Dale

:-)  :-) 

Laurence Perkins wrote:
>> By the way, if someone wants to take this and make a how to out of it
>> somewhere, I think it would be great. This is doable. I even rebooted
>> and even tho the drives changed SATA ports, it worked fine. So, I guess
>> I did it right, even if no one thought it could be done. lol
>>
>> I really hope this helps someone else. My brain hurts a little. :/
>>
>> Dale
>>
>> :-) :-)
> If you want, you can slap it into your personal page on the wiki and then it will show up in searches when people go looking for how to do things with LVM and/or cryptsetup.
>
> LMP


Well, I know now.  For future reference tho, you can also tell from the
output of lsblk -f which I think is installed as part of @system.  It
shows the layers.  Funny tho, I've used lsblk to get UUIDs for a long
while and never once noticed it shows that.  It was after Rich posted
that I noticed it.  I was looking for UUIDs and that was all I saw I
guess.  lol

Always good to have more than one tool to do something tho.  ;-)

Dale

:-)  :-)