Mailing List Archive

On Wed, 20 Sep 2023 20:24:17 -0000 (UTC), Grant Edwards wrote:

> However, GParted apparently does. Can GParted move an ext4 filesystem
> to a destination location that overlaps its starting location?
>
> For example, I have a 500GB partition containing an ext4 filesystem
> starting at sector 2048 (1MiB). I want to move that filesystem so that
> it starts at sector 3*2048 (3MiB).
>
> Can that be done in-place?
>
> Or should I just back up the filesystem to a second drive and start
> from scratch?

Given that you'd want to backup before such an operation anyway, you may
as well then restore from that backup. I'm sure it will be a lot quicker
than GParted's moving all the data around.


--
Neil Bothwick

He who laughs last thinks slowest!

On 2023-09-20, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 20 Sep 2023 20:24:17 -0000 (UTC), Grant Edwards wrote:
>
>> For example, I have a 500GB partition containing an ext4 filesystem
>> starting at sector 2048 (1MiB). I want to move that filesystem so that
>> it starts at sector 3*2048 (3MiB).
>>
>> Can that be done in-place?
>>
>> Or should I just back up the filesystem to a second drive and start
>> from scratch?
>
> Given that you'd want to backup before such an operation anyway,

It's a machine with very limited uses, so I'd probably only back up
/etc and /root. Reinstalling probably wouldn't take too much longer
than backing up and restoring /.

> you may as well then restore from that backup. I'm sure it will be a
> lot quicker than GParted's moving all the data around.

That depends on how long it takes me to decide on tar vs. rsync and
what the appropriate options are. After 40 years using Unix, you'd
think I'd know that (or have it written down somewhere). :)

That said, I think I will go with the backup, repartition, restore
method. It's been many, many years since I used GParted, and I can
probably have the whole job done from the command-line before I can
figure out how the GParted GUI works.

On 20/09/2023 22:16, Neil Bothwick wrote:
> On Wed, 20 Sep 2023 20:24:17 -0000 (UTC), Grant Edwards wrote:
>
>> However, GParted apparently does. Can GParted move an ext4 filesystem
>> to a destination location that overlaps its starting location?
>>
>> For example, I have a 500GB partition containing an ext4 filesystem
>> starting at sector 2048 (1MiB). I want to move that filesystem so that
>> it starts at sector 3*2048 (3MiB).
>>
>> Can that be done in-place?
>>
>> Or should I just back up the filesystem to a second drive and start
>> from scratch?
>
> Given that you'd want to backup before such an operation anyway, you may
> as well then restore from that backup. I'm sure it will be a lot quicker
> than GParted's moving all the data around.
>
Or, assuming the people who wrote gparted have two brain cells to rub
together, I'm pretty sure they use the same technique as memmove.

"If the regions overlap, make sure you start from whichever end won't
overwrite the source, otherwise start at whichever end you like".

Barring screw-ups (a very unsafe assumption :-), I'm pretty certain you
don't even need a backup!

I suspect the man-page even confirms this behaviour.

Cheers,
Wol

On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
> That depends on how long it takes me to decide on tar vs. rsync and
> what the appropriate options are.

I've done this a number of times for various reasons over the last 1-2
years, most recently a few months ago due to hard drive swap, and I
find tar works just fine:

$ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Likewise to extract, but make sure "--xattrs" is present

Provided backup space isn't an issue, I wouldn't bother with
compression. It could be a lot quicker too depending on the size of
your root partition.

> That said, I think I will go with the backup, repartition, restore
> method.
Sounds like a sound course of action to me, reinstalling is just too
much faff and can often be avoided. I have a long standing distrust of
partition management utilities that claim to move filesystem data too.
I'm sure they work juuuuust fine :)

Just make sure you update /etc/fstab and bootloader config file with
the new filesystem UUID or partition indices.

Regards,
V

Am Wed, Sep 20, 2023 at 10:57:00PM +0100 schrieb Victor Ivanov:

> On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> >
> > That depends on how long it takes me to decide on tar vs. rsync and
> > what the appropriate options are.
>
> I've done this a number of times for various reasons over the last 1-2
> years, most recently a few months ago due to hard drive swap, and I
> find tar works just fine:
>
> $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .

Does that stop at file system boundaries (because you tar up '/')? I think
it must be, otherwise you wouldn’t use it that way.
But when copying a root file system, out of habit I first bind-mount it in a
subdirectory and tar/rsync from there instead. This will also make files
visible which might be hidden under an active mount.

This is not necessary if you do it from a live system, but then you wouldn’t
tar up / in the first place.

> Likewise to extract, but make sure "--xattrs" is present
>
> Provided backup space isn't an issue, I wouldn't bother with
> compression. It could be a lot quicker too depending on the size of
> your root partition.

Or not, depending on the speed of the backup device. ;-)
LZO compression (or zstd with a low setting) has negligible CPU cost, but
can lower the file size quite nicely, specially with large binaries or debug
files.

--
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Do you steel taglines, too?

On 2023-09-20, Wol <antlists@youngman.org.uk> wrote:

> Or, assuming the people who wrote gparted have two brain cells to rub
> together, I'm pretty sure they use the same technique as memmove.
>
> "If the regions overlap, make sure you start from whichever end won't
> overwrite the source, otherwise start at whichever end you like".
>
> Barring screw-ups (a very unsafe assumption :-), I'm pretty certain you
> don't even need a backup!
>
> I suspect the man-page even confirms this behaviour.

Not that I could see. The only mention of "move" on the man page is
in this list of features.

With gparted you can accomplish the following tasks:
- Create a partition table on a disk device.
- Enable and disable partition flags such as boot and hidden.
- Perform actions with partitions such as create, delete,
resize, move, check, label, copy, and paste.

Assuming GParted is smart enough to do overlapping moves, is it smart
enough to only copy filesystem data and not copy "empty" sectors?
According to various forum posts, it is not: moving a partion copies
every sector. [That's certainly the obvious, safe thing to do.]

The partition in question is 200GB, but only 7GB is used, so I think
backup/restore is the way to go...

--
Grant

On 2023-09-20, Victor Ivanov <vic.m.ivanov@gmail.com> wrote:
> On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>
>> That depends on how long it takes me to decide on tar vs. rsync and
>> what the appropriate options are.
>
> I've done this a number of times for various reasons over the last 1-2
> years, most recently a few months ago due to hard drive swap, and I
> find tar works just fine:
>
> $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
>
> Likewise to extract, but make sure "--xattrs" is present

Yep, that's pretty much what I decided on based on the tar command
shown at

https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage

Interestingly, the Arch Linux Wiki recommends using bsdtar because
"GNU tar with --xattrs will not preserve extended attributes".

> Provided backup space isn't an issue, I wouldn't bother with
> compression. It could be a lot quicker too depending on the size of
> your root partition.

Both the drive being "fixed" and the backup drive are in a USB3
attached dual slot drive dock, so I'm thinking compression might be
worthwhile.

> Just make sure you update /etc/fstab and bootloader config file with
> the new filesystem UUID or partition indices.

I always forget one or the other until after I try to boot the first
time. That's why I keep systemrescuecd and Gentoo minimal install
USB drives on hand.

--
Grant

On Wed, 20 Sep 2023 21:28:29 -0000 (UTC), Grant Edwards wrote:

> > you may as well then restore from that backup. I'm sure it will be a
> > lot quicker than GParted's moving all the data around.
>
> That depends on how long it takes me to decide on tar vs. rsync and
> what the appropriate options are. After 40 years using Unix, you'd
> think I'd know that (or have it written down somewhere). :)

If you written everything down, after 40 years you'd have so many bits of
paper you wouldn't be able to find anything ;-)

> That said, I think I will go with the backup, repartition, restore
> method. It's been many, many years since I used GParted, and I can
> probably have the whole job done from the command-line before I can
> figure out how the GParted GUI works.

:)


--
Neil Bothwick

The facts, although interesting, are usually irrelevant.

On 2023-09-20, Frank Steinmetzger <Warp_7@gmx.de> wrote:
> Am Wed, Sep 20, 2023 at 10:57:00PM +0100 schrieb Victor Ivanov:
>
>> On Wed, 20 Sept 2023 at 22:29, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>> >
>> > That depends on how long it takes me to decide on tar vs. rsync and
>> > what the appropriate options are.
>>
>> I've done this a number of times for various reasons over the last 1-2
>> years, most recently a few months ago due to hard drive swap, and I
>> find tar works just fine:
>>
>> $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
>
> Does that stop at file system boundaries (because you tar up '/')? I think
> it must be, otherwise you wouldn’t use it that way.
> But when copying a root file system, out of habit I first bind-mount it in a
> subdirectory and tar/rsync from there instead. This will also make files
> visible which might be hidden under an active mount.

The partition/fs being backed up isn't live (it's mounted, but it's
not the root partition of the host doing the backup), so nothing is
mounted within it and there aren't any /proc or /sys entries in
it. So, in this case there's no need to worry about crossing
filesystem boundaries.

--
Grant

On Thu, 21 Sept 2023 at 02:01, Frank Steinmetzger <Warp_7@gmx.de> wrote:
> > $ tar -cpf /path/to/backup.tar --xattrs --xattrs-include='*.*' -C / .
>
> Does that stop at file system boundaries (because you tar up '/')? I think
> it must be, otherwise you wouldn’t use it that way.
No, it doesn't. It will archive everything recursively including
mounted directories, so fair point for raising this. It's something
that I do not normally consider, as I tend not to do full root backup
on the running system itself. But how cool is using bind mount for
this situation as you suggest? Simple and effective, I like it.

Actually, the presence of -C in the above example is purely out of
habit. I use it when pointing to a full path, e.g. "/path/to/dir" so
it doesn't end up creating the "path/to/dir" path prefix inside the
archive and only archives the contents of the path.

It's effectively changing to that directory and archiving everything
there, but saves you from doing the "cd".

Naturally, for "/" this is superfluous and "-C / ." can be replaced
with just "/".

> > Provided backup space isn't an issue, I wouldn't bother with
> > compression. It could be a lot quicker too depending on the size of
> > your root partition.
>
> Or not, depending on the speed of the backup device. ;-)
> LZO compression (or zstd with a low setting) has negligible CPU cost, but
> can lower the file size quite nicely, specially with large binaries or debug
> files.
>
That's true :) I had somehow forgotten of tar's support for LZO and
zstd as my default finger memory approach is to use -J for xz. Good
memory nudge here!

Regards,
V

On Wed, 20 Sept 2023 at 23:58, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> Yep, that's pretty much what I decided on based on the tar command
> shown at
>
> https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Stage
>
> Interestingly, the Arch Linux Wiki recommends using bsdtar because
> "GNU tar with --xattrs will not preserve extended attributes".
I remember coming across this too as I've previously had issues
preserving some extended attributes, notably on files under my home
directory. However, I found that using "--xattrs-include='*.*'" in
addition to "--xattrs" works pretty well and does record attributes
that would otherwise be excluded with just "--xattrs".

I cannot comment, however, if it truly includes "everything" in every
possible scenario.

>
> Both the drive being "fixed" and the backup drive are in a USB3
> attached dual slot drive dock, so I'm thinking compression might be
> worthwhile.
>
Then LZO or zstd might indeed be a better approach as suggested by Frank.

> > Just make sure you update /etc/fstab and bootloader config file with
> > the new filesystem UUID or partition indices.
>
> I always forget one or the other until after I try to boot the first
> time. That's why I keep systemrescuecd and Gentoo minimal install
> USB drives on hand.

Me too, even just recently when I migrated my OS to another build I
decided to do a few partition touch ups and fell once more into this
trap. I updated fstab but not the bootloader. Luckily, Gentoo minimal
install image is so tiny a bootable medium can literally be created in
minutes.

Good luck!

Regards,
V

On 2023-09-21, Victor Ivanov <vic.m.ivanov@gmail.com> wrote:
> On Wed, 20 Sept 2023 at 23:58, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
>>> Just make sure you update /etc/fstab and bootloader config file
>>> with the new filesystem UUID or partition indices.
>>
>> I always forget one or the other until after I try to boot the
>> first time. That's why I keep systemrescuecd and Gentoo minimal
>> install USB drives on hand.
>
> Me too, even just recently when I migrated my OS to another build I
> decided to do a few partition touch ups and fell once more into this
> trap. I updated fstab but not the bootloader. Luckily, Gentoo
> minimal install image is so tiny a bootable medium can literally be
> created in minutes.

The tar backup restore worked just fine (and didn't take long, even
though both drives were connected via USB). I've since fixed a second
machine by adding a bios-boot partition. I should have started using
them when I switched from MBR to GPT, but I think I got bios-boot
partitions confused with UEFI boot partitions. :/

I'm also working on switching to using either labels or uuids in fstab
and grub configs so that changes in partition numbers don't cause
problems. Of course I've discovered for the Nth time in the past
10-15 years, that for the root= command line argument, the kernel
doesn't grok LABEL or UUID values -- it only understands device names
and PARTUUID.

--
Grant

On 9/21/23 16:23, Grant Edwards wrote:
> On 2023-09-21, Victor Ivanov <vic.m.ivanov@gmail.com> wrote:
>> On Wed, 20 Sept 2023 at 23:58, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>>> Just make sure you update /etc/fstab and bootloader config file
>>>> with the new filesystem UUID or partition indices.
>>> I always forget one or the other until after I try to boot the
>>> first time. That's why I keep systemrescuecd and Gentoo minimal
>>> install USB drives on hand.
>> Me too, even just recently when I migrated my OS to another build I
>> decided to do a few partition touch ups and fell once more into this
>> trap. I updated fstab but not the bootloader. Luckily, Gentoo
>> minimal install image is so tiny a bootable medium can literally be
>> created in minutes.
> The tar backup restore worked just fine (and didn't take long, even
> though both drives were connected via USB). I've since fixed a second
> machine by adding a bios-boot partition. I should have started using
> them when I switched from MBR to GPT, but I think I got bios-boot
> partitions confused with UEFI boot partitions. :/
>
> I'm also working on switching to using either labels or uuids in fstab
> and grub configs so that changes in partition numbers don't cause
> problems. Of course I've discovered for the Nth time in the past
> 10-15 years, that for the root= command line argument, the kernel
> doesn't grok LABEL or UUID values -- it only understands device names
> and PARTUUID.
while my Gentoo grub.cfg has root=PARTUUID=, my Artix Linux install
(using openrc) has root=UUID=.  I wasn't aware they had mucked with grub
(2.12-rc1) nor do I know if it's a recent change in grub.

On 2023-09-21, Jack <ostroffjh@users.sourceforge.net> wrote:
>
>> [...] Of course I've discovered for the Nth time in the past 10-15
>> years, that for the root= command line argument, the kernel doesn't
>> grok LABEL or UUID values -- it only understands device names and
>> PARTUUID.
>
> while my Gentoo grub.cfg has root=PARTUUID=, my Artix Linux install
> (using openrc) has root=UUID=.  I wasn't aware they had mucked with
> grub (2.12-rc1) nor do I know if it's a recent change in grub.

AFAIK, it's not grub (grub does know how to handle LABEL and UUID when
setting grub's root). For the kernel, it's something in the initrd's
'init' executable that parses the root=UUID= or root=LABEL=, searches
the available filesystems to find a match, then mounts the matching
filesystem and does a chroot to it (or someting like that).

If you don't have an initrd, then the kernel itself has to handle the
root=<whatever> and that code only knows about device names and
partition UUIDs. It doesn't know anything about filesystems (which
doesn't make much sense, since the next step is to mount the specified
partition, and it obviously knows about filesystems at that point).

At least that's what I've read...

On 20/09/2023 23:39, Grant Edwards wrote:
> Assuming GParted is smart enough to do overlapping moves, is it smart
> enough to only copy filesystem data and not copy "empty" sectors?
> According to various forum posts, it is not: moving a partion copies
> every sector. [That's certainly the obvious, safe thing to do.]

Seeing as it knows nothing about filesystems, and everything about
partitions, it will treat the partition as an opaque blob and move it as
a single object ...
>
> The partition in question is 200GB, but only 7GB is used, so I think
> backup/restore is the way to go...

You would think so :-)

I use ext4, and make heavy use of hard links. Last time I tried a
straight copy (not backup/restore) I think the copied partition would
have been three times the size of the original - that is if it hadn't
run out of space first :-)

But it sounds like that would work well for you.

Cheers,
Wol

Den 22.09.2023 08:48, skrev Wols Lists:
> On 20/09/2023 23:39, Grant Edwards wrote:
>> Assuming GParted is smart enough to do overlapping moves, is it smart
>> enough to only copy filesystem data and not copy "empty" sectors?
>> According to various forum posts, it is not: moving a partion copies
>> every sector. [That's certainly the obvious, safe thing to do.]
>
> Seeing as it knows nothing about filesystems, and everything about
> partitions, it will treat the partition as an opaque blob and move it
> as a single object ...
>>
>> The partition in question is 200GB, but only 7GB is used, so I think
>> backup/restore is the way to go...
>
> You would think so :-)
>
> I use ext4, and make heavy use of hard links. Last time I tried a
> straight copy (not backup/restore) I think the copied partition would
> have been three times the size of the original - that is if it hadn't
> run out of space first :-)
>
Just for completeness, you should check out the e2image command, though
NOT for direct in-place overlapping move. Something like "e2image -ra
$sourcevol $targetvol" has worked very well for me. That is obviously
NOT for overlapping move, but for a backup it is quite fast. If you are
moving to a bigger volume, I'd do resize as a separate step after the move.