Mailing List Archive: [OT] rsync rules question

[OT] rsync rules question

waltdnes at waltdnes

Oct 14, 2020, 6:58 AM

Post #1 of 7 (725 views)

I'd like to keep my "hot backup" machine more up-to-date. The most
important directory is my home directory. So far, I've been doing
occasional tarballs of my home directory and pushing them over. I
exclude some directories for "reasons". I'd like to switch to rsync
and run it more often. I've done the RTFM, but I'd more eyes to check
this out before the first run.

* The script will be run from root.
* It will copy changes and new files from the local /home/waltdnes
directory to /home/waltdnes on machine "d531".
* It will *NOT* delete files on the destination that have been deleted
at the source.
* Directories ".cache", ".ccache", ".ssh", "palemoon", and "pm" will
*NOT* be mirrored.

Any comments on the ruleset below?

#!/bin/bash
rsync --no-devices --no-specials --compress --keep-dirlinks --links \
--group --itemize-changes --owner --perms --progress \
--recursive --stats --times --verbose \
--filter='- .cache/' \
--filter='- .ccache/' \
--filter='- .ssh/' \
--filter='- palemoon/' \
--filter='- pm/' \
--filter='+ /' \
--filter='+ /home/' \
--filter='+ /home/waltdnes/' \
--filter='+ /home/waltdnes/***' \
--filter='- *' \
--log-file=/root/rsync_log.txt \
/home/waltdnes waltdnes@d531:/ 2>/root/rserrors.txt

--
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [OT] rsync rules question [ In reply to ]

antlists at youngman

Oct 14, 2020, 10:51 AM

Post #2 of 7 (725 views)

On 14/10/2020 14:58, Walter Dnes wrote:
> I'd like to keep my "hot backup" machine more up-to-date. The most
> important directory is my home directory. So far, I've been doing
> occasional tarballs of my home directory and pushing them over. I
> exclude some directories for "reasons". I'd like to switch to rsync
> and run it more often. I've done the RTFM, but I'd more eyes to check
> this out before the first run.

I haven't yet set up my system, but a couple of tweaks I'm planning to
add ...

Either use btrfs, or an lvm partition, so you can take snapshots.

Do an in-place rsync (ie if part of a file has changed, it only writes
the bit that's changed).

That way, you get a full backup for the price of an incremental.

The main change from your approach is that this will keep both an old
and new copy of any file that has changed.

The big change you could make from your approach is that you CAN delete
files from the backup if they've been deleted on the live system,
without losing them.

Horses for courses, but if you are planning to keep your backup
long-term, this could do a good job, provided you remember when you
deleted that lost file from your live system :-)

I'm planning to back up 6TB of live data onto a 12TB shingled disk,
which shouldn't be a problem, and given that not much changes (apart
from adding new photos), each backup will probably use a few gigs at
most. Dunno how long the disk will last, but it should be ages.

Cheers,
Wol

Re: [OT] rsync rules question [ In reply to ]

waltdnes at waltdnes

Oct 14, 2020, 12:10 PM

Post #3 of 7 (725 views)

On Wed, Oct 14, 2020 at 06:51:39PM +0100, antlists wrote
>
> Horses for courses, but if you are planning to keep your backup
> long-term, this could do a good job, provided you remember when you
> deleted that lost file from your live system :-)

I'm not looking for long-term backups. I have other methods for that.
The whole point of a "hot backup" is to have another machine ready to go
if/when the main machine dies. That means a /home/<userid> on the
backup machine with all the files from the main machine.

--
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [OT] rsync rules question [ In reply to ]

neil at digimed

Oct 14, 2020, 12:22 PM

Post #4 of 7 (725 views)

On Wed, 14 Oct 2020 15:10:59 -0400, Walter Dnes wrote:

> > Horses for courses, but if you are planning to keep your backup
> > long-term, this could do a good job, provided you remember when you
> > deleted that lost file from your live system :-)
>
> I'm not looking for long-term backups. I have other methods for that.
> The whole point of a "hot backup" is to have another machine ready to go
> if/when the main machine dies. That means a /home/<userid> on the
> backup machine with all the files from the main machine.

Try net-misc/unison. I use it to keep the home directories, or at least
the important parts of them, in sync between my desktop and laptop. You
can include or exclude paths to copy from its config file. I run it
manually, from a hot key so I've got in the habit of doing it often, but
you can call it from cron.

Alternatively, for more instant updates, you could look at
net-p2p/syncthing.

--
Neil Bothwick

The cow is nothing but a machine that makes grass fit for us people to
eat.

Re: [OT] rsync rules question [ In reply to ]

antlists at youngman

Oct 14, 2020, 12:48 PM

Post #5 of 7 (725 views)

On 14/10/2020 20:22, Neil Bothwick wrote:
> Alternatively, for more instant updates, you could look at
> net-p2p/syncthing.

Not sure what you use for it, but there are mirrored filesystems that
run over a network. So the two systems will update in sync if they're
both switched on, or will defer updates and sync as soon as the second
system powers up.

Cheers,
Wol

Re: [OT] rsync rules question [ In reply to ]

gentoo at jonesmz

Oct 14, 2020, 1:19 PM

Post #6 of 7 (725 views)

On Wed, Oct 14, 2020 at 2:48 PM antlists <antlists@youngman.org.uk> wrote:

> On 14/10/2020 20:22, Neil Bothwick wrote:
> > Alternatively, for more instant updates, you could look at
> > net-p2p/syncthing.
>
> Not sure what you use for it, but there are mirrored filesystems that
> run over a network. So the two systems will update in sync if they're
> both switched on, or will defer updates and sync as soon as the second
> system powers up.
>
> Cheers,
> Wol
>
>
Could you list a few examples to help people find filesystems like this?

Re: [OT] rsync rules question [ In reply to ]

neil at digimed

Oct 14, 2020, 4:53 PM

Post #7 of 7 (725 views)

On Wed, 14 Oct 2020 20:48:04 +0100, antlists wrote:

> On 14/10/2020 20:22, Neil Bothwick wrote:
> > Alternatively, for more instant updates, you could look at
> > net-p2p/syncthing.
>
> Not sure what you use for it, but there are mirrored filesystems that
> run over a network. So the two systems will update in sync if they're
> both switched on, or will defer updates and sync as soon as the second
> system powers up.

I mainly use syncthing to make files available on mobile devices, and
vice versa. I use unison to sync between Linux computers and use it
manually, as I can reverse individual transfers in case of an oops moment.

--
Neil Bothwick

Irritable? Who the bloody hell are you calling irritable?