Mailing List Archive

On 15/08/2022 08:52, Dale wrote:
> I just hope this 10TB drive isn't a SMR.  I googled around and the best
> I could find is anything above 8TB is CMR.  It's a WD101EDBZ-11B1DA0.  I
> hope that is right.  I'm not totally opposed to SMR even as a backup but
> I'd rather not.  The deal I found was for a pull and costs about $110
> including shipping.  I looked at a 14TB but my jaw dropped.  $$$$$$$$

Just done a search for you (it helps I know what I'm looking for), but
CMR it is ...

https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/

I tend to go for OEM stuff, and know the names of the range I'm looking
for - Seagate Ironwolf, Toshiba N300 - about £170 for 8TB ...

Cheers,
Wol

On Monday, August 15, 2022 9:52:26 AM CEST Dale wrote:
> Julien Roy wrote:
> > Hello,
> >
> > On 8/14/22 18:44, Dale wrote:
> >> Thoughts? Ideas?
> >
> > You might be interested in borgbackup [1]
> > It takes delta backups and has de-duplication and compression to save
> > some space. It supports encryption too.
> > It's packaged in ::gentoo and you run it on whatever machine you want
> > to backup and give it its destination, it can be local or on a remote
> > machine.
> >
> > I've been using it for a while and it works well. I have it configured
> > on a crontab and it backups my files every night
> >
> > [1] https://www.borgbackup.org/
>
> Since my drives are external, I do my backups manually. Well, I start
> it when the drives are connected and ready. I think borgbackup was one
> I looked into and it sounded more like a online backup where you store
> the data on a server somewhere. I may be wrong on that tho. I looked
> at several and it got confusing after a bit. Plus, some were still as
> clear as mud. Why do people link to a place that doesn't tell what
> their software does and how anyway. It seems most think github and such
> are good places to link to when it really doesn't tell you anything
> unless you want to help develop the software or something. It would be
> like Ford linking to CAD models to sell cars. :/
>
> To all: I found a good deal on a 10TB drive. That should suffice for
> now. I might add, it will give me time to figure out a path forward and
> I can make other use of that SMR drive. One thing I thought of as a
> negative for a NAS, I can't lock it into my safe, unless it is really
> tiny. As it is, even if a fire comes along, I still got backups.

I looked into this as well. A safe works like a nice little oven and the
temperature inside the safe can go really high if it's inside a fire.
Not all storage system (HDDs included) are reliable when the temperatures go
to these extremes.

Make sure the safe is, apart from resistent to fire, also capable of keeping
the heat outside.

--
Joost

On Mon, Aug 15, 2022 at 2:32 PM Wol <antlists@youngman.org.uk> wrote:
>
> On 15/08/2022 11:11, Rich Freeman wrote:
> > I see lots of talk of NAS and zfs/btrfs and snapshots. IMO these are
> > NOT really great solutions for backup. NAS can work of course but it
> > is overkill for backup storage.
>
> Do you want multiple *independent* backups, or do you want *incremental*
> backups so you can go back in time. It's nice to have both, but
> snapshotting gives you full backups for the price of incremental.

Snapshots don't give you backups at all, unless you first make a
snapshot and then take a backup (full or incremental) of the snapshot,
or serialize them using a send-like mechanism (which can also be full
or incremental).

If you destroy the drives containing the original copy, then you
destroy all the snapshots as well.

COW snapshots are great, but they're more about how you handle your
LIVE data. They don't address the same failure modes as backups. I
use both.

--
Rich

On Mon, Aug 15, 2022 at 2:34 PM J. Roeleveld <joost@antarean.org> wrote:
>
> Actually, there still is a piece of software that does this:
> " app-backup/dar "
> You can tell it to split the backups into slices of a specific size.

dar is a great tool, but unless something changed I don't think you
can tell it to pause to remount the destination directory when it
fills up. As was pointed out, tar does do this (which I thought was
limited to tape devices, but apparently it works for disk as well).

For most backup tools, if you want to backup 100TB of data to a bunch
of 10TB drives, you basically need to have 100TB of scratch storage
lying around to hold the backup before you split it up, or be able to
mount all your drives at the same time in some kind of combined
filesystem.

--
Rich

J. Roeleveld wrote:
> On Monday, August 15, 2022 12:44:11 AM CEST Dale wrote:
>> Howdy,
>>
>> With my new fiber internet, my poor disks are getting a work out, and
>> also filling up. First casualty, my backup disk. I have one directory
>> that is . . . well . . . huge. It's about 7TBs or so. This is where it
>> is right now and it's still trying to pack in files.
>>
>> /dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
> <snipped>
>
>> Thoughts? Ideas?
> Plenty, see below:
>
> For backups to external disks, I would recommend having a look at "dar" :
> $ eix -e dar
> * app-backup/dar
> Available versions: 2.7.6^t ~2.7.7^t {argon2 curl dar32 dar64 doc gcrypt
> gpg lz4 lzo nls rsync threads xattr}
> Homepage: http://dar.linux.free.fr/
> Description: A full featured backup tool, aimed for disks
>
> It's been around for a while and the developer is active and responds quite
> well to questions.
> It supports compression (different compression methods), incremental backups
> (only need a catalogue of the previous backup for the incremental) and
> encryption.
>
> The NAS options others mentioned would also work as they can compress data on
> disk and you'd only notice a delay in writing/reading (depending on the
> compression method used). I would recommend using one that uses ZFS on-disk as
> it's more reliable and robust then BTRFS.
>
> One option that comes available for you now that you are no longer limited to
> slow ADSL: Cloud backups.
>
> I use Backblaze (B2) to store compressed backups that haven't been stored on
> tape to off-site locations.
>
> But, you can also encrypt the backups locally and store the
> encrypted+compressed backupfiles on other cloud storage.
>
> --
> Joost
>


Dar does sound interesting.  It sounds a lot like what I used way back
in the 90's.  I'm sure it is different software but could work on
floppies then like it does on USB sticks etc today.  Same principal. 

I looked into ZFS as well.  Google helped me find a interesting page.  I
notice it is also used on some NAS setups as well.  It seems to be
advanced and maintained well.  It sounds a little like LVM but may have
more features, such as compression maybe?  I haven't read that far yet. 
I notice it mentions snapshots which LVM also uses. 

Getting plenty of ideas.  I just wish I had a separate building to put a
NAS in that would be safe and climate controlled.  I got a out building
but it gets plenty hot in the summer.  No A/C or anything.  I only heat
it enough to prevent freezing but computers would likely like that anyway. 

Dale

:-)  :-) 

Mark Knecht wrote:
>
>
> On Sun, Aug 14, 2022 at 4:21 PM Dale <rdalek1967@gmail.com
> <mailto:rdalek1967@gmail.com>> wrote:
> >
> > Mark Knecht wrote:
> >
> >
> >
>
> >
> > Do you happen to have an old computer laying around? If so check
> > out TrueNAS Core.
> <SNIP>
> > That may be a option later.  I'm actually considering build a NAS
> but right now, costs are preventing that.  I almost have enough that I
> could build another computer.  I have a mobo, memory, CPU and such.  I
> think I only need a power supply and maybe a video card.  Could use a
> case for it to but could mount it on a wall somewhere.  Good air flow.
>  lol
> <SNIP>
> > This new fiber thing is going to take some getting used too.  ;-)
>
> I experienced much of the same thing (more data) when my connection
> got faster. 
>
> Expense of a separate system to build a NAS is always an issue and
> you've received excellent guidance from other folks here about how to
> do it locally so I think you're set.
>
> A couple of things:
>
> 1) I didn't see mentioned so I will - the NAS, being on the network,
> is connected over gigabit Ethernet in my case so backups are
> significantly faster than using USB drives, or at least much faster
> than my older USB. I get about 800mbit/Sec sustained transfers. Once
> you get the main backup done the incremental ones are very fast. (Go
> to the kitchen fast) 
>
> 2) The NAS, when attached, is mounted over NFS as a directory and I
> use rsync to do the transfers so it's all very familiar on the client
> side. I think that's important to you today but likely won't be as
> much of an issue if you get used to some new backup application.
>
> 3) Compression is done on the NAS and is transparent from the client
> side. I can browse directories and retrieve individual files. As I
> think you mentioned you won't get much compression - close to zero -
> for movies but for my general data and VMs overall I'm getting about
> 40% so there's a big disk saving. Compute requirements are pretty low.
> I bought a used MB with a 6th gen i5 Core processor with 4 cores and
> it hardly works to do the compression. 
>
> Good luck with whatever you do.
>
> Mark


As it is, I have several options.  In a way, I wish I could tell rsync
to do 1st half of alphabet to one drive and then with next command tell
it to do the 2nd half of alphabet.  That would likely split the data in
half for each one.  If videos can't be compressed, that would be the
best idea, outside a NAS that works as a backup.  On that note, power
from the wall wise, I like the Raspberry Pi idea.  Dang things are
pretty powerful and pull very little power.  I think 20 watts or so at
most for the board itself then add a little for hard drive power.  I
can't imagine a complete setup pulling more than 50 or 75 watts when
idle and maybe even when really busy. 

I need to do something but not sure what yet.  :/   USPS says larger
drive will be here Friday.  That should give me time to run SMART tests
and then update backups Saturday or Sunday.  I already planned what goes
where.  Then I'll have a spare 3TB drive.  I really wish I had bought
several more Rosewill external drive enclosures.  So far, I have yet to
have a drive problem while using one of those.  The eSATA ports are nice
too. 

Dale

:-)  :-) 

On Monday, August 15, 2022 8:56:30 PM CEST Rich Freeman wrote:
> On Mon, Aug 15, 2022 at 2:34 PM J. Roeleveld <joost@antarean.org> wrote:
> > Actually, there still is a piece of software that does this:
> > " app-backup/dar "
> > You can tell it to split the backups into slices of a specific size.
>
> dar is a great tool, but unless something changed I don't think you
> can tell it to pause to remount the destination directory when it
> fills up. As was pointed out, tar does do this (which I thought was
> limited to tape devices, but apparently it works for disk as well).

Actually, you can with the "-p / --pause" option.
Also, as per the man-page, if you forget this, the process will simply inform
you the target location is full and you can move slices away to a different
location:
"
If the destination filesystem is too small to contain all the slices of the
backup, the -p option (pausing before starting new slices) might be of
interest. Else, in the case the filesystem is full, dar will suspend the
operation, asking for the user to make free space, then continue its
operation. To make free space, the only thing you cannot do is to touch the
slice being written.
"

The pause-option will actually stop between slices and you can umount the
target location and mount a different disk there.

This option has been around for a while.

--
Joost

On Monday, August 15, 2022 9:07:41 PM CEST Dale wrote:
> J. Roeleveld wrote:
> > On Monday, August 15, 2022 12:44:11 AM CEST Dale wrote:
> >> Howdy,
> >>
> >> With my new fiber internet, my poor disks are getting a work out, and
> >> also filling up. First casualty, my backup disk. I have one directory
> >> that is . . . well . . . huge. It's about 7TBs or so. This is where it
> >> is right now and it's still trying to pack in files.
> >>
> >> /dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
> >
> > <snipped>
> >
> >> Thoughts? Ideas?
> >
> > Plenty, see below:
> >
> > For backups to external disks, I would recommend having a look at "dar" :
> > $ eix -e dar
> > * app-backup/dar
> >
> > Available versions: 2.7.6^t ~2.7.7^t {argon2 curl dar32 dar64 doc
> > gcrypt
> >
> > gpg lz4 lzo nls rsync threads xattr}
> >
> > Homepage: http://dar.linux.free.fr/
> > Description: A full featured backup tool, aimed for disks
> >
> > It's been around for a while and the developer is active and responds
> > quite
> > well to questions.
> > It supports compression (different compression methods), incremental
> > backups (only need a catalogue of the previous backup for the
> > incremental) and encryption.
> >
> > The NAS options others mentioned would also work as they can compress data
> > on disk and you'd only notice a delay in writing/reading (depending on
> > the compression method used). I would recommend using one that uses ZFS
> > on-disk as it's more reliable and robust then BTRFS.
> >
> > One option that comes available for you now that you are no longer limited
> > to slow ADSL: Cloud backups.
> >
> > I use Backblaze (B2) to store compressed backups that haven't been stored
> > on tape to off-site locations.
> >
> > But, you can also encrypt the backups locally and store the
> > encrypted+compressed backupfiles on other cloud storage.
> >
> > --
> > Joost
>
> Dar does sound interesting. It sounds a lot like what I used way back
> in the 90's. I'm sure it is different software but could work on
> floppies then like it does on USB sticks etc today. Same principal.

If it was during the 90's, then it wasn't. First version was released in 2002.

> I looked into ZFS as well. Google helped me find a interesting page. I
> notice it is also used on some NAS setups as well. It seems to be
> advanced and maintained well. It sounds a little like LVM but may have
> more features, such as compression maybe? I haven't read that far yet.
> I notice it mentions snapshots which LVM also uses.

ZFS does a lot more then just LVM+Ext4. But it really needs multiple disks for
all the anti-corruption features as well.

> Getting plenty of ideas. I just wish I had a separate building to put a
> NAS in that would be safe and climate controlled. I got a out building
> but it gets plenty hot in the summer. No A/C or anything. I only heat
> it enough to prevent freezing but computers would likely like that anyway.

If you can keep it between optimal temperatures (and stable) the NAS should
manage. There is NO need to keep it at 18C (like some places do).

Also, consider a small AC unit that only cools a small box big enough for the
NAS. No need to cool an entire room.

--
Joost

Michael wrote:
> On Monday, 15 August 2022 11:58:14 BST Gerrit Kühn wrote:
>> Am Mon, 15 Aug 2022 12:50:37 +0200
>>
>> schrieb Gerrit Kühn <gerrit.kuehn@aei.mpg.de>:
>>> Being a happy restic user myself, I'd like to mention that compression is
>>> available meanwhile
>>> (https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html
>>> #compression). However, the feature is rather new, I did not use it so
>>> far.
>> https://forum.restic.net/t/compression-support-has-landed-in-master/4997
>>
>> Just adding another link to the official announcement from earlier this
>> year.
>>
>>
>> cu
>> Gerrit
> I think In Dale's use case compression is a solution seeking to address the
> problem of not enough storage space for backups, but it only makes sense if
> the data can be effectively and efficiently compressed. He mentioned 99.99%
> of his backup data is video. Video files are not particularly compressible,
> although small space savings can be achieved. For example using basic enough
> zst parameters '-19 --rsyncable -z' I got just a 1.6% file reduction:
>
> Frames Skips Compressed Uncompressed Ratio Check
> 1 0 88.9 MiB 90.3 MiB 1.016 XXH64
>
> Even if compression delivers some small space saving, given Dale's new faster
> Internet link and local video storage tendencies, compression will only kick
> the can down the road. If these are not private or rare videos and remain
> available on public streaming platforms, perhaps local storage is no longer
> necessary?


This is correct.  My main reason for compression was to squeeze a little
more mustard in the jar.  I just did a test on a dozen or so videos. 
Uncompressed the videos was 214,931,665.  Compressed it is 200,878,554. 
It's a little saved but other videos may not compress that much.  Just
depends.  Even with that little savings, compression really isn't much
of a solution especially since I do not want any loss in data. 

Given my weird way of doing backups, rsync may be the best option. 
Plus, easier to restore from as well since it just requires a copy
command, any of them will do. 

Looks like bigger hard drives is the best idea for now.  I really need a
proper backup plan tho.  I just wish a NAS would fit in my fire safe.  ;-)

Dale

:-)  :-)

Wol wrote:
> On 15/08/2022 08:52, Dale wrote:
>> I just hope this 10TB drive isn't a SMR.  I googled around and the best
>> I could find is anything above 8TB is CMR.  It's a WD101EDBZ-11B1DA0.  I
>> hope that is right.  I'm not totally opposed to SMR even as a backup but
>> I'd rather not.  The deal I found was for a pull and costs about $110
>> including shipping.  I looked at a 14TB but my jaw dropped.  $$$$$$$$
>
> Just done a search for you (it helps I know what I'm looking for), but
> CMR it is ...
>
> https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/
>
> I tend to go for OEM stuff, and know the names of the range I'm
> looking for - Seagate Ironwolf, Toshiba N300 - about £170 for 8TB ...
>
> Cheers,
> Wol

I can't express how glad I am to hear that.  While the current one is
SMR, I didn't know they even existed until I noticed it kept doing this
bumping thing long after the updates were done.  I could unmount it but
I can tell it was busy doing something but no clue what.  I actually
posted about it on this list and Rich figured it out.  Since then, I try
to avoid them just in case in the future I need to use the drive for
something SMR isn't suited for.  The almost full 8Tb drive will still be
used for backups but I'd never want to put it in my computer and use it
with anything that needs to write a lot.  Heck, even with me downloading
files over this new faster internet would put serious strain on that
drive.  Once the CMR buffer thingy fills up, it slows to about
40MBs/sec.  My internet can do 50MBs/sec easily.  I've seen it hit 70 or
better a few times. 

Glad to know what I found was good info.  I just wonder how long it will
be before even 10TB drives will be SMR.  I also dread having to search
out a 14TB drive later.  :/

Dale

:-)  :-) 

On Mon, Aug 15, 2022 at 3:20 PM J. Roeleveld <joost@antarean.org> wrote:
>
> Actually, you can with the "-p / --pause" option.
> Also, as per the man-page, if you forget this, the process will simply inform
> you the target location is full and you can move slices away to a different
> location:
> "
> If the destination filesystem is too small to contain all the slices of the
> backup, the -p option (pausing before starting new slices) might be of
> interest. Else, in the case the filesystem is full, dar will suspend the
> operation, asking for the user to make free space, then continue its
> operation. To make free space, the only thing you cannot do is to touch the
> slice being written.
> "
>
> The pause-option will actually stop between slices and you can umount the
> target location and mount a different disk there.
>
> This option has been around for a while.

Hmm, sounds kind of non-ideal.

It sounds like you can either have it pause when full, or pause
between slices. Neither is great.

If it pauses when full, and you can't touch the slice being written,
then you can't unmount the drive it is being written to. So you end
up having to write to a scratch area and keep moving slices off of
that onto another drive. At best that is extra IO which slows things
down, and of course you need scratch space.

If you pause between slices, then you have to have drives of equal
size to store to, otherwise you'll have to swap drives that aren't
completely full.

Ideally you'd want to write until the drive is almost full, then
finish a slice and pause.

However, it at least seems workable if slow. Just have scratch space
for a bunch of slices, let it pause when full, then move slices off as
they are done, and accept that your backups will run at maybe 25% of
the speed of the scratch drive since it will be constantly seeking
between writing new slices and reading old ones. Or if you have
enough RAM you could use a tmpfs for that but that seems really
cumbersome unless you use very small slices and have the shuffling
scripted.

--
Rich

On Mon, Aug 15, 2022 at 3:28 PM Dale <rdalek1967@gmail.com> wrote:
>
> Given my weird way of doing backups, rsync may be the best option.
> Plus, easier to restore from as well since it just requires a copy
> command, any of them will do.
>

If you don't make small changes inside of large files, you might
actually be fine with tar. It can do compression, though you won't
benefit much from that. It can split archives easily. It can do
incrementals, but I think they are limited to using metadata so it
depends on well-behaved software. What it doesn't do is let you
backup just a small part of a file containing larger changes.

--
Rich

Wol wrote:
> On 15/08/2022 10:45, John Covici wrote:
>> zfs would solve your problem of corruption, even without versioning.
>> You do a scrub at short intervals and at least you would know if the
>> file is corrupted.  Of course, redundancy is better, such as mirroring
>> and backups take a very short time because sending from one zfs to
>> another it knows exactly what bytes to send.
>
> I don't think he means a corrupted file, he means a corrupted video.
> If the drive faithfully records the corrupted feed, the filesystem is
> not going to catch it!
>
> Cheers,
> Wol

Yep.  Every once in a while, I download a video with better resolution
later to find out it is bad.  It gets part way through and crashes,
stops dead and sits there or just plain doesn't open.  Quite often, it
will have the correct thumbnail so it looks good but it's bad.  If I've
already trashed the old one and updated my backups, I have to go find it
again.  Given how some sites censor stuff, it could be gone for good. 
Generally, I can either catch it in the trash or on the backup that
hasn't been updated yet.  Given time, I'll miss one one day. 

The issues having a lot of files causes.  lol 

Dale

:-)  :-) 

> As it is, I have several options. In a way, I wish I could tell rsync to
do 1st half of alphabet to one drive and then with next command tell it to
do the 2nd half of alphabet. That would likely split the data in half for
each one.

You can do that, at least with a small kludge I'm sure. rsync supports
excluding directories and file names. As an example:

rsync -avx --port=873 --exclude={.cache,.nv,'google-chrome*',DiskImages}
/home/mark mark@truenas1:/mn
t/MyPool/mark/Backups/science/.

There's a test open (?? -n maybe ??) that allows you to see what it would
do.

I'm sure you can figure that part out. The above line is just in a script
file for me

HTH,
Mark

On Mon, Aug 15, 2022 at 3:41 PM Dale <rdalek1967@gmail.com> wrote:
>
> Glad to know what I found was good info. I just wonder how long it will
> be before even 10TB drives will be SMR. I also dread having to search
> out a 14TB drive later. :/
>

I think it will be a long time if ever, and here is why.

There are good reasons and bad reasons to use SMR. The reason you
would WANT to use SMR is that you have a task that is well-suited to
their limitations like backup or applications that can use log-style
storage. Ideally you'd want host-managed SMR for this. The benefit
is higher density for the cost, so you'd be doing it to get a drive
that is cheaper than it otherwise would be. However, these are all
things that would appeal to experts who really know what they're
doing.

The bad reason to use SMR is that you're a manufacturer trying to
squeeze out a bit more profit margin, not passing on the savings. In
this case you want to sell the drive to somebody who DOESN'T know what
they're doing, and make it drive-managed.

This is why we've seen SMR in medium-sized drives and not big ones as
would be expected if you assumed it would be employed for the good
reasons. The only people buying 14TB hard drives are people who tend
to know what they're doing, which makes them less of a target for
unscrupulous manufacturers. You wouldn't see them as much in small
drives as the return in capacity isn't as much. The medium sized
drives are big enough to get a return out of using SMR, but small
enough that suckers will be willing to buy them.

At least, that's my theory...

--
Rich

Mark Knecht wrote:
>
> > As it is, I have several options.  In a way, I wish I could tell
> rsync to do 1st half of alphabet to one drive and then with next
> command tell it to do the 2nd half of alphabet.  That would likely
> split the data in half for each one.  
>
> You can do that, at least with a small kludge I'm sure. rsync supports
> excluding directories and file names. As an example:
>
> rsync -avx --port=873
> --exclude={.cache,.nv,'google-chrome*',DiskImages} /home/mark
> mark@truenas1:/mn
> t/MyPool/mark/Backups/science/.
>
> There's a test open (?? -n maybe ??) that allows you to see what it
> would do.
>
> I'm sure you can figure that part out. The above line is just in a
> script file for me
>
> HTH,
> Mark


In the directory I'm backing up, there is over 400 directories.  That
would be a LOT of --exclude options.  Also, I would have to adjust the
exclude options each time I added a new directory, which can be several
a day sometimes.  The word nightmare comes to mind.  Loss of hair is
also a thought.  :-D 

I'm just glad I got a bigger hard drive coming.  That's the easiest fix
at the moment. 

Dale

:-)  :-) 

On Mon, Aug 15, 2022 at 1:17 PM Dale <rdalek1967@gmail.com> wrote:
>
> Mark Knecht wrote:
> >
> > > As it is, I have several options. In a way, I wish I could tell
> > rsync to do 1st half of alphabet to one drive and then with next
> > command tell it to do the 2nd half of alphabet. That would likely
> > split the data in half for each one.
> >
> > You can do that, at least with a small kludge I'm sure. rsync supports
> > excluding directories and file names. As an example:
> >
> > rsync -avx --port=873
> > --exclude={.cache,.nv,'google-chrome*',DiskImages} /home/mark
> > mark@truenas1:/mn
> > t/MyPool/mark/Backups/science/.
> >
> > There's a test open (?? -n maybe ??) that allows you to see what it
> > would do.
> >
> > I'm sure you can figure that part out. The above line is just in a
> > script file for me
> >
> > HTH,
> > Mark
>
>
> In the directory I'm backing up, there is over 400 directories. That
> would be a LOT of --exclude options. Also, I would have to adjust the
> exclude options each time I added a new directory, which can be several
> a day sometimes. The word nightmare comes to mind. Loss of hair is
> also a thought. :-D
>
> I'm just glad I got a bigger hard drive coming. That's the easiest fix
> at the moment.
>
> Dale
>
> :-) :-)

I have my movies in a directory called VideoLib and then subdirectories A,
B, C, etc. If you rearranged a little bit you could subdivide your movies
and use symbolic links to make it look more flat. That sort of arrangement
would mean 2 files with 13 excludes in each for the movies. You could leave
the rest alone, create a 3rd file and exclude the top level movie
directories.

Anyway, I just wanted to provide an option. Only you know what works for
your work flow.

Good luck,
Mark

Rich Freeman wrote:
> On Mon, Aug 15, 2022 at 3:41 PM Dale <rdalek1967@gmail.com> wrote:
>> Glad to know what I found was good info. I just wonder how long it will
>> be before even 10TB drives will be SMR. I also dread having to search
>> out a 14TB drive later. :/
>>
> I think it will be a long time if ever, and here is why.
>
> There are good reasons and bad reasons to use SMR. The reason you
> would WANT to use SMR is that you have a task that is well-suited to
> their limitations like backup or applications that can use log-style
> storage. Ideally you'd want host-managed SMR for this. The benefit
> is higher density for the cost, so you'd be doing it to get a drive
> that is cheaper than it otherwise would be. However, these are all
> things that would appeal to experts who really know what they're
> doing.
>
> The bad reason to use SMR is that you're a manufacturer trying to
> squeeze out a bit more profit margin, not passing on the savings. In
> this case you want to sell the drive to somebody who DOESN'T know what
> they're doing, and make it drive-managed.
>
> This is why we've seen SMR in medium-sized drives and not big ones as
> would be expected if you assumed it would be employed for the good
> reasons. The only people buying 14TB hard drives are people who tend
> to know what they're doing, which makes them less of a target for
> unscrupulous manufacturers. You wouldn't see them as much in small
> drives as the return in capacity isn't as much. The medium sized
> drives are big enough to get a return out of using SMR, but small
> enough that suckers will be willing to buy them.
>
> At least, that's my theory...
>


And that theory makes sense.  After you pointed out that my bumpy thing
was due to it being a SMR drive, I found out that several drive makers
were not telling people they were getting 2nd best drives.  I read about
a LOT of people who use RAID and such getting hit hard with this.  Some
seem to have taken basically new drives and created a doorstop pretty
quick.  I hope it was within the warranty time so the maker can foot the
bill for their deception.  It seems that consumers learned a lesson. 
I'm not sure about the makers but at least now they do make public what
drives are SMR so people who have use cases that result in dead SMR
drives can avoid buying one. 

I'm just glad to know what I ordered is what I expect.  No unpleasant
surprises.  I hope.

Dale

:-)  :-) 

>-----Original Message-----
>From: Dale <rdalek1967@gmail.com>
>Sent: Monday, August 15, 2022 12:47 PM
>To: gentoo-user@lists.gentoo.org
>Subject: Re: [gentoo-user] Backup program that compresses data but only changes new files.
>
>Wol wrote:
>> On 15/08/2022 10:45, John Covici wrote:
>>> zfs would solve your problem of corruption, even without versioning.
>>> You do a scrub at short intervals and at least you would know if the
>>> file is corrupted. Of course, redundancy is better, such as
>>> mirroring and backups take a very short time because sending from one
>>> zfs to another it knows exactly what bytes to send.
>>
>> I don't think he means a corrupted file, he means a corrupted video.
>> If the drive faithfully records the corrupted feed, the filesystem is
>> not going to catch it!
>>
>> Cheers,
>> Wol
>
>Yep. Every once in a while, I download a video with better resolution later to find out it is bad. It gets part way through and crashes, stops dead and sits there or just plain doesn't open. Quite often, it will have the correct thumbnail so it looks good but it's bad. If I've already trashed the old one and updated my backups, I have to go find it again. Given how some sites censor stuff, it could be gone for good. Generally, I can either catch it in the trash or on the backup that hasn't been updated yet. Given time, I'll miss one one day.
>
>The issues having a lot of files causes. lol
>
>Dale
>
>:-) :-)

You might consider just running ffmpeg or something headless like that over the file to see if there are errors before trashing the old version. Should even be pretty easy to script. If pass, replace old file, if fail raise notice box. Then you don't have to sit and wait for results.

LMP

>-----Original Message-----
>From: Rich Freeman <rich0@gentoo.org>
>Sent: Monday, August 15, 2022 12:52 PM
>To: gentoo-user@lists.gentoo.org
>Subject: Re: [gentoo-user] Backup program that compresses data but only changes new files.
>
>On Mon, Aug 15, 2022 at 3:41 PM Dale <rdalek1967@gmail.com> wrote:
>>
>> Glad to know what I found was good info. I just wonder how long it
>> will be before even 10TB drives will be SMR. I also dread having to
>> search out a 14TB drive later. :/
>>
>
>I think it will be a long time if ever, and here is why.
>
>There are good reasons and bad reasons to use SMR. The reason you would WANT to use SMR is that you have a task that is well-suited to their limitations like backup or applications that can use log-style storage. Ideally you'd want host-managed SMR for this. The benefit is higher density for the cost, so you'd be doing it to get a drive that is cheaper than it otherwise would be. However, these are all things that would appeal to experts who really know what they're doing.
>
>The bad reason to use SMR is that you're a manufacturer trying to squeeze out a bit more profit margin, not passing on the savings. In this case you want to sell the drive to somebody who DOESN'T know what they're doing, and make it drive-managed.
>
>This is why we've seen SMR in medium-sized drives and not big ones as would be expected if you assumed it would be employed for the good reasons. The only people buying 14TB hard drives are people who tend to know what they're doing, which makes them less of a target for unscrupulous manufacturers. You wouldn't see them as much in small drives as the return in capacity isn't as much. The medium sized drives are big enough to get a return out of using SMR, but small enough that suckers will be willing to buy them.
>
>At least, that's my theory...
>
>--
>Rich
>
>

A big chunk of it is that, when SMR drives came out, there was no reliable OS support for it, so it basically had to be drive-managed. Which then had horrible performance, and the cherry on top was that the drive manufacturers tried to cover up what they'd changed. So that made lots of the big companies doing big storage applications decide that SMR was crap and they simply will not buy SMR drives at any price.

Which is a real pity because there are lots of large-data applications where the write order is pretty much entirely sequential, so a properly designed, host managed system would see virtually no performance loss from SMR, and be able to take advantage of the higher density.

The moral is: be transparent with your customers.

LMP

Laurence Perkins wrote:
>
>> -----Original Message-----
>> From: Rich Freeman <rich0@gentoo.org>
>> Sent: Monday, August 15, 2022 12:52 PM
>> To: gentoo-user@lists.gentoo.org
>> Subject: Re: [gentoo-user] Backup program that compresses data but only changes new files.
>>
>> On Mon, Aug 15, 2022 at 3:41 PM Dale <rdalek1967@gmail.com> wrote:
>>> Glad to know what I found was good info. I just wonder how long it
>>> will be before even 10TB drives will be SMR. I also dread having to
>>> search out a 14TB drive later. :/
>>>
>> I think it will be a long time if ever, and here is why.
>>
>> There are good reasons and bad reasons to use SMR. The reason you would WANT to use SMR is that you have a task that is well-suited to their limitations like backup or applications that can use log-style storage. Ideally you'd want host-managed SMR for this. The benefit is higher density for the cost, so you'd be doing it to get a drive that is cheaper than it otherwise would be. However, these are all things that would appeal to experts who really know what they're doing.
>>
>> The bad reason to use SMR is that you're a manufacturer trying to squeeze out a bit more profit margin, not passing on the savings. In this case you want to sell the drive to somebody who DOESN'T know what they're doing, and make it drive-managed.
>>
>> This is why we've seen SMR in medium-sized drives and not big ones as would be expected if you assumed it would be employed for the good reasons. The only people buying 14TB hard drives are people who tend to know what they're doing, which makes them less of a target for unscrupulous manufacturers. You wouldn't see them as much in small drives as the return in capacity isn't as much. The medium sized drives are big enough to get a return out of using SMR, but small enough that suckers will be willing to buy them.
>>
>> At least, that's my theory...
>>
>> --
>> Rich
>>
>>
> A big chunk of it is that, when SMR drives came out, there was no reliable OS support for it, so it basically had to be drive-managed. Which then had horrible performance, and the cherry on top was that the drive manufacturers tried to cover up what they'd changed. So that made lots of the big companies doing big storage applications decide that SMR was crap and they simply will not buy SMR drives at any price.
>
> Which is a real pity because there are lots of large-data applications where the write order is pretty much entirely sequential, so a properly designed, host managed system would see virtually no performance loss from SMR, and be able to take advantage of the higher density.
>
> The moral is: be transparent with your customers.
>
> LMP


I do think that there are use cases where a SMR is just fine.  Thing is,
the customer should know what they getting to make sure they have a use
case where SMR works fine.  The drive makers should have told users up
front to avoid such problems and damage to the new technology.  As you
point out, the makers by hiding it, damaged their own reputations when
they tried to hide the facts.  For my backup, it does take longer to
sort the data out but it does work OK.  After I'm done with my backups,
I unmount and close the encryption on the drive.  Then I just let it sit
there until the bumpy thing stops.  Quite often, it's 5 or 10 minutes
but on occasion it will do its bumpy thing for 30 minutes or so.  Some
backup changes are quite large.  Would I have bought it if I knew what
it was and the extra time it takes, no.  While it works, it is
inconvenient for me to have to wait. 

Now that I know about SMR drives, I try to avoid them because I do
sometimes rotate drives around and even tho it is fine for backups, I
can't for example use that drive if I were to build a RAID system in a
NAS or other use cases where SMR drives perform badly.  I think when a
drive is sold that is SMR, it should be in the description to make sure
it is known.  It should have been from the start.  Sadly, it's rare that
it is even now. 

As always, it just depends on what you plan to do with it. 

Dale

:-)  :-) 

P. S.  Please pardon my time between replies.  Since I use a VPN and
most all of them block email, I have to stop the VPN to fetch and send
emails.  Eventually I'll figure out how to tunnel Seamonkey through the
VPN without having to stop it.  Eventually.  Maybe.  lol 

Mark Knecht wrote:
>
>
> On Mon, Aug 15, 2022 at 1:17 PM Dale <rdalek1967@gmail.com
> <mailto:rdalek1967@gmail.com>> wrote:
> >
> > Mark Knecht wrote:
> > >
> > > > As it is, I have several options.  In a way, I wish I could tell
> > > rsync to do 1st half of alphabet to one drive and then with next
> > > command tell it to do the 2nd half of alphabet.  That would likely
> > > split the data in half for each one.  
> > >
> > > You can do that, at least with a small kludge I'm sure. rsync supports
> > > excluding directories and file names. As an example:
> > >
> > > rsync -avx --port=873
> > > --exclude={.cache,.nv,'google-chrome*',DiskImages} /home/mark
> > > mark@truenas1:/mn
> > > t/MyPool/mark/Backups/science/.
> > >
> > > There's a test open (?? -n maybe ??) that allows you to see what it
> > > would do.
> > >
> > > I'm sure you can figure that part out. The above line is just in a
> > > script file for me
> > >
> > > HTH,
> > > Mark
> >
> >
> > In the directory I'm backing up, there is over 400 directories.  That
> > would be a LOT of --exclude options.  Also, I would have to adjust the
> > exclude options each time I added a new directory, which can be several
> > a day sometimes.  The word nightmare comes to mind.  Loss of hair is
> > also a thought.  :-D
> >
> > I'm just glad I got a bigger hard drive coming.  That's the easiest fix
> > at the moment.
> >
> > Dale
> >
> > :-)  :-)
>
> I have my movies in a directory called VideoLib and then
> subdirectories A, B, C, etc. If you rearranged a little bit you could
> subdivide your movies and use symbolic links to make it look more
> flat. That sort of arrangement would mean 2 files with 13 excludes in
> each for the movies. You could leave the rest alone, create a 3rd file
> and exclude the top level movie directories.
>
> Anyway, I just wanted to provide an option. Only you know what works
> for your work flow.
>
> Good luck,
> Mark
>
>


I did think about doing it that way but sometimes I can't recall the
name of something I'm looking for and just have to scan down the list. 
Breaking it up that way would complicate the matter because I'd have to
go look deeper.  As time goes on, I'd have to break it up even more
which would result in having to go even deeper.  There may be a way to
do it with linking or something but that could get interesting too. 
I've already got some divided up but even that causes issues sometimes. 

It's interesting, having all this data is neat but it can cause brain
teasers to sort through.  ;-)  I'm still considering that Raspberry Pi
thing tho.  Small, power efficient etc etc.  Lots of ideas.

Dale

:-)  :-) 

J. Roeleveld wrote:
> On Monday, August 15, 2022 9:07:41 PM CEST Dale wrote:
>> J. Roeleveld wrote:
>>> On Monday, August 15, 2022 12:44:11 AM CEST Dale wrote:
>>>> Howdy,
>>>>
>>>> With my new fiber internet, my poor disks are getting a work out, and
>>>> also filling up. First casualty, my backup disk. I have one directory
>>>> that is . . . well . . . huge. It's about 7TBs or so. This is where it
>>>> is right now and it's still trying to pack in files.
>>>>
>>>> /dev/mapper/8tb 7.3T 7.1T 201G 98% /mnt/8tb
>>> <snipped>
>>>
>>>> Thoughts? Ideas?
>>> Plenty, see below:
>>>
>>> For backups to external disks, I would recommend having a look at "dar" :
>>> $ eix -e dar
>>> * app-backup/dar
>>>
>>> Available versions: 2.7.6^t ~2.7.7^t {argon2 curl dar32 dar64 doc
>>> gcrypt
>>>
>>> gpg lz4 lzo nls rsync threads xattr}
>>>
>>> Homepage: http://dar.linux.free.fr/
>>> Description: A full featured backup tool, aimed for disks
>>>
>>> It's been around for a while and the developer is active and responds
>>> quite
>>> well to questions.
>>> It supports compression (different compression methods), incremental
>>> backups (only need a catalogue of the previous backup for the
>>> incremental) and encryption.
>>>
>>> The NAS options others mentioned would also work as they can compress data
>>> on disk and you'd only notice a delay in writing/reading (depending on
>>> the compression method used). I would recommend using one that uses ZFS
>>> on-disk as it's more reliable and robust then BTRFS.
>>>
>>> One option that comes available for you now that you are no longer limited
>>> to slow ADSL: Cloud backups.
>>>
>>> I use Backblaze (B2) to store compressed backups that haven't been stored
>>> on tape to off-site locations.
>>>
>>> But, you can also encrypt the backups locally and store the
>>> encrypted+compressed backupfiles on other cloud storage.
>>>
>>> --
>>> Joost
>> Dar does sound interesting. It sounds a lot like what I used way back
>> in the 90's. I'm sure it is different software but could work on
>> floppies then like it does on USB sticks etc today. Same principal.
> If it was during the 90's, then it wasn't. First version was released in 2002.
>
>> I looked into ZFS as well. Google helped me find a interesting page. I
>> notice it is also used on some NAS setups as well. It seems to be
>> advanced and maintained well. It sounds a little like LVM but may have
>> more features, such as compression maybe? I haven't read that far yet.
>> I notice it mentions snapshots which LVM also uses.
> ZFS does a lot more then just LVM+Ext4. But it really needs multiple disks for
> all the anti-corruption features as well.
>
>> Getting plenty of ideas. I just wish I had a separate building to put a
>> NAS in that would be safe and climate controlled. I got a out building
>> but it gets plenty hot in the summer. No A/C or anything. I only heat
>> it enough to prevent freezing but computers would likely like that anyway.
> If you can keep it between optimal temperatures (and stable) the NAS should
> manage. There is NO need to keep it at 18C (like some places do).
>
> Also, consider a small AC unit that only cools a small box big enough for the
> NAS. No need to cool an entire room.
>
> --
> Joost

If I built a NAS with a Raspberry Pi thing, it would be small enough
that I could just add a handle to it.  Then I could bring it in the
house, do backups and such and then store it in a out building away from
the house.  If I had a climate controlled outbuilding, I could either
run a ethernet cable or use wireless to do backups.  Of course, all this
costs money as does anything I need to do.  :/ 

I did check the price of a Raspberry Pi thingy a couple weeks ago. 
About $150 to $200 would likely get it off to a good start. It's not bad
really but that is a chunk of change.  I was actually looking into that
to run my VPN thing on.  Still thinking on that. 

Dale

:-)  :-) 

Am Mon, 15 Aug 2022 12:43:19 +0100
schrieb Michael <confabulate@kintzios.com>:

> Even if compression delivers some small space saving, given Dale's new
> faster Internet link and local video storage tendencies, compression
> will only kick the can down the road. If these are not private or rare
> videos and remain available on public streaming platforms, perhaps local
> storage is no longer necessary?

Compression for pre-compressed video data certainly does not achieve much
improvement. I just wanted to point out that restic meanwhile supports it
(as there might be other people reading this who are looking for backup
solutions in different situations).

Getting back to backing up publically available video data. I really can't
tell how much of a hassle it would be (in case of desaster) to get the data
back over the network, how long it would take, if that would be acceptable
etc. For distribution of content, e.g., netflix relies on big storage
boxes using zfs. This is quite reminiscent of the already suggested NAS
solution (and filesystems like zfs come with features like compression,
snapshots and incremental send/receive to faciliate backups).


cu
Gerrit