Mailing List Archive

On Friday, 3 January 2020 01:37:49 GMT Dale wrote:

> I'll try to reboot the new kernel in a bit. It's building at the
> moment. Thanks for posting about this. I did not see it in other
> replies. I thought it might be in Rich's but didn't see it. The extra
> nudge was helpful.
>
> Dale
>
> :-) :-)

I had mentioned about this kernel module in a previous post of mine. It may
make some difference, or it may not. It depends on the drive and its specific
data management mechanism:

https://en.wikipedia.org/wiki/Shingled_magnetic_recording#Data_management

--
Regards,

Mick

On 2020/01/01 at 08:00pm, Dale wrote:
> Grant Taylor wrote:
> > On 1/1/20 5:09 PM, Dale wrote:

> > Note:? umount will normally block until buffers are flushed to disk.
> >
> >> Is it safe to turn it off even tho it is doing whatever it is doing?
> >
> > I wouldn't.
> >
> >> Should I wait?
> >
> > I would.
> >
> >> Does it matter?
> >
> > Maybe.

[snip]

> If I touch the enclosure and feel it doing something, I leave it on,
> just in case.? I actually been wondering about this for a while.?
> Sometimes it will stop after a couple minutes, sometimes it is still
> doing its thing 30 minutes later.? In the case of the first, I was
> concerned about files being cached etc.

I have a similar drive (8TB external. WD, iirc). It did something
similar - it would seem to still be in use even after I umounted it -
and I wasn't sure if it was okay to unplug (no physical off
switch). Somewhere I found this command to shut down the drive:

udisksctl power-off --block-device /dev/sdx

I didn't see that command mentioned in the thread yet. I've been using
it, after umount, for about 8 months for roughly weekly backups and
some misc storage. So far, I've not seen any problems with it. The drive
immediately shuts down, and there haven't been any data or performance
issues.

But because no one else has mentioned it, I wonder udisksctl is not the
best tool in this case?


--
Chris Spackman chris@osugisakae.com

ESL Coordinator The Graham Family of Schools
ESL Instructor Columbus State Community College
Japan Exchange and Teaching Program Wajima, Ishikawa 1995-1998
Linux user since 1998 Linux User #137532

On Fri, Jan 3, 2020 at 8:51 AM Spackman, Chris <chris@osugisakae.com> wrote:
>
> udisksctl power-off --block-device /dev/sdx
>
> I didn't see that command mentioned in the thread yet. I've been using
> it, after umount, for about 8 months for roughly weekly backups and
> some misc storage. So far, I've not seen any problems with it. The drive
> immediately shuts down, and there haven't been any data or performance
> issues.
>
> But because no one else has mentioned it, I wonder udisksctl is not the
> best tool in this case?
>

I suspect that running this command will either not power off the
drive in this case, or if it does it will have the exact same impact
as pulling the plug. If the drive is unmounted then the kernel has
received guarantees from the drive that all data is written
persistently. It is unlikely that a drive will lose the data after
this point, but if has some bug then short of a firmware update I
doubt there is any reliable workaround short of leaving it on forever.

--
Rich

On 2020-01-02 14:12, Rich Freeman wrote:

> > Device Model: ST8000AS0003-2HH188
> >
> > I recall reading about SMR but can't recall the details of what it is.
> > far as I know, this is just a basic 8TB drive.
>
> This is an SMR drive. You should DEFINITELY read up on what they are.

How do you know? The identfying string doesn't appear in the kernel
source (I did a case-insensitive recursive grep).

--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

On Sat, Jan 4, 2020 at 10:13 PM Ian Zimmerman <itz@very.loosely.org> wrote:
>
> On 2020-01-02 14:12, Rich Freeman wrote:
>
> > > Device Model: ST8000AS0003-2HH188
> > >
> > > I recall reading about SMR but can't recall the details of what it is.
> > > far as I know, this is just a basic 8TB drive.
> >
> > This is an SMR drive. You should DEFINITELY read up on what they are.
>
> How do you know? The identfying string doesn't appear in the kernel
> source (I did a case-insensitive recursive grep).
>

https://www.seagate.com/www-content/product-content/hdd-fam/seagate-archive-hdd/en-us/docs/100827317b.pdf

These drives provide the following key features:
• 8TB capacity for efficient storage-per-slot.
• Affordable efficiencies with 2TB-per-disk Drive Managed SMR-based
hard drive technology.
...

--
Rich

Dale wrote:
> <<< SNIP >>>
> Should I change the mounting options for this drive?  I've read some
> people use certain options for SSD and they are needed.  This is the
> options being used according to mount:
>
> /dev/sdj1 on /run/media/dale/8tb-backup type ext4
> (rw,nosuid,nodev,relatime,uhelper=udisks2)
>
> That I guess is the default way KDE/udisks/etc does it. 
>
> One other question, what should I look for to avoid these types of
> drives?  I'll go back and look but I don't recall it saying anything
> about this.  I'm also trying to figure out if this is a good thing or not. 
>
> Thanks much for the info.  I'll read this one again shortly. Lot of info
> to absorb. 
>
> Dale
>
> :-)  :-) 
>


Finally got around to rebooting.  It was a experience for sure.  X
wouldn't come up, the plasma thingy doing its thing.  Jeepers.  No
wonder I hate rebooting.  ROFL  I enabled the options mentioned in the
kernel and finally rebooted using it. This is the option and the current
mount info:


root@fireball / # zcat /proc/config.gz | grep DM_ZONED
CONFIG_DM_ZONED=y
root@fireball / #

/dev/sdj1 on /run/media/dale/8tb-backup type ext4
(rw,nosuid,nodev,relatime,uhelper=udisks2)


It looks the same to me.  While it seems safe as is, should I be
changing something to make it use either additional or other options? 

While it is a backup, I'd like to make sure they are worth something if
the need should arise. 

Also, when looking for a drive to buy, what should one look at to see if
it is a SMR drive?  While it may be OK for my backups, I'd like to avoid
them on the drives inside my rig that are used for the OS or /home.  I
dunno, just a gut thing. 

Thanks.

Dale

:-)  :-) 

Dale,

"Dale" <rdalek1967@gmail.com>, 06.01.2020, 09:29:

> Also, when looking for a drive to buy, what should one look at to see if
> it is a SMR drive? While it may be OK for my backups, I'd like to avoid
> them on the drives inside my rig that are used for the OS or /home. I
> dunno, just a gut thing.

it's not "just a gut thing". SMR drives are not meant for random
access writing; they write like a tape and read like a disk.

A while ago, one of my clients bought one of those things
to replace an older failing backup drive. The next night, the
backup took hours instead of minutes. No knowing what was inside
the box, I did some measurements and discovered that the first
few files were written quickly, then things got really slow,
with the rsync process waiting (state "D") for the drive to
finish.

tar-based backups went much quicker, though, which matches the
expected behaviour of SMR drives; the drive did not need to rewrite
many large areas due to many small changes, instead it only had to
write one large area due to one large change.

s.

On Monday, 6 January 2020 08:48:13 GMT Stefan Schmiedl wrote:
> Dale,
>
> "Dale" <rdalek1967@gmail.com>, 06.01.2020, 09:29:
> > Also, when looking for a drive to buy, what should one look at to see if
> > it is a SMR drive?

You will need to visit the OEMs website and dig into the documentation they
provide. Keywords like "archive drive/disk/format", "shingled magnetic
recording" and "SMR", would be a giveaway this is not a normal PC drive.


> > While it may be OK for my backups, I'd like to avoid
> > them on the drives inside my rig that are used for the OS or /home. I
> > dunno, just a gut thing.
>
> it's not "just a gut thing". SMR drives are not meant for random
> access writing; they write like a tape and read like a disk.
>
> A while ago, one of my clients bought one of those things
> to replace an older failing backup drive. The next night, the
> backup took hours instead of minutes. No knowing what was inside
> the box, I did some measurements and discovered that the first
> few files were written quickly, then things got really slow,
> with the rsync process waiting (state "D") for the drive to
> finish.
>
> tar-based backups went much quicker, though, which matches the
> expected behaviour of SMR drives; the drive did not need to rewrite
> many large areas due to many small changes, instead it only had to
> write one large area due to one large change.
>
> s.

Stefan reinforced a point made earlier by Richard (I think). These drives are
only good for linear backups, like tar performs when it appends newer files to
an existing tarball. If they are used as normal PC drives for regular writing
of data, or with back up commands which use rsync, cp, etc. then the disk will
fail much sooner than expected because of repeated multiple areas being
deleted, before each smaller write. I recall reading about how short the life
of SMR drives was shown to be when used in NAS devices - check google or
youtube if you're interested in the specifics.

Personally, I would only use such a drive for 'keepers'. Say, films I intend
to write once and watch many times, ripped music albums, family photos, etc.
For OS files and other temporary backups I would use a normal PC drive.

PS. When you put together a tar script do not forget to add --xattrs. If not,
you'll find some commands break when you run them from a restored fs.
--
Regards,
Mick

On Mon, Jan 6, 2020 at 8:25 AM Mick <michaelkintzios@gmail.com> wrote:
>
> If they are used as normal PC drives for regular writing
> of data, or with back up commands which use rsync, cp, etc. then the disk will
> fail much sooner than expected because of repeated multiple areas being
> deleted, before each smaller write. I recall reading about how short the life
> of SMR drives was shown to be when used in NAS devices - check google or
> youtube if you're interested in the specifics.

Can you give a link - I'm not finding anything, and I'm a bit dubious
of this claim, because they still are just hard drives. These aren't
SSDs and hard drives should not have any kind of erasure limit.

Now, an SMR used for random writes is going to be a REALLY busy drive,
so I could see the drive being subject to a lot more wear and tear.
I'm just not aware of any kind of serious study. And of course any
particular model of hard drive can have reliability issues (just look
up the various reliability studies).

> Personally, I would only use such a drive for 'keepers'. Say, films I intend
> to write once and watch many times, ripped music albums, family photos, etc.
> For OS files and other temporary backups I would use a normal PC drive.

Certainly I would never use an SMR for an OS or /home. Backups should
be fine, as long as you're using a sequential backup file format.
tar/duplicity should be fine. Dar is probably fine but I'd need to
check (I think it just writes the index to the end, so the seeking
issues are on reading and not writing). Even zip/etc is going to be
fine. What is going to be a problem is anything that just replicates
the original data as all the separate files/directories that exist on
the original drive, like rsync/rsnapshot/etc. Those formats are of
course attractive because the backup is just a replica of the
original, but they involve random writes. Most formats that just
create a bunch of files named archive-001.foo that need a special
command to restore are going to be fine.

I personally haven't encountered a need to consider an SMR drive as
you can shuck those 12TB Easystore drives for something like $180 on
sale, at least in the US. Those are just standard drives (often with
red firmware). I couldn't even use them for my multimedia as I'm
storing that stuff on lizardfs right now and that breaks everything
into chunks. Granted, I don't rewrite it often but unless zfs is
SMR-aware it is still going to be writing lots of modest-sized files
as the original files get chunked up and distributed across the nodes.
On the disk lizardfs data just looks like a browser cache, with
everything in numbered files about 60MB in size in my case. The files
also appear to turn over a bit during rebalancing.

--
Rich

Rich Freeman wrote:
> On Mon, Jan 6, 2020 at 8:25 AM Mick <michaelkintzios@gmail.com> wrote:
>> If they are used as normal PC drives for regular writing
>> of data, or with back up commands which use rsync, cp, etc. then the disk will
>> fail much sooner than expected because of repeated multiple areas being
>> deleted, before each smaller write. I recall reading about how short the life
>> of SMR drives was shown to be when used in NAS devices - check google or
>> youtube if you're interested in the specifics.
> Can you give a link - I'm not finding anything, and I'm a bit dubious
> of this claim, because they still are just hard drives. These aren't
> SSDs and hard drives should not have any kind of erasure limit.
>
> Now, an SMR used for random writes is going to be a REALLY busy drive,
> so I could see the drive being subject to a lot more wear and tear.
> I'm just not aware of any kind of serious study. And of course any
> particular model of hard drive can have reliability issues (just look
> up the various reliability studies).
>

I ran up on this article however, it is a short time frame.  Still might
be a interesting read tho.

https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/

I'm still a bit curious and somewhat untrusting of those things tho. 
Regular hard drives go bad often enough as it is.  We don't need some
fancy unknown thing inserted just to add more issues.  Sort of reminds
me of the init thingy.  Each thing added is another failure point. 

I'm going to test my ebay skills and see if I can find some non-SMR
drives.  It sounds like some require some research to know if they are
or not.  :/

Dale

:-)  :-) 

On Mon, Jan 6, 2020 at 9:18 AM Dale <rdalek1967@gmail.com> wrote:
>
> Rich Freeman wrote:
> > On Mon, Jan 6, 2020 at 8:25 AM Mick <michaelkintzios@gmail.com> wrote:
> >> If they are used as normal PC drives for regular writing
> >> of data, or with back up commands which use rsync, cp, etc. then the disk will
> >> fail much sooner than expected because of repeated multiple areas being
> >> deleted, before each smaller write. I recall reading about how short the life
> >> of SMR drives was shown to be when used in NAS devices - check google or
> >> youtube if you're interested in the specifics.
> > Can you give a link - I'm not finding anything, and I'm a bit dubious
> > of this claim, because they still are just hard drives. These aren't
> > SSDs and hard drives should not have any kind of erasure limit.
> >
> > Now, an SMR used for random writes is going to be a REALLY busy drive,
> > so I could see the drive being subject to a lot more wear and tear.
> > I'm just not aware of any kind of serious study. And of course any
> > particular model of hard drive can have reliability issues (just look
> > up the various reliability studies).
> >
>
> I ran up on this article however, it is a short time frame. Still might
> be a interesting read tho.
>
> https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/

That article makes no mention of reliability issues with SMR. In
fact, they mention that they want 40% of their storage to be on SMR by
now. Clearly they wouldn't be doing that if the drives failed
frequently.

Note that they did modify their software to have write patterns
suitable for SMR. That is the key here. You absolutely have to
engineer your application to be suitable for SMR, or only choose SMR
if your application is already suitable. You can't just expect these
drives to perform remotely acceptably if you just throw random writes
at them.

> I'm still a bit curious and somewhat untrusting of those things tho.
> Regular hard drives go bad often enough as it is. We don't need some
> fancy unknown thing inserted just to add more issues. Sort of reminds
> me of the init thingy. Each thing added is another failure point.

Obviously they're relatively new, but they seem reliable enough.
They're just not suitable for general purpose use.

> I'm going to test my ebay skills and see if I can find some non-SMR
> drives. It sounds like some require some research to know if they are
> or not. :/

That's pretty simple. Find a drive that looks reasonable
price/capacity/etc-wise. Then just google the model number to confirm
it isn't SMR.

If you're in the US though you're probably best off shucking drives
from Best Buy these days. A drive that costs $350 as a bare drive
will get sold for $180 in a USB enclosure. I think it is just market
segmentation. They want to get top dollar from enterprise users, and
they aren't going to be shucking drives from Best Buy bought on "limit
1 item per customer" sales. By shucking I'm getting 12TB red drives
for less than the cost of a 6TB green drive. Just be aware that if
your PSU is old you'll need to tape over some of the SATA power pins.
New PSUs - even cheap ones - haven't given me any trouble.

I'm sure there are more up-to-date guides as these days the drives are
12TB, but here is the gist of it:
https://www.reddit.com/r/DataHoarder/comments/7fx0i0/wd_easystore_8tb_compendium/

If you aren't in the US I have no idea whether equivalent deals are
available. That subreddit is a good place to go for info on cheap
hard drives though.

--
Rich

Rich Freeman wrote:
> On Mon, Jan 6, 2020 at 9:18 AM Dale <rdalek1967@gmail.com> wrote:
>> Rich Freeman wrote:
>>> On Mon, Jan 6, 2020 at 8:25 AM Mick <michaelkintzios@gmail.com> wrote:
>>>> If they are used as normal PC drives for regular writing
>>>> of data, or with back up commands which use rsync, cp, etc. then the disk will
>>>> fail much sooner than expected because of repeated multiple areas being
>>>> deleted, before each smaller write. I recall reading about how short the life
>>>> of SMR drives was shown to be when used in NAS devices - check google or
>>>> youtube if you're interested in the specifics.
>>> Can you give a link - I'm not finding anything, and I'm a bit dubious
>>> of this claim, because they still are just hard drives. These aren't
>>> SSDs and hard drives should not have any kind of erasure limit.
>>>
>>> Now, an SMR used for random writes is going to be a REALLY busy drive,
>>> so I could see the drive being subject to a lot more wear and tear.
>>> I'm just not aware of any kind of serious study. And of course any
>>> particular model of hard drive can have reliability issues (just look
>>> up the various reliability studies).
>>>
>> I ran up on this article however, it is a short time frame. Still might
>> be a interesting read tho.
>>
>> https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/
> That article makes no mention of reliability issues with SMR. In
> fact, they mention that they want 40% of their storage to be on SMR by
> now. Clearly they wouldn't be doing that if the drives failed
> frequently.
>
> Note that they did modify their software to have write patterns
> suitable for SMR. That is the key here. You absolutely have to
> engineer your application to be suitable for SMR, or only choose SMR
> if your application is already suitable. You can't just expect these
> drives to perform remotely acceptably if you just throw random writes
> at them.

True but they likely have the drives that have it handled in software,
host managed I think they call it.  Another article I read was talking
about three different approaches to SMR.  The drive I have is likely
done on the drive itself, device managed, which is good for me.  The
ones in the article appear to manage the data transfer in software.  I
noticed they also use SSDs as sort of a temporary storage, if I
understood that correctly.  I think they did that to speed things up a bit.


>> I'm still a bit curious and somewhat untrusting of those things tho.
>> Regular hard drives go bad often enough as it is. We don't need some
>> fancy unknown thing inserted just to add more issues. Sort of reminds
>> me of the init thingy. Each thing added is another failure point.
> Obviously they're relatively new, but they seem reliable enough.
> They're just not suitable for general purpose use.
>
>> I'm going to test my ebay skills and see if I can find some non-SMR
>> drives. It sounds like some require some research to know if they are
>> or not. :/
> That's pretty simple. Find a drive that looks reasonable
> price/capacity/etc-wise. Then just google the model number to confirm
> it isn't SMR.
>
> If you're in the US though you're probably best off shucking drives
> from Best Buy these days. A drive that costs $350 as a bare drive
> will get sold for $180 in a USB enclosure. I think it is just market
> segmentation. They want to get top dollar from enterprise users, and
> they aren't going to be shucking drives from Best Buy bought on "limit
> 1 item per customer" sales. By shucking I'm getting 12TB red drives
> for less than the cost of a 6TB green drive. Just be aware that if
> your PSU is old you'll need to tape over some of the SATA power pins.
> New PSUs - even cheap ones - haven't given me any trouble.
>
> I'm sure there are more up-to-date guides as these days the drives are
> 12TB, but here is the gist of it:
> https://www.reddit.com/r/DataHoarder/comments/7fx0i0/wd_easystore_8tb_compendium/
>
> If you aren't in the US I have no idea whether equivalent deals are
> available. That subreddit is a good place to go for info on cheap
> hard drives though.
>


I've been noticing that too.  Only bad thing is, I can't always tell
what is in the enclosure.  Sometimes the info is given but sometimes
not.  I've also seen a few people complain that what they got was not
the model of drive they thought. 

I suspect one can get a adapter for that P/S connector.  I can't recall
when I got the P/S I currently have but it is a few years old.  I think
it's a ThermalTake or something like that.  I got to overclockers forum
where they list good ones.  I'm almost certain it has standard
connectors which may be a problem.  I've read about having to cover up a
pin or something but never seen one in person. 

This is a educational thread. I didn't even know SMR was a thing until
this thread came along. 

Dale

:-)  :-) 

On 2020-01-06, Dale <rdalek1967@gmail.com> wrote:

> Finally got around to rebooting.  It was a experience for sure.  X
> wouldn't come up, the plasma thingy doing its thing.  Jeepers.  No
> wonder I hate rebooting.

I try to remember to reboot all of my machines once a month or so when
I've got some spare time (especially after significant updates).
Otherwise, something will demand/cause a reboot in the middle of
something urgent and then you know what happens...

--
Grant Edwards grant.b.edwards Yow! for ARTIFICIAL
at FLAVORING!!
gmail.com

On Mon, Jan 6, 2020 at 10:05 AM Dale <rdalek1967@gmail.com> wrote:
>
> The drive I have is likely
> done on the drive itself, device managed, which is good for me.

Really the ideal situation are the Host Aware drives. I have no idea
what percentage of the markets they make. They fall back to being
device managed if the host doesn't do anything to manage them.

Your drive is device managed. See the link I posted earlier to the mfr info.

> I've been noticing that too. Only bad thing is, I can't always tell
> what is in the enclosure. Sometimes the info is given but sometimes
> not. I've also seen a few people complain that what they got was not
> the model of drive they thought.

Yeah, there are no guarantees as to what you'll get if you go the
shucking route. If you absolutely need a certain amount of cache or a
red firmware then you're just going to have to pay double to get that
guarantee.

For my application I'd definitely prefer the red firmware, but it
isn't really the end of the world if the drive takes a few seconds to
timeout on that one failure every 5 years. I'm not going to pay
double just to guarantee a particular model. If it were $20 more that
would be another matter, but we're talking $180 vs $350 here.

The other gotcha is that if you want to do a warranty replacement at
some point you're going to have to put it back in the enclosure to do
so. That means hanging onto enclosures, and is of course a bit of a
pain besides. Again, with a 50% reduction in cost you'd need to have
a lot of drive failures to be worth worrying about, especially since I
believe drive warranties are getting shorter anyway.

If you're in a typical enterprise situation then you're going to want
to just buy the bare drives with the standard warranty/etc. If you
buy in bulk chances are you're getting a discount anyway (and then
maybe you can get them in caddies or whatever). The Best Buy deals
are sporadic anyway and limited in quantity - you could never run a
data center that way. That's why they're priced the way they are...

--
Rich

On Monday, 6 January 2020 13:53:41 GMT Rich Freeman wrote:
> On Mon, Jan 6, 2020 at 8:25 AM Mick <michaelkintzios@gmail.com> wrote:
> > If they are used as normal PC drives for regular writing
> > of data, or with back up commands which use rsync, cp, etc. then the disk
> > will fail much sooner than expected because of repeated multiple areas
> > being deleted, before each smaller write. I recall reading about how
> > short the life of SMR drives was shown to be when used in NAS devices -
> > check google or youtube if you're interested in the specifics.
>
> Can you give a link - I'm not finding anything, and I'm a bit dubious
> of this claim, because they still are just hard drives. These aren't
> SSDs and hard drives should not have any kind of erasure limit.

This (random) link strongly recommends against usage in NAS, but gives no
reliability data:

https://www.storagereview.com/seagate_archive_hdd_review_8tb

This is a youtube video where someone was comparing SMR failures on a NAS:

https://www.youtube.com/watch?v=CR_bfbOTY1o


> Now, an SMR used for random writes is going to be a REALLY busy drive,
> so I could see the drive being subject to a lot more wear and tear.
> I'm just not aware of any kind of serious study. And of course any
> particular model of hard drive can have reliability issues (just look
> up the various reliability studies).

Right, I haven't seen any lab reliability studies published. I would think
more information could be sourced in IRC/ML where datacenter sysadmins hide to
compare their ... hardware. :-)

Reading another random link it seems Dale's 8TB SMR drive has a 20GB
conventional PMR platter/area in it to catch and cache any small writes. The
firmware will subsequently transfer the cached data on the SMR area of the
drive in due course, after it deletes the requisite adjacent overlapping
tracks. This means up to 20GB of initial writes will be normal, dropping to
lower speeds thereafter as the PMR cache needs to be flushed:

https://www.ixsystems.com/community/threads/smr-hard-drives-do-you-think-they-are-proper-nas-drives.35805/

If this is so, it explains Dale's observation of a hyperactive disk, well
after it was dismounted. Its firmware's been busy!

[snip ...]

> Granted, I don't rewrite it often but unless zfs is
> SMR-aware it is still going to be writing lots of modest-sized files
> as the original files get chunked up and distributed across the nodes.
> On the disk lizardfs data just looks like a browser cache, with
> everything in numbered files about 60MB in size in my case. The files
> also appear to turn over a bit during rebalancing.

I would think bit flipping between the 20GB PMR cache and the 8TB SMR tracks
represents an increased risk, vis A vis a single-step data transfer. Data
scrubbing well after the write has completed and committed to the SMR tracks
would reveal any anomalies.

What would seriously mess things up is creating a raid with mixed PMR and SMR
disks and running big (bigger than the internal cache) data writes. Some PMR
disks will complete well before the SMR. I/O blocking and timeouts could
ensue and the applications performing the writing could hang/fail.

Anyway, write once - read often, fits well the use case for these disks. They
should be right at home for long term video and media storage.
--
Regards,
Mick

Howdy all,

Little update here.  Rich, I think you mentioned it would slow down when
it ran out of PMR space while trying to redo the shingled part.  Up
until now, I hadn't ran into that issue.  It seems the PMR section for
this drive is somewhere around 40 or 50GBs, maybe 60GBs.  I hadn't had
time for backups in over a week so it was a good bit larger than usual. 
It was around 70GBs, maybe 75.  When it got close to the end of the
rsync process, I noticed it slowed down quite a bit.  I'd guess about
half or so.  Usually it runs at around 180 to 190MBs/sec for larger
files.  Pretty close to the end, rsync was showing around 100MBs/sec at
best.  It was a little over on some but mostly a little below that. 
Earlier in the process, it was the normal speed.

I might add, even tho the copy process has been done for a while now, 20
minutes or more, I can still feel that little bumpy thing it does.  It
seems to have stopped while typing that in.  Unless it is taking a break
and starts up again.  ;-) 

Still, for this use case, it works OK.  I wouldn't want SMR on my /home
or some other partition that needs to be fairly fast at all times.  For
windoze users, well, they used to that slow down and freezes so they
wouldn't notice the difference.  ROFL 

Dale

:-)  :-) 

On Mon, Aug 3, 2020 at 1:30 AM Dale <rdalek1967@gmail.com> wrote:
>
> Little update here. Rich, I think you mentioned it would slow down when it ran out of PMR space while trying to redo the shingled part. Up until now, I hadn't ran into that issue. It seems the PMR section for this drive is somewhere around 40 or 50GBs, maybe 60GBs. I hadn't had time for backups in over a week so it was a good bit larger than usual. It was around 70GBs, maybe 75. When it got close to the end of the rsync process, I noticed it slowed down quite a bit. I'd guess about half or so. Usually it runs at around 180 to 190MBs/sec for larger files. Pretty close to the end, rsync was showing around 100MBs/sec at best. It was a little over on some but mostly a little below that. Earlier in the process, it was the normal speed.

I doubt this particular drop is the result of SMR, assuming 100MB/s is
the instantaneous speed. 100MB/s is still reasonable for a hard drive
- on newer CMR drives I've seen the speed of dd drop from 200MB/s to
100MB/s for sequential writes as the heads move from one end of the
drive to the other, and then it goes back up to 200MB/s if you start
over at the beginning (badblocks testing and so on).

That level of drop is probably more likely to be due to filesystem
overhead and so on - fragmentation/etc. When SMR buffer overrun
occurs you REALLY hit a wall and the rates drop quite a bit more than
that. If it were a difference of only 50% most would probably
tolerate it.

Now, if 100MB/s were an updating average across the entire run then
that would be a different matter, because that would mean that it was
running at 200MB/s for most of the run, and then probably dropping
much closer to 0 for a while so as to drive the overall average down
to 100MB/s. I'm not sure where you're getting those numbers from so I
don't know what period that 100MB/s reflects. For an instantaneous
speed I'd consider it a completely reasonable performance for a
typical hard drive when you're writing to a filesystem. If you were
using dd or maybe copies of very large files on an efficient
filesystem you would get better results.

--
Rich

Rich Freeman wrote:
> On Mon, Aug 3, 2020 at 1:30 AM Dale <rdalek1967@gmail.com> wrote:
>> Little update here. Rich, I think you mentioned it would slow down when it ran out of PMR space while trying to redo the shingled part. Up until now, I hadn't ran into that issue. It seems the PMR section for this drive is somewhere around 40 or 50GBs, maybe 60GBs. I hadn't had time for backups in over a week so it was a good bit larger than usual. It was around 70GBs, maybe 75. When it got close to the end of the rsync process, I noticed it slowed down quite a bit. I'd guess about half or so. Usually it runs at around 180 to 190MBs/sec for larger files. Pretty close to the end, rsync was showing around 100MBs/sec at best. It was a little over on some but mostly a little below that. Earlier in the process, it was the normal speed.
> I doubt this particular drop is the result of SMR, assuming 100MB/s is
> the instantaneous speed. 100MB/s is still reasonable for a hard drive
> - on newer CMR drives I've seen the speed of dd drop from 200MB/s to
> 100MB/s for sequential writes as the heads move from one end of the
> drive to the other, and then it goes back up to 200MB/s if you start
> over at the beginning (badblocks testing and so on).
>
> That level of drop is probably more likely to be due to filesystem
> overhead and so on - fragmentation/etc. When SMR buffer overrun
> occurs you REALLY hit a wall and the rates drop quite a bit more than
> that. If it were a difference of only 50% most would probably
> tolerate it.
>
> Now, if 100MB/s were an updating average across the entire run then
> that would be a different matter, because that would mean that it was
> running at 200MB/s for most of the run, and then probably dropping
> much closer to 0 for a while so as to drive the overall average down
> to 100MB/s. I'm not sure where you're getting those numbers from so I
> don't know what period that 100MB/s reflects. For an instantaneous
> speed I'd consider it a completely reasonable performance for a
> typical hard drive when you're writing to a filesystem. If you were
> using dd or maybe copies of very large files on an efficient
> filesystem you would get better results.
>


I use the --progress option with rsync.  It shows the copy speed,
elapsed time etc for each file.  Some files are small and the speed it
shows isn't accurate at all because the files are just to tiny to get a
accurate measure.  I ignore that info on the smaller files.  Videos
however are larger files and sometimes can take a lot longer to copy
over.  Those tend to be more accurate.  Anyway, that is where the
numbers came from.  I wish I had saved the output but sadly I had
finished my OS updates and needed to logout and back in again.

In the past, I've never seen the drive on the larger files be that slow
even toward the end.  Generally, it stays pretty close to 180MBs/sec or
so which is what I usually get with PMR drives.  Of course, the PMR
drives don't keep doing the bumpy thing for a while when it is done
either.  Maybe it is something else but it sure did the bumpy thing a
lot longer this time. 

Either way, just wanted to update that a large copy made it slow down. 
Other than the initial copy, this is the largest rsync I've ever done to
that drive.  Most are around 20, maybe 25GBs. 

Dale

:-)  :-) 

On Mon, Aug 3, 2020 at 8:24 AM Dale <rdalek1967@gmail.com> wrote:
>
> In the past, I've never seen the drive on the larger files be that slow even toward the end. Generally, it stays pretty close to 180MBs/sec or so which is what I usually get with PMR drives.

Yeah, just hard to be certain without ditching the filesystem layer or
doing some kind of comparison. The difference in write speed across
the drive on recent drives I've gotten is more pronounced than I've
seen on other drives, but these drives tend to be around 12TB.

Definitely the thing to watch out for is a big drop in transfer rate
once a large number of blocks have been transferred continuously, and
then performance returns after you let the drive thrash for a while.
I've seen complaints of zfs rebuilds going from hours/days to
weeks/months in length, so it isn't just a 50% drop when you're doing
worst-case access patterns. On the other hand I hear that mdadm isn't
so bad, so if the writes are sequential the drive might be better at
skipping the cache, and maybe zfs just does its rebuild
non-sequentially (which isn't really ideal anyway).

I haven't really dug into the guts of how zfs metadata works, but with
btrfs I believe the chunks are basically their own layer, and the
filesystem can scrub them without really any care about what files are
stored in them. That allows them to be easily scrubbed sequentially.
When I did rebuilds on btrfs they tended to run at about the max
throughput of the drives as long as there wasn't any other access
going on. It can also do read-only scrubs to check data integrity
sequentially across the disk, which suggests the checksums are stored
at a lower layer and so the data can be verified without worrying
about file fragmentation and so on. This layering also lets btrfs
switch "RAID modes" on the fly with half of the disk being RAID1 and
half the disk being RAID5 and so on - each region of a disk is
independent from the others and so mode changes only impact new
regions until you do a full rebalance to rewrite everything. Of
course, zfs has its own advantages.

--
Rich

Rich Freeman wrote:
> On Mon, Aug 3, 2020 at 8:24 AM Dale <rdalek1967@gmail.com> wrote:
>> In the past, I've never seen the drive on the larger files be that slow even toward the end. Generally, it stays pretty close to 180MBs/sec or so which is what I usually get with PMR drives.
> Yeah, just hard to be certain without ditching the filesystem layer or
> doing some kind of comparison. The difference in write speed across
> the drive on recent drives I've gotten is more pronounced than I've
> seen on other drives, but these drives tend to be around 12TB.
>
> Definitely the thing to watch out for is a big drop in transfer rate
> once a large number of blocks have been transferred continuously, and
> then performance returns after you let the drive thrash for a while.
> I've seen complaints of zfs rebuilds going from hours/days to
> weeks/months in length, so it isn't just a 50% drop when you're doing
> worst-case access patterns. On the other hand I hear that mdadm isn't
> so bad, so if the writes are sequential the drive might be better at
> skipping the cache, and maybe zfs just does its rebuild
> non-sequentially (which isn't really ideal anyway).
>
> I haven't really dug into the guts of how zfs metadata works, but with
> btrfs I believe the chunks are basically their own layer, and the
> filesystem can scrub them without really any care about what files are
> stored in them. That allows them to be easily scrubbed sequentially.
> When I did rebuilds on btrfs they tended to run at about the max
> throughput of the drives as long as there wasn't any other access
> going on. It can also do read-only scrubs to check data integrity
> sequentially across the disk, which suggests the checksums are stored
> at a lower layer and so the data can be verified without worrying
> about file fragmentation and so on. This layering also lets btrfs
> switch "RAID modes" on the fly with half of the disk being RAID1 and
> half the disk being RAID5 and so on - each region of a disk is
> independent from the others and so mode changes only impact new
> regions until you do a full rebalance to rewrite everything. Of
> course, zfs has its own advantages.
>

This drive is formatted with ext4.  It doesn't have LVM or anything just
straight ext4.  Given it is external, I didn't see the point of having
LVM on it and adding another layer to deal with when there is no
benefits to it.  While it does delete some files and overwrite others,
it mostly adds new files.  I'd guess it just throws them on the end but
who knows what it is really doing under the hood that neither of us
knows about.  ;-)

I did notice that is has reached 80% full.  I'm going to start figuring
out what not to backup and such pretty soon.  If I have my videos, I'm
good.  I could backup Documents and some other OS related stuff on
another drive.  I usually backup /etc and the world file.  Hmmm, may
want to grab my local overlay too.  I hadn't thought about that.  Funny
how typing in a email makes me think of things like that. 

Maybe when I do a large backup next time, I'll think to save the output
so I can share it.  That might shed some light on the situation.  I just
thought it was interesting that when it hit about 50 or 60GBs of data it
got a good deal slower and then did the bumpy thing a lot longer too. 
To me, I figured it ran out of PMR space and was slinging stuff good
trying to catch up.  There was several files where it just sat there,
for many seconds with nothing moving.  Usually, 10 seconds is about the
longest wait but it is a large file, movie or something that is GBs or
so.  I'd guess close to a minute for some this last time.  Big
difference.  Most files were around 300MBs too.  Found a source for good
HD stuff that isn't to large.  ;-)

Anyway, thought it worth mentioning.  The drive serves its purpose well
enough for what it does.  Still wouldn't want it somewhere more critical
tho.  Sticking with PMR/CMR until they get things sorted out better.  If
I bought another drive just for external backup use tho, I might get a
SMR if the price was right.  It wouldn't be my first choice tho.

Dale

:-)  :-) 

On 04/08/20 04:17, Dale wrote:
> This drive is formatted with ext4. It doesn't have LVM or anything just
> straight ext4. Given it is external, I didn't see the point of having
> LVM on it and adding another layer to deal with when there is no
> benefits to it.

LVM has a big advantage if not much (relatively) changes between
backups. If you have let's say 900GB of data, your backup disk is 1TB,
and say 20GB of data changes between backups, with LVM that drive could
store 5 *full* backups! You could use btrfs to the same effect.

Both LVM and btrfs offer snapshotting, so you take a snapshot before
doing an in-place rsync, giving you one full backup per snapshot, but
the drive is actually only storing the changes between snapshots.
Probably run the backup much faster too.

If this drive is used to store full backups, and only stores the one
copy, it won't be that expensive/risky to reformat and try that?

Cheers,
Wol

On 04/08/20 08:42, Wols Lists wrote:
> Both LVM and btrfs offer snapshotting, so you take a snapshot before
> doing an in-place rsync, giving you one full backup per snapshot, but
> the drive is actually only storing the changes between snapshots.
> Probably run the backup much faster too.

Just strikes me this would be near ideal for an SMR drive, because this
would be copy-on-write, so the backup would just be streaming new data
to disk.

And by judiciously choosing when to delete snapshots, you have
considerable control over when the drive decides to do a defrag.

Cheers,
Wol

Wols Lists wrote:
> On 04/08/20 08:42, Wols Lists wrote:
>> Both LVM and btrfs offer snapshotting, so you take a snapshot before
>> doing an in-place rsync, giving you one full backup per snapshot, but
>> the drive is actually only storing the changes between snapshots.
>> Probably run the backup much faster too.
> Just strikes me this would be near ideal for an SMR drive, because this
> would be copy-on-write, so the backup would just be streaming new data
> to disk.
>
> And by judiciously choosing when to delete snapshots, you have
> considerable control over when the drive decides to do a defrag.
>
> Cheers,
> Wol
>
>


I've never used those features of LVM before.  Most likely, I should. 
I've had occasion to do a backup and then wish I had a old file back
that was deleted during the new backup process.  Example.  I have a copy
of a video for a particular show.  I find a better version, HD or
something, and download it.  I then remove the old one and find out
shortly after that it's the wrong episode or something.  At that point,
I'm missing a episode.  I'd rather have a standard definition version
than none at all.  I suspect doing it the way you mention I'd be able to
get that old copy back provided that snapshot hasn't been deleted yet. 
The way I do it now, once I update the backups, old stuff is deleted. 

Also, I just did another fairly large update on the backups.  Once it
hit around 50GBs or so, it started slowing down again.  It was even
slower than last time.  It was transferring at around 70MBs/sec.  Of
course, it could be partly because that drive is filling up.  It's
around 80% or so. 

Anyway.  I need to look into the snapshot thing.  Gotta find a howto.  ;-)

Dale

:-)  :-)