Mailing List Archive

On 1/1/20 5:09 PM, Dale wrote:
> Howdy,

Hi,

> As some may recall, I have a 8TB external SATA hard drive that I do
> back ups on. Usually, I back up once a day, more often if needed.
> Usually I turn the power on, mount it, do the back ups, unmount and
> turn the power back off. Usually it is powered up for 5 minutes or so.
> When I unmount it tho, I sometimes notice it is still doing something.
> I can feel the mechanism for the heads moving. It has a slight
> vibration to it. Questions are, what is it doing and should I let it
> finish before powering it off? I'd assume that once it in unmounted,
> the copy process is done so the files are safe. I guess it is doing
> some sort of internal checks or something but I'm not sure.

There might be some activity for up to 30 seconds after umount finishes
and returns to the command prompt.

Note: umount will normally block until buffers are flushed to disk.

> Is it safe to turn it off even tho it is doing whatever it is doing?

I wouldn't.

> Should I wait?

I would.

> Does it matter?

Maybe.

Is the drive SATA connected or USB connected to the machine?

In some ways it doesn't matter. You can tell the kernel to eject the
drive. Once that finishes, there is no active remnants of the drive in
kernel. 5–15 seconds after that and you should be quite safe to power
the drive off.

echo 1 > /sys/class/block/$DEVICENAME/device/delete

That will cause the kernel to gracefully disconnect the drive.

> Thanks.

:-)



--
Grant. . . .
unix || die





--
Grant. . . .
unix || die

On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
> Howdy,
>
> As some may recall, I have a 8TB external SATA hard drive that I do back
> ups on. Usually, I back up once a day, more often if needed. Usually I
> turn the power on, mount it, do the back ups, unmount and turn the power
> back off. Usually it is powered up for 5 minutes or so. When I unmount
> it tho, I sometimes notice it is still doing something. I can feel the
> mechanism for the heads moving. It has a slight vibration to it.
> Questions are, what is it doing and should I let it finish before
> powering it off? I'd assume that once it in unmounted, the copy process
> is done so the files are safe. I guess it is doing some sort of
> internal checks or something but I'm not sure.

There is some delay with data still in the buffers between rsync/cp/tar/what-
ever saying it's finished on your terminal and the drive itself finishing
storing the data on the platters.

If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
Normally, if you try to unmount a drive while it is still being written to,
the umount/udisks command will complain the drive is busy.


> Is it safe to turn it off even tho it is doing whatever it is doing?
> Should I wait? Does it matter?
>
> Thanks.
>
> Dale
>
> :-) :-)

If you wait for a few seconds after the backup is completed before you unmount
the drive, you should be OK. Although it may slow down or any LEDs flash less
frequently the drive may not stop spinning, unless there is some power save
process taking control of it.


> P. S. Down to last router that was discussed in another thread so I
> bought it while they had it. Price may go up if I didn't. Did more
> research on old modem, it is risky to try to convert to AT&T. Some say
> not possible.

Right, ISP controlled firmware typically requires re-flashing the device with
the new ISP's firmware version. In some cases even the boot code needs
replacing. Should you flash the router with a wrong firmware build, you could
sometimes derive a door stop without additional cost. In this case you'll
need a JTAG and access to its circuit board with an OEM boot/firmware version
to recover it. In most cases OEMs support lines will redirect you to your
ISP, who run an overseas support line and will ask you to reboot your
MSWindows PC ... O_o

This is a reason I avoid these kind of routers as much as I can.

--
Regards,
Mick

Grant Taylor wrote:
> On 1/1/20 5:09 PM, Dale wrote:
>> Howdy,
>
> Hi,
>
>> As some may recall, I have a 8TB external SATA hard drive that I do
>> back ups on.  Usually, I back up once a day, more often if needed.
>> Usually I turn the power on, mount it, do the back ups, unmount and
>> turn the power back off.  Usually it is powered up for 5 minutes or
>> so. When I unmount it tho, I sometimes notice it is still doing
>> something. I can feel the mechanism for the heads moving.  It has a
>> slight vibration to it.  Questions are, what is it doing and should I
>> let it finish before powering it off?  I'd assume that once it in
>> unmounted, the copy process is done so the files are safe.  I guess
>> it is doing some sort of internal checks or something but I'm not sure.
>
> There might be some activity for up to 30 seconds after umount
> finishes and returns to the command prompt.
>
> Note:  umount will normally block until buffers are flushed to disk.
>
>> Is it safe to turn it off even tho it is doing whatever it is doing?
>
> I wouldn't.
>
>> Should I wait?
>
> I would.
>
>> Does it matter?
>
> Maybe.
>
> Is the drive SATA connected or USB connected to the machine?
>
> In some ways it doesn't matter.  You can tell the kernel to eject the
> drive.  Once that finishes, there is no active remnants of the drive
> in kernel.  5–15 seconds after that and you should be quite safe to
> power the drive off.
>
> echo 1 > /sys/class/block/$DEVICENAME/device/delete
>
> That will cause the kernel to gracefully disconnect the drive.
>
>> Thanks.
>
> :-)
>
>
>

It is connected with a eSATA cable.  It has a USB connector as well but
I just don't trust USB as much as I do SATA.  Other USB enclosures had
"issues".  ;-) 

If I touch the enclosure and feel it doing something, I leave it on,
just in case.  I actually been wondering about this for a while. 
Sometimes it will stop after a couple minutes, sometimes it is still
doing its thing 30 minutes later.  In the case of the first, I was
concerned about files being cached etc.  Thing is, it *should* do that
before unmounting.  In the case of it going on for 30 minutes or more, I
was wondering if it was doing some sort of housekeeping, media tests or
something.  I just didn't know if it was important to wait or not.  On
occasion I would leave the drive on overnight or all day.  I figure it
would be able to at least do a quick self test and report errors if any. 

Thanks for the extra info.  It helps a bit. 

Dale

:-)  :-) 

Mick wrote:
> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
>> Howdy,
>>
>> As some may recall, I have a 8TB external SATA hard drive that I do back
>> ups on. Usually, I back up once a day, more often if needed. Usually I
>> turn the power on, mount it, do the back ups, unmount and turn the power
>> back off. Usually it is powered up for 5 minutes or so. When I unmount
>> it tho, I sometimes notice it is still doing something. I can feel the
>> mechanism for the heads moving. It has a slight vibration to it.
>> Questions are, what is it doing and should I let it finish before
>> powering it off? I'd assume that once it in unmounted, the copy process
>> is done so the files are safe. I guess it is doing some sort of
>> internal checks or something but I'm not sure.
> There is some delay with data still in the buffers between rsync/cp/tar/what-
> ever saying it's finished on your terminal and the drive itself finishing
> storing the data on the platters.
>
> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
> Normally, if you try to unmount a drive while it is still being written to,
> the umount/udisks command will complain the drive is busy.
>

When it does it for a somewhat short period of time, I can understand
that.  It's one reason I try to leave it on when it "feels" that it is
still busy.  Thing is, there are times when it goes on for 30 minutes or
more.  At those times, even a USB stick should be done.  One would think
at least.  It makes me curious as to what it is doing in that case. 
Still, I'd rather the unmount command force a wait until it is done. 
Honestly, I wouldn't want a drive or software that says something is
done when it isn't.  It's not good even when shutting a system down. 
Given the speed of drives, I would think a few seconds at most.  Best to
be safe.  ;-)  I just wonder, is it doing two different things?  One
when it is busy for short periods of time and something else when it
goes on for a while.  This is what sort of puzzles me.  Selftest maybe??

>> Is it safe to turn it off even tho it is doing whatever it is doing?
>> Should I wait? Does it matter?
>>
>> Thanks.
>>
>> Dale
>>
>> :-) :-)
> If you wait for a few seconds after the backup is completed before you unmount
> the drive, you should be OK. Although it may slow down or any LEDs flash less
> frequently the drive may not stop spinning, unless there is some power save
> process taking control of it.
>

Given the speed, it is likely done when I tell the KDE thingy to
unmount.  Usually, I start the backup and walk away for a few minutes. 
I do it with one of my scripts, if one can call what I do a script, and
it does the date command at the end.  Even if there was a lot of
changes, I can tell how long it was completed.  I try to give it a
couple minutes.  Still, good point.  This is one reason I'm asking about
this.  It's hard to know exactly what is going on here. 

>> P. S. Down to last router that was discussed in another thread so I
>> bought it while they had it. Price may go up if I didn't. Did more
>> research on old modem, it is risky to try to convert to AT&T. Some say
>> not possible.
> Right, ISP controlled firmware typically requires re-flashing the device with
> the new ISP's firmware version. In some cases even the boot code needs
> replacing. Should you flash the router with a wrong firmware build, you could
> sometimes derive a door stop without additional cost. In this case you'll
> need a JTAG and access to its circuit board with an OEM boot/firmware version
> to recover it. In most cases OEMs support lines will redirect you to your
> ISP, who run an overseas support line and will ask you to reboot your
> MSWindows PC ... O_o
>
> This is a reason I avoid these kind of routers as much as I can.
>

Keep in mind, two pieces of hardware.  Router for the first two
sentences and Modem for next two.  Tried to be short so . . . . Anyway,
router should be flashable with Openwrt.  It's a slightly older model. 
New model may be ready for flashing in a year or two but not so much at
the moment so I went with the older model. The modem, I never could find
the firmware.  I found links to it but those links ended up being dead. 
Even if I had it, it was unlikely to work.  Possible but I'd be
concerned about its stability and such even if it did take it. I have a
modem and router on the way.  I just didn't want to miss the deal on the
router.  They had several a couple weeks or so ago.  I got the last
one.  Waiting for their arrival. 

Dale

:-)  :-)

Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
someone on here could use a spare or just needs one period, moving or
something.  Modem is wireless with a router as well.  Nice modem I guess. 

On 2/1/20 10:27 am, Dale wrote:
> Mick wrote:
>> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
>>> Howdy,
>>>
>>> As some may recall, I have a 8TB external SATA hard drive that I do back
>>> ups on. Usually, I back up once a day, more often if needed. Usually I
>>> turn the power on, mount it, do the back ups, unmount and turn the power
>>> back off. Usually it is powered up for 5 minutes or so. When I unmount
>>> it tho, I sometimes notice it is still doing something. I can feel the
>>> mechanism for the heads moving. It has a slight vibration to it.
>>> Questions are, what is it doing and should I let it finish before
>>> powering it off? I'd assume that once it in unmounted, the copy process
>>> is done so the files are safe. I guess it is doing some sort of
>>> internal checks or something but I'm not sure.
>> There is some delay with data still in the buffers between rsync/cp/tar/what-
>> ever saying it's finished on your terminal and the drive itself finishing
>> storing the data on the platters.
>>
>> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
>> Normally, if you try to unmount a drive while it is still being written to,
>> the umount/udisks command will complain the drive is busy.
>>
> When it does it for a somewhat short period of time, I can understand
> that.  It's one reason I try to leave it on when it "feels" that it is
> still busy.  Thing is, there are times when it goes on for 30 minutes or
> more.  At those times, even a USB stick should be done.  One would think
> at least.  It makes me curious as to what it is doing in that case.
> Still, I'd rather the unmount command force a wait until it is done.
> Honestly, I wouldn't want a drive or software that says something is
> done when it isn't.  It's not good even when shutting a system down.
> Given the speed of drives, I would think a few seconds at most.  Best to
> be safe.  ;-)  I just wonder, is it doing two different things?  One
> when it is busy for short periods of time and something else when it
> goes on for a while.  This is what sort of puzzles me.  Selftest maybe??
>
>>> Is it safe to turn it off even tho it is doing whatever it is doing?
>>> Should I wait? Does it matter?
>>>
>>> Thanks.
>>>
>>> Dale
>>>
>>> :-) :-)
>> If you wait for a few seconds after the backup is completed before you unmount
>> the drive, you should be OK. Although it may slow down or any LEDs flash less
>> frequently the drive may not stop spinning, unless there is some power save
>> process taking control of it.
>>
> Given the speed, it is likely done when I tell the KDE thingy to
> unmount.  Usually, I start the backup and walk away for a few minutes.
> I do it with one of my scripts, if one can call what I do a script, and
> it does the date command at the end.  Even if there was a lot of
> changes, I can tell how long it was completed.  I try to give it a
> couple minutes.  Still, good point.  This is one reason I'm asking about
> this.  It's hard to know exactly what is going on here.
>
>>> P. S. Down to last router that was discussed in another thread so I
>>> bought it while they had it. Price may go up if I didn't. Did more
>>> research on old modem, it is risky to try to convert to AT&T. Some say
>>> not possible.
>> Right, ISP controlled firmware typically requires re-flashing the device with
>> the new ISP's firmware version. In some cases even the boot code needs
>> replacing. Should you flash the router with a wrong firmware build, you could
>> sometimes derive a door stop without additional cost. In this case you'll
>> need a JTAG and access to its circuit board with an OEM boot/firmware version
>> to recover it. In most cases OEMs support lines will redirect you to your
>> ISP, who run an overseas support line and will ask you to reboot your
>> MSWindows PC ... O_o
>>
>> This is a reason I avoid these kind of routers as much as I can.
>>
> Keep in mind, two pieces of hardware.  Router for the first two
> sentences and Modem for next two.  Tried to be short so . . . . Anyway,
> router should be flashable with Openwrt.  It's a slightly older model.
> New model may be ready for flashing in a year or two but not so much at
> the moment so I went with the older model. The modem, I never could find
> the firmware.  I found links to it but those links ended up being dead.
> Even if I had it, it was unlikely to work.  Possible but I'd be
> concerned about its stability and such even if it did take it. I have a
> modem and router on the way.  I just didn't want to miss the deal on the
> router.  They had several a couple weeks or so ago.  I got the last
> one.  Waiting for their arrival.
>
> Dale
>
> :-)  :-)
>
> Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
> someone on here could use a spare or just needs one period, moving or
> something.  Modem is wireless with a router as well.  Nice modem I guess.
>

Try atop from sys-process/atop - it will show you how busy individual
disks are (and a lot of other stats as well.)

You can issue a sync command to flush any disk buffers before unmounting
(umounting should sync as well.).  The heads may keep moving because of
the internal data management modern disks do. The disks should be safe
to power off despite this (they have an internal flush/save/park routine
on power loss, with enough energy stored to take care of it)

BillK

Bill Kenworthy wrote:
> On 2/1/20 10:27 am, Dale wrote:
>> Mick wrote:
>>> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
>>>> Howdy,
>>>>
>>>> As some may recall, I have a 8TB external SATA hard drive that I do
>>>> back
>>>> ups on.  Usually, I back up once a day, more often if needed. 
>>>> Usually I
>>>> turn the power on, mount it, do the back ups, unmount and turn the
>>>> power
>>>> back off.  Usually it is powered up for 5 minutes or so.  When I
>>>> unmount
>>>> it tho, I sometimes notice it is still doing something.  I can feel
>>>> the
>>>> mechanism for the heads moving.  It has a slight vibration to it.
>>>> Questions are, what is it doing and should I let it finish before
>>>> powering it off?  I'd assume that once it in unmounted, the copy
>>>> process
>>>> is done so the files are safe.  I guess it is doing some sort of
>>>> internal checks or something but I'm not sure.
>>> There is some delay with data still in the buffers between
>>> rsync/cp/tar/what-
>>> ever saying it's finished on your terminal and the drive itself
>>> finishing
>>> storing the data on the platters.
>>>
>>> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
>>> Normally, if you try to unmount a drive while it is still being
>>> written to,
>>> the umount/udisks command will complain the drive is busy.
>>>
>> When it does it for a somewhat short period of time, I can understand
>> that.  It's one reason I try to leave it on when it "feels" that it is
>> still busy.  Thing is, there are times when it goes on for 30 minutes or
>> more.  At those times, even a USB stick should be done.  One would think
>> at least.  It makes me curious as to what it is doing in that case.
>> Still, I'd rather the unmount command force a wait until it is done.
>> Honestly, I wouldn't want a drive or software that says something is
>> done when it isn't.  It's not good even when shutting a system down.
>> Given the speed of drives, I would think a few seconds at most.  Best to
>> be safe.  ;-)  I just wonder, is it doing two different things?  One
>> when it is busy for short periods of time and something else when it
>> goes on for a while.  This is what sort of puzzles me.  Selftest maybe??
>>
>>>> Is it safe to turn it off even tho it is doing whatever it is doing?
>>>> Should I wait?  Does it matter?
>>>>
>>>> Thanks.
>>>>
>>>> Dale
>>>>
>>>> :-)  :-)
>>> If you wait for a few seconds after the backup is completed before
>>> you unmount
>>> the drive, you should be OK.  Although it may slow down or any LEDs
>>> flash less
>>> frequently the drive may not stop spinning, unless there is some
>>> power save
>>> process taking control of it.
>>>
>> Given the speed, it is likely done when I tell the KDE thingy to
>> unmount.  Usually, I start the backup and walk away for a few minutes.
>> I do it with one of my scripts, if one can call what I do a script, and
>> it does the date command at the end.  Even if there was a lot of
>> changes, I can tell how long it was completed.  I try to give it a
>> couple minutes.  Still, good point.  This is one reason I'm asking about
>> this.  It's hard to know exactly what is going on here.
>>
>>>> P. S. Down to last router that was discussed in another thread so I
>>>> bought it while they had it.  Price may go up if I didn't.  Did more
>>>> research on old modem, it is risky to try to convert to AT&T.  Some
>>>> say
>>>> not possible.
>>> Right, ISP controlled firmware typically requires re-flashing the
>>> device with
>>> the new ISP's firmware version.  In some cases even the boot code needs
>>> replacing.  Should you flash the router with a wrong firmware build,
>>> you could
>>> sometimes derive a door stop without additional cost.  In this case
>>> you'll
>>> need a JTAG and access to its circuit board with an OEM
>>> boot/firmware version
>>> to recover it.  In most cases OEMs support lines will redirect you
>>> to your
>>> ISP, who run an overseas support line and will ask you to reboot your
>>> MSWindows PC ... O_o
>>>
>>> This is a reason I avoid these kind of routers as much as I can.
>>>
>> Keep in mind, two pieces of hardware.  Router for the first two
>> sentences and Modem for next two.  Tried to be short so . . . . Anyway,
>> router should be flashable with Openwrt.  It's a slightly older model.
>> New model may be ready for flashing in a year or two but not so much at
>> the moment so I went with the older model. The modem, I never could find
>> the firmware.  I found links to it but those links ended up being dead.
>> Even if I had it, it was unlikely to work.  Possible but I'd be
>> concerned about its stability and such even if it did take it. I have a
>> modem and router on the way.  I just didn't want to miss the deal on the
>> router.  They had several a couple weeks or so ago.  I got the last
>> one.  Waiting for their arrival.
>>
>> Dale
>>
>> :-)  :-)
>>
>> Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
>> someone on here could use a spare or just needs one period, moving or
>> something.  Modem is wireless with a router as well.  Nice modem I
>> guess.
>>
>
> Try atop from sys-process/atop - it will show you how busy individual
> disks are (and a lot of other stats as well.)
>
> You can issue a sync command to flush any disk buffers before
> unmounting (umounting should sync as well.).  The heads may keep
> moving because of the internal data management modern disks do. The
> disks should be safe to power off despite this (they have an internal
> flush/save/park routine on power loss, with enough energy stored to
> take care of it)
>
> Bill

I'll give atop a try.  I forgot about that command.  I use top, htop,
iftop etc but forgot about atop.  Come to think of it, iotop may be
worth looking into as well.  I'm not sure what the difference is between
atop and iotop tho.  I'll try both.  ;-) 

I was sort of thinking that once it is unmounted, it should be safe. 
Thing is, I wasn't sure.  What really confused me, when it goes on for
30 minutes or more.  That just seems to long for it to clear buffers or
something.  A selftest would make sense but when I run those manually, I
can't usually "feel" it doing anything.  It was puzzling me a bit. 

It seems it should be safe to power off after a few minutes.  Everyone
seems to agree on that.  The data should be safe anyway.  I guess what
remains is what it is doing when active for longer times.  Most likely
what you say, something internal such as media testing or something of
that nature.  I guess that doesn't matter so much but I think I'll leave
it powered on whenever I can.  Sort of let it do its thing.  I suspect
most drive manufacturers sort of expect drives to run for longer times
than what I'm doing.  When possible, I'll give them their time, just in
case. 

Thanks to all.  At least I have some things to try to nose around with
and am pretty sure my data will be safe. 

Dale

:-)  :-) 

Bill Kenworthy wrote:
> On 2/1/20 10:27 am, Dale wrote:
>> Mick wrote:
>>> On Thursday, 2 January 2020 00:09:14 GMT Dale wrote:
>>>> Howdy,
>>>>
>>>> As some may recall, I have a 8TB external SATA hard drive that I do
>>>> back
>>>> ups on.  Usually, I back up once a day, more often if needed. 
>>>> Usually I
>>>> turn the power on, mount it, do the back ups, unmount and turn the
>>>> power
>>>> back off.  Usually it is powered up for 5 minutes or so.  When I
>>>> unmount
>>>> it tho, I sometimes notice it is still doing something.  I can feel
>>>> the
>>>> mechanism for the heads moving.  It has a slight vibration to it.
>>>> Questions are, what is it doing and should I let it finish before
>>>> powering it off?  I'd assume that once it in unmounted, the copy
>>>> process
>>>> is done so the files are safe.  I guess it is doing some sort of
>>>> internal checks or something but I'm not sure.
>>> There is some delay with data still in the buffers between
>>> rsync/cp/tar/what-
>>> ever saying it's finished on your terminal and the drive itself
>>> finishing
>>> storing the data on the platters.
>>>
>>> If you look at vmstat, or keep an eye on Gkrelm you'll see what I mean.
>>> Normally, if you try to unmount a drive while it is still being
>>> written to,
>>> the umount/udisks command will complain the drive is busy.
>>>
>> When it does it for a somewhat short period of time, I can understand
>> that.  It's one reason I try to leave it on when it "feels" that it is
>> still busy.  Thing is, there are times when it goes on for 30 minutes or
>> more.  At those times, even a USB stick should be done.  One would think
>> at least.  It makes me curious as to what it is doing in that case.
>> Still, I'd rather the unmount command force a wait until it is done.
>> Honestly, I wouldn't want a drive or software that says something is
>> done when it isn't.  It's not good even when shutting a system down.
>> Given the speed of drives, I would think a few seconds at most.  Best to
>> be safe.  ;-)  I just wonder, is it doing two different things?  One
>> when it is busy for short periods of time and something else when it
>> goes on for a while.  This is what sort of puzzles me.  Selftest maybe??
>>
>>>> Is it safe to turn it off even tho it is doing whatever it is doing?
>>>> Should I wait?  Does it matter?
>>>>
>>>> Thanks.
>>>>
>>>> Dale
>>>>
>>>> :-)  :-)
>>> If you wait for a few seconds after the backup is completed before
>>> you unmount
>>> the drive, you should be OK.  Although it may slow down or any LEDs
>>> flash less
>>> frequently the drive may not stop spinning, unless there is some
>>> power save
>>> process taking control of it.
>>>
>> Given the speed, it is likely done when I tell the KDE thingy to
>> unmount.  Usually, I start the backup and walk away for a few minutes.
>> I do it with one of my scripts, if one can call what I do a script, and
>> it does the date command at the end.  Even if there was a lot of
>> changes, I can tell how long it was completed.  I try to give it a
>> couple minutes.  Still, good point.  This is one reason I'm asking about
>> this.  It's hard to know exactly what is going on here.
>>
>>>> P. S. Down to last router that was discussed in another thread so I
>>>> bought it while they had it.  Price may go up if I didn't.  Did more
>>>> research on old modem, it is risky to try to convert to AT&T.  Some
>>>> say
>>>> not possible.
>>> Right, ISP controlled firmware typically requires re-flashing the
>>> device with
>>> the new ISP's firmware version.  In some cases even the boot code needs
>>> replacing.  Should you flash the router with a wrong firmware build,
>>> you could
>>> sometimes derive a door stop without additional cost.  In this case
>>> you'll
>>> need a JTAG and access to its circuit board with an OEM
>>> boot/firmware version
>>> to recover it.  In most cases OEMs support lines will redirect you
>>> to your
>>> ISP, who run an overseas support line and will ask you to reboot your
>>> MSWindows PC ... O_o
>>>
>>> This is a reason I avoid these kind of routers as much as I can.
>>>
>> Keep in mind, two pieces of hardware.  Router for the first two
>> sentences and Modem for next two.  Tried to be short so . . . . Anyway,
>> router should be flashable with Openwrt.  It's a slightly older model.
>> New model may be ready for flashing in a year or two but not so much at
>> the moment so I went with the older model. The modem, I never could find
>> the firmware.  I found links to it but those links ended up being dead.
>> Even if I had it, it was unlikely to work.  Possible but I'd be
>> concerned about its stability and such even if it did take it. I have a
>> modem and router on the way.  I just didn't want to miss the deal on the
>> router.  They had several a couple weeks or so ago.  I got the last
>> one.  Waiting for their arrival.
>>
>> Dale
>>
>> :-)  :-)
>>
>> Oh, I may post and see if anyone needs a Frontier modem later.  Maybe
>> someone on here could use a spare or just needs one period, moving or
>> something.  Modem is wireless with a router as well.  Nice modem I
>> guess.
>>
>
> Try atop from sys-process/atop - it will show you how busy individual
> disks are (and a lot of other stats as well.)
>
> You can issue a sync command to flush any disk buffers before
> unmounting (umounting should sync as well.).  The heads may keep
> moving because of the internal data management modern disks do. The
> disks should be safe to power off despite this (they have an internal
> flush/save/park routine on power loss, with enough energy stored to
> take care of it)
>
> Bill

Tried atop, it works best for this since it showed the drive device
instead of process name like iotop.  In my case, it shows sdj.  I turned
the drive on, atop showed nothing until I mounted it.  I then did a
quick back up and it showed up like it should.  Once done, it showed no
activity until I told KDE to unmount.  After it was unmounted, it showed
no further activity according to atop, iotop or anything else for that
matter.  I could however feel it doing something.  The heads were moving
around every few seconds or so.  Sometimes close together, sometimes
several seconds or more apart.  It's sort of intermittent type activity.

I might add, the light that shows activity on the enclosure shows
nothing either.  The light is odd, it's always on but blinks when there
is activity.  Usually they are off and come on when active.  Sort of
weird but OK. 

I think once unmounted, the files are done, or a short time afterwards. 
After that, I think you and other are right, it's doing some internal
stuff. 

Thanks again to all who replied.  I'm not 100% sure of what it is doing
but pretty sure I can power it off once the back up is done and it is
unmounted. 

Dale

:-)  :-) 

On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:

> > If you wait for a few seconds after the backup is completed before
> > you unmount the drive, you should be OK. Although it may slow down
> > or any LEDs flash less frequently the drive may not stop spinning,
> > unless there is some power save process taking control of it.
> >
>
> Given the speed, it is likely done when I tell the KDE thingy to
> unmount.  Usually, I start the backup and walk away for a few minutes. 
> I do it with one of my scripts, if one can call what I do a script, and
> it does the date command at the end.

Add a sync command to the end of the script to make sure all filesystem
buffers are flushed to disk before it finishes.


--
Neil Bothwick

We've all heard that a million monkeys banging on a million
typewriters will eventually reproduce the works of Shakespeare.
Now, thanks to the Internet, we know this is not true.

Neil Bothwick wrote:
> On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:
>
>>> If you wait for a few seconds after the backup is completed before
>>> you unmount the drive, you should be OK. Although it may slow down
>>> or any LEDs flash less frequently the drive may not stop spinning,
>>> unless there is some power save process taking control of it.
>>>
>> Given the speed, it is likely done when I tell the KDE thingy to
>> unmount.  Usually, I start the backup and walk away for a few minutes. 
>> I do it with one of my scripts, if one can call what I do a script, and
>> it does the date command at the end.
> Add a sync command to the end of the script to make sure all filesystem
> buffers are flushed to disk before it finishes.
>
>

Now there's a idea.  Between the sync command and the unmount process,
it should be safe for sure.  You made good use of those brain cells. 
lol  I didn't think of that and I don't recall anyone else thinking of
it either.  Still wonder what the heck it is doing sometimes tho. 

Dale

:-)  :-) 

Em qui., 2 de jan. de 2020 às 07:49, Dale <rdalek1967@gmail.com> escreveu:

> Neil Bothwick wrote:
> > On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:
> >
> >>> If you wait for a few seconds after the backup is completed before
> >>> you unmount the drive, you should be OK. Although it may slow down
> >>> or any LEDs flash less frequently the drive may not stop spinning,
> >>> unless there is some power save process taking control of it.
> >>>
> >> Given the speed, it is likely done when I tell the KDE thingy to
> >> unmount. Usually, I start the backup and walk away for a few minutes.
> >> I do it with one of my scripts, if one can call what I do a script, and
> >> it does the date command at the end.
> > Add a sync command to the end of the script to make sure all filesystem
> > buffers are flushed to disk before it finishes.
> >
> >
>
> Now there's a idea. Between the sync command and the unmount process,
> it should be safe for sure. You made good use of those brain cells.
> lol I didn't think of that and I don't recall anyone else thinking of
> it either. Still wonder what the heck it is doing sometimes tho.
>
> Dale
>
> :-) :-)
>
>
Another thing to be considered is S.M.A.R.T. activity, that is proper to
the hard drive firmware. Don't know if it is safe or not to power it down
during that process though. I would wait until it really finishes all
activity.

But do you really need to power it down? Mechanic (magnetic) hard drives
are known to have shorter lives if they are turned on and off frequently.
My desk computer is rarely turned off, I have 4 hard drives, the oldest is
close to 10 years old and SMART diagnostics are always good.

Best regards,
Francisco

On Thu, Jan 2, 2020 at 5:49 AM Dale <rdalek1967@gmail.com> wrote:
>
> lol I didn't think of that and I don't recall anyone else thinking of
> it either.

That is because syncing before unmounting doesn't do anything. Unless
you use --lazy umount blocks until all writes are complete to a
device. The instant it returns as far as the kernel is concerned the
device should be safe to power off.

If you do a sync first then of course the umount will complete more
quickly, since all writes should already be flushed.

I have no idea what your device is doing after it is unmounted, but it
doesn't have anything to do with the linux kernel unless some process
is directly accessing the raw device (very unlikely). Maybe the drive
firmware is doing some kind of housekeeping, or maybe the drive has
some kind of vibration in it that just makes it feel like it is doing
something. Or maybe the NSA or Red Army has hacked your firmware and
it is doing who knows what (yes, the NSA bit at least is a thing). In
any case, chances are the drive manufacturer has accounted for sudden
power loss in the design because if they didn't there would be a ton
of complaints, since there is nothing you can do about this sort of
thing assuming the firmware is up to something.

Out of curiosity, what model drive is it? Is it by chance an SMR /
archive drive? Due to the limitations on how those write data out I
could see them implementing an internal filesystem that journals
incoming data and then writes it back out after the fact. If so then
that might happen even after the kernel thinks it is unmounted.
However, such a drive firmware would probably use a journal that
ensures data is safe even if power is cut mid-operation. The drive
isn't supposed to report that a write is completed until it is
durable.

--
Rich

On Thursday, 2 January 2020 14:43:58 GMT Rich Freeman wrote:


> Out of curiosity, what model drive is it? Is it by chance an SMR /
> archive drive?

Good catch! I hadn't thought of this - the Linux kernel will need to have
DM_ZONED enabled I think, for the OS to manager the shingled writes
sequentially, but I don't have this enabled here because AFAIK I have no such
drives in my possession.


> Due to the limitations on how those write data out I
> could see them implementing an internal filesystem that journals
> incoming data and then writes it back out after the fact.

SMR drives which implement a 'device managed' write mechanism, will use their
own firmware to control data storage. The OS would not be aware of anything
being different to a conventional drive.


> If so then
> that might happen even after the kernel thinks it is unmounted.
> However, such a drive firmware would probably use a journal that
> ensures data is safe even if power is cut mid-operation. The drive
> isn't supposed to report that a write is completed until it is
> durable.

Which I take it to mean the drive would not be unmounted by the OS until it is
safe to do so and for all intends and purposes it will also be safe to be
powered down thereafter. I would think this would be within seconds of
successfully unmounting it. Spinning for 30 minutes or more after it is
unmounted sounds excessive to me, if it is only being spun by the firmware for
flushing its journal buffers. I have a conventional USB drive (WD passport)
which is always spinning whether it is being written to or not.
--
Regards,

Mick

On Thu, Jan 2, 2020 at 11:23 AM Mick <michaelkintzios@gmail.com> wrote:
>
> On Thursday, 2 January 2020 14:43:58 GMT Rich Freeman wrote:
>
>
> > Out of curiosity, what model drive is it? Is it by chance an SMR /
> > archive drive?
>
> Good catch! I hadn't thought of this - the Linux kernel will need to have
> DM_ZONED enabled I think, for the OS to manager the shingled writes
> sequentially, but I don't have this enabled here because AFAIK I have no such
> drives in my possession.

I haven't looked into the details of how it works. Certainly the
kernel should be able to optimize writes to such disks and utilize
TRIM if supported by the drive. But it isn't strictly needed as you
go on to say.

> > Due to the limitations on how those write data out I
> > could see them implementing an internal filesystem that journals
> > incoming data and then writes it back out after the fact.
>
> SMR drives which implement a 'device managed' write mechanism, will use their
> own firmware to control data storage. The OS would not be aware of anything
> being different to a conventional drive.

Correct.

> > If so then
> > that might happen even after the kernel thinks it is unmounted.
> > However, such a drive firmware would probably use a journal that
> > ensures data is safe even if power is cut mid-operation. The drive
> > isn't supposed to report that a write is completed until it is
> > durable.
>
> Which I take it to mean the drive would not be unmounted by the OS until it is
> safe to do so and for all intends and purposes it will also be safe to be
> powered down thereafter.

Yes - even if the drive is doing its own data shuffling after being
unmounted, it should still be safe to power off.

In any case, I was just speculating as to why it might be doing writes
when not mounted. I don't know the internal details of how these
drives all work.

> I would think this would be within seconds of
> successfully unmounting it. Spinning for 30 minutes or more after it is
> unmounted sounds excessive to me, if it is only being spun by the firmware for
> flushing its journal buffers.

Indeed. That really makes me wonder whether it is actually writing
anything. It could just be that the drive has some vibration or
otherwise is giving the sensation that it is doing something when it
isn't.

Though, if the thing really does have a large journal inside to
improve performance it could actually buffer a pretty large number of
writes. SMR drives can have a very large amount of write
amplification. If each erase region on the disk contains 10k blocks
then every 1MB of data in the journal could potentially lead to 10GB
of disk rewrites, assuming that writes are randomly distributed. It
also makes sense that when replaying the journal the drive is going to
prioritize erase regions with the most updates, which means that by
the time you're unmounting the drive you're going to expect the stuff
left in the journal to lead to the most write amplification.

I would imagine that a drive that works in this manner is going to use
a logfs for the journal (if the journal is even in SMR format), and
then it would just keep an erase region free at all times. Anytime a
partial write happens in a region the data would get written to the
free region, then that region would be remapped in place of the old
region, and now the old region is unallocated for the next write.
This could be interrupted due to power loss at any time without any
real loss to data, since data is not overwritten in place. Before the
journal record is closed out the old region is still valid, and after
the new region is valid. An interrupted write is just repeated on
power up since the new region can just be overwritten again safely.

In any case, the bottom line is that a drive should be safe to unplug
if all filesystems on it are umounted and the umount commands all
return. If the disk loses data after that point it is a manufacturing
design flaw.

--
Rich

On 2020-01-01 18:09, Dale wrote:

> As some may recall, I have a 8TB external SATA hard drive that I do
> back ups on. Usually, I back up once a day, more often if
> needed. Usually I turn the power on, mount it, do the back ups,
> unmount and turn the power back off. Usually it is powered up for 5
> minutes or so. When I unmount it tho, I sometimes notice it is still
> doing something. I can feel the mechanism for the heads moving. It has
> a slight vibration to it. Questions are, what is it doing and should
> I let it finish before powering it off? I'd assume that once it in
> unmounted, the copy process is done so the files are safe. I guess it
> is doing some sort of internal checks or something but I'm not sure.

I have observed the same thing. But in my case, I also disconnect the
_cable_ from the computer to the enclosure when I am done ... and still
the drive activity goes on. From that I conclude that it is the drive
circuitry itself doing some kind of internal housekeeping, and there is
no point in worrying about it because one would wait forever for it to
end.

--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

Francisco Ares wrote:
>
>
> Em qui., 2 de jan. de 2020 às 07:49, Dale <rdalek1967@gmail.com
> <mailto:rdalek1967@gmail.com>> escreveu:
>
> Neil Bothwick wrote:
> > On Wed, 1 Jan 2020 20:27:34 -0600, Dale wrote:
> >
> >>> If you wait for a few seconds after the backup is completed before
> >>> you unmount the drive, you should be OK.  Although it may slow
> down
> >>> or any LEDs flash less frequently the drive may not stop spinning,
> >>> unless there is some power save process taking control of it.
> >>> 
> >> Given the speed, it is likely done when I tell the KDE thingy to
> >> unmount.  Usually, I start the backup and walk away for a few
> minutes. 
> >> I do it with one of my scripts, if one can call what I do a
> script, and
> >> it does the date command at the end.
> > Add a sync command to the end of the script to make sure all
> filesystem
> > buffers are flushed to disk before it finishes.
> >
> >
>
> Now there's a idea.  Between the sync command and the unmount process,
> it should be safe for sure.  You made good use of those brain cells. 
> lol  I didn't think of that and I don't recall anyone else thinking of
> it either.  Still wonder what the heck it is doing sometimes tho. 
>
> Dale
>
> :-)  :-) 
>
>
> Another thing to be considered is S.M.A.R.T. activity, that is proper
> to the hard drive firmware.  Don't know if it is safe or not to power
> it down during that process though.  I would wait until it really
> finishes all activity.
>
> But do you really need to power it down?  Mechanic (magnetic) hard
> drives are known to have shorter lives if they are turned on and off
> frequently.  My desk computer is rarely turned off, I have 4 hard
> drives, the oldest is close to 10 years old and SMART diagnostics are
> always good.
>
> Best regards,
> Francisco


That's what I meant when talking about selftest or media checks.  As we
know, most all drives have this function nowadays.  As far as I know,
all drives do this now, except SSD anyway.  They may have some other
thing they do tho.

I've thought about the power on/off cycle and it's one reason I only
back up once a day most all the time.  Sometimes, I may not backup for a
couple days or so depending on what's going on.  While my puter runs
24/7, I've had drives go bad in them before too.  Something always
happens whether it is power on/off or media going bad or just plain
mechanical/electrical failure.  It's likely a gamble no matter what one
does.  It's always something to consider tho. 

Dale

:-)  :-) 

Rich Freeman wrote:
> On Thu, Jan 2, 2020 at 5:49 AM Dale <rdalek1967@gmail.com> wrote:
>> lol I didn't think of that and I don't recall anyone else thinking of
>> it either.
> That is because syncing before unmounting doesn't do anything. Unless
> you use --lazy umount blocks until all writes are complete to a
> device. The instant it returns as far as the kernel is concerned the
> device should be safe to power off.
>
> If you do a sync first then of course the umount will complete more
> quickly, since all writes should already be flushed.
>
> I have no idea what your device is doing after it is unmounted, but it
> doesn't have anything to do with the linux kernel unless some process
> is directly accessing the raw device (very unlikely). Maybe the drive
> firmware is doing some kind of housekeeping, or maybe the drive has
> some kind of vibration in it that just makes it feel like it is doing
> something. Or maybe the NSA or Red Army has hacked your firmware and
> it is doing who knows what (yes, the NSA bit at least is a thing). In
> any case, chances are the drive manufacturer has accounted for sudden
> power loss in the design because if they didn't there would be a ton
> of complaints, since there is nothing you can do about this sort of
> thing assuming the firmware is up to something.
>
> Out of curiosity, what model drive is it? Is it by chance an SMR /
> archive drive? Due to the limitations on how those write data out I
> could see them implementing an internal filesystem that journals
> incoming data and then writes it back out after the fact. If so then
> that might happen even after the kernel thinks it is unmounted.
> However, such a drive firmware would probably use a journal that
> ensures data is safe even if power is cut mid-operation. The drive
> isn't supposed to report that a write is completed until it is
> durable.
>


This is the drive info:


root@fireball / # smartctl -i /dev/sdj
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-4.19.40-gentoo] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     ST8000AS0003-2HH188
Serial Number:    WCT0BQ2Y
LU WWN Device Id: 5 000c50 0ac7d172a
Firmware Version: 0003
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jan  2 12:27:14 2020 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

root@fireball / #


I recall reading about SMR but can't recall the details of what it is. 
As far as I know, this is just a basic 8TB drive.  I didn't get to fancy
since I knew I wouldn't be running it to much.  Honestly, for this task
most any drive would do. 

I added the sync command to my little script just as a added measure. 
It may not matter but at least I know it should be in sync according to
the kernel.  What the drive does when it gets to it, only the drive knows. 

I might add, it doesn't always have the same feel.  There are times when
I unmount the drive and it just sits there.  I can tell it is spinning
but the heads aren't moving.  Most of the time tho, it has this little
bumpy feel.  It sort of seems random.  It's a lot like it feels when I'm
doing a backup just not nearly as much.  When doing a backup it has that
bumpy feel a lot.  I can feel it on my keyboard even.  Once unmounted,
it still does it but a lot less frequent.  The drive finished a run of
the script while typing the last paragraph.  I've typed this paragraph
and I have not felt a single bump.  It's still mounted even but still no
bumpy feel.  I just unmounted it and I felt a few bumps but then it went
back to idle.  Still no bumpy feel and I'm a bit of a slow typer. 

My biggest confusion, was the files safe?  I just felt a small set of
bumps.  Felt like three or four but back to nothing again.  The lights
on the enclosure didn't change either.  A couple more bumps.  It's weird
because I can never predict when it will do it. 

Things get weird sometimes.  lol  It seems I always run into these weird
things too.  :/ 

Dale

:-)  :-) 

Ian Zimmerman wrote:
> On 2020-01-01 18:09, Dale wrote:
>
>> As some may recall, I have a 8TB external SATA hard drive that I do
>> back ups on. Usually, I back up once a day, more often if
>> needed. Usually I turn the power on, mount it, do the back ups,
>> unmount and turn the power back off. Usually it is powered up for 5
>> minutes or so. When I unmount it tho, I sometimes notice it is still
>> doing something. I can feel the mechanism for the heads moving. It has
>> a slight vibration to it. Questions are, what is it doing and should
>> I let it finish before powering it off? I'd assume that once it in
>> unmounted, the copy process is done so the files are safe. I guess it
>> is doing some sort of internal checks or something but I'm not sure.
> I have observed the same thing. But in my case, I also disconnect the
> _cable_ from the computer to the enclosure when I am done ... and still
> the drive activity goes on. From that I conclude that it is the drive
> circuitry itself doing some kind of internal housekeeping, and there is
> no point in worrying about it because one would wait forever for it to
> end.
>


That's one thing that makes it unnervey.  I'll put my hand around to the
back and feel those little bumps.  I wait until I think it is done but
just as I'm about to power it off, it bumps again.  It's so
unpredictable, I never know if it is done doing its thing or not.  Just
like now, it's unmounted, did that during last reply, it hasn't did the
bump thingy while reading your reply or me typing mine in so far.  Now
as soon as I reach around to turn it off, it'll likely do the bump thing
again.  lol 

One thing is for sure tho, if you unplug the cable, whatever it is
doing, it's internal.  Sort of hard for the puter to be doing something
when it isn't connected.  That narrows the options down a lot.  That's a
good piece of info there.

Thanks.

Dale

:-)  :-) 

On Thu, Jan 2, 2020 at 1:41 PM Dale <rdalek1967@gmail.com> wrote:
>
> Rich Freeman wrote:
> >
> > Out of curiosity, what model drive is it? Is it by chance an SMR /
> > archive drive?
>
> Device Model: ST8000AS0003-2HH188
>
> I recall reading about SMR but can't recall the details of what it is.
> As far as I know, this is just a basic 8TB drive.

This is an SMR drive. You should DEFINITELY read up on what they are.

For reads they're completely normal. For sequential writes to unused
space they're completely normal. For random writes or overwrites they
are significantly different from traditional hard drives.

They work a bit like an SSD in the sense that blocks are arranged into
larger erase regions. Within a region blocks can only be written
sequentially. If you want to overwrite one block in the middle of a
region, the drive will read the entire region into RAM, then write the
entire region sequentially with the overwritten block to a new spot on
the disk. This is just like in an SSD where if try to overwrite a
block in a region with any unTRIMmed blocks the drive must read the
entire region, erase the region, and write the modified region.

Except that in an SSD those extra reads/writes operate with SSD access
times. With an SMR drive those extra reads/writes operate with hard
drive latencies, so they're MUCH more costly.

For backup use they're usually fine, IF you're writing in a sequential
file format that is appended to. If you're using rsync to do your
backups then that isn't what you're doing and you're probably paying a
heavy penalty. If you were doing incremental backups using
tar/duplicity/whatever then you'd probably be fine.

Some filesystems might be optimized for these drives to reduce the
amount of overwriting in place. I haven't looked into it. I'd expect
a log-based filesystem to work fairly well, though those can have high
levels of fragmentation which is better suited for SSD than SMR.

These drives all have fairly active firmware that manages this special
overwrite process so that they can be used with operating systems that
are naive to how they work. I wouldn't be surprised if this is what
is causing the drive to be active after you unmount it. In theory it
should be harmless to power it off. However, leaving it powered on
probably will improve its performance as it can take care of any
garbage collection before the next time you use it. If whatever
journal it is using to speed things up gets full then you'll feel the
full brunt of any write penalties until it is flushed.

You might want to seriously consider changing to a backup format that
just creates big tail-appended files containing incremental changes.
Something like rsync that just outputs bazillions of little files is
going to create lots of random writes when things change, vs
consolidating all those changes into one file that just grows at the
end. Treat them the way you would a tape (which is what tar was
designed for).

Nothing wrong with SMR drives per se - they can potentially be cheaper
especially for backup (using an appropriate file format), and are just
as fast for reading so they're also great for infrequently changing
bulky data. However, random writes are very costly and you should be
aware of that going in...

--
Rich

??????? Original Message ???????
On Thursday, January 2, 2020 12:12 PM, Rich Freeman <rich0@gentoo.org> wrote:

> On Thu, Jan 2, 2020 at 1:41 PM Dale rdalek1967@gmail.com wrote:
>
> > Rich Freeman wrote:
> >
> > > Out of curiosity, what model drive is it? Is it by chance an SMR /
> > > archive drive?
> >
> > Device Model: ST8000AS0003-2HH188
> > I recall reading about SMR but can't recall the details of what it is.
> > As far as I know, this is just a basic 8TB drive.
>
> This is an SMR drive. You should DEFINITELY read up on what they are.
>
> For reads they're completely normal. For sequential writes to unused
> space they're completely normal. For random writes or overwrites they
> are significantly different from traditional hard drives.
>
> They work a bit like an SSD in the sense that blocks are arranged into
> larger erase regions. Within a region blocks can only be written
> sequentially. If you want to overwrite one block in the middle of a
> region, the drive will read the entire region into RAM, then write the
> entire region sequentially with the overwritten block to a new spot on
> the disk. This is just like in an SSD where if try to overwrite a
> block in a region with any unTRIMmed blocks the drive must read the
> entire region, erase the region, and write the modified region.
>
> Except that in an SSD those extra reads/writes operate with SSD access
> times. With an SMR drive those extra reads/writes operate with hard
> drive latencies, so they're MUCH more costly.
>
> For backup use they're usually fine, IF you're writing in a sequential
> file format that is appended to. If you're using rsync to do your
> backups then that isn't what you're doing and you're probably paying a
> heavy penalty. If you were doing incremental backups using
> tar/duplicity/whatever then you'd probably be fine.
>
> Some filesystems might be optimized for these drives to reduce the
> amount of overwriting in place. I haven't looked into it. I'd expect
> a log-based filesystem to work fairly well, though those can have high
> levels of fragmentation which is better suited for SSD than SMR.
>
> These drives all have fairly active firmware that manages this special
> overwrite process so that they can be used with operating systems that
> are naive to how they work. I wouldn't be surprised if this is what
> is causing the drive to be active after you unmount it. In theory it
> should be harmless to power it off. However, leaving it powered on
> probably will improve its performance as it can take care of any
> garbage collection before the next time you use it. If whatever
> journal it is using to speed things up gets full then you'll feel the
> full brunt of any write penalties until it is flushed.

> Rich

Thank you for the excellent education! I haven't read the full thread but I'd also suggest that running Wireshark on the USB port would likely help diagnose any other issues. I'm having similiar problems with an external drive "freeezing" and refusing to unmount normally and this will be my next step to diagnose it.

On Thursday, 2 January 2020 18:48:03 GMT Dale wrote:
> Ian Zimmerman wrote:
> > On 2020-01-01 18:09, Dale wrote:
> >> As some may recall, I have a 8TB external SATA hard drive that I do
> >> back ups on. Usually, I back up once a day, more often if
> >> needed. Usually I turn the power on, mount it, do the back ups,
> >> unmount and turn the power back off. Usually it is powered up for 5
> >> minutes or so. When I unmount it tho, I sometimes notice it is still
> >> doing something. I can feel the mechanism for the heads moving. It has
> >> a slight vibration to it. Questions are, what is it doing and should
> >> I let it finish before powering it off? I'd assume that once it in
> >> unmounted, the copy process is done so the files are safe. I guess it
> >> is doing some sort of internal checks or something but I'm not sure.
> >
> > I have observed the same thing. But in my case, I also disconnect the
> > _cable_ from the computer to the enclosure when I am done ... and still
> > the drive activity goes on. From that I conclude that it is the drive
> > circuitry itself doing some kind of internal housekeeping, and there is
> > no point in worrying about it because one would wait forever for it to
> > end.
>
> That's one thing that makes it unnervey. I'll put my hand around to the
> back and feel those little bumps. I wait until I think it is done but
> just as I'm about to power it off, it bumps again. It's so
> unpredictable, I never know if it is done doing its thing or not. Just
> like now, it's unmounted, did that during last reply, it hasn't did the
> bump thingy while reading your reply or me typing mine in so far. Now
> as soon as I reach around to turn it off, it'll likely do the bump thing
> again. lol

I suspect it is now reading your mind! Is this a case of AI?!

LOL!


> One thing is for sure tho, if you unplug the cable, whatever it is
> doing, it's internal. Sort of hard for the puter to be doing something
> when it isn't connected. That narrows the options down a lot. That's a
> good piece of info there.
>
> Thanks.
>
> Dale
>
> :-) :-)

In case you missed it in a previous post - have you compiled CONFIG_DM_ZONED
in your kernel to see if its performance changes?

https://zonedstorage.io/linux/config/

It's behaviour may not change whatsoever, if the drive only has an internal
(SMR firmware) data write mechanism. However, if the drive is exposing an I/O
API to the OS, then you could well see a difference to how its data storage
bumps, spins and shakes as it flushes its journal and goes about its garbage
collection process when the Linux kernel 'talks' to it.

--
Regards,

Mick

Rich Freeman wrote:
> On Thu, Jan 2, 2020 at 1:41 PM Dale <rdalek1967@gmail.com> wrote:
>> Rich Freeman wrote:
>>> Out of curiosity, what model drive is it? Is it by chance an SMR /
>>> archive drive?
>> Device Model: ST8000AS0003-2HH188
>>
>> I recall reading about SMR but can't recall the details of what it is.
>> As far as I know, this is just a basic 8TB drive.
> This is an SMR drive. You should DEFINITELY read up on what they are.
>
> For reads they're completely normal. For sequential writes to unused
> space they're completely normal. For random writes or overwrites they
> are significantly different from traditional hard drives.
>
> They work a bit like an SSD in the sense that blocks are arranged into
> larger erase regions. Within a region blocks can only be written
> sequentially. If you want to overwrite one block in the middle of a
> region, the drive will read the entire region into RAM, then write the
> entire region sequentially with the overwritten block to a new spot on
> the disk. This is just like in an SSD where if try to overwrite a
> block in a region with any unTRIMmed blocks the drive must read the
> entire region, erase the region, and write the modified region.
>
> Except that in an SSD those extra reads/writes operate with SSD access
> times. With an SMR drive those extra reads/writes operate with hard
> drive latencies, so they're MUCH more costly.
>
> For backup use they're usually fine, IF you're writing in a sequential
> file format that is appended to. If you're using rsync to do your
> backups then that isn't what you're doing and you're probably paying a
> heavy penalty. If you were doing incremental backups using
> tar/duplicity/whatever then you'd probably be fine.
>
> Some filesystems might be optimized for these drives to reduce the
> amount of overwriting in place. I haven't looked into it. I'd expect
> a log-based filesystem to work fairly well, though those can have high
> levels of fragmentation which is better suited for SSD than SMR.
>
> These drives all have fairly active firmware that manages this special
> overwrite process so that they can be used with operating systems that
> are naive to how they work. I wouldn't be surprised if this is what
> is causing the drive to be active after you unmount it. In theory it
> should be harmless to power it off. However, leaving it powered on
> probably will improve its performance as it can take care of any
> garbage collection before the next time you use it. If whatever
> journal it is using to speed things up gets full then you'll feel the
> full brunt of any write penalties until it is flushed.
>
> You might want to seriously consider changing to a backup format that
> just creates big tail-appended files containing incremental changes.
> Something like rsync that just outputs bazillions of little files is
> going to create lots of random writes when things change, vs
> consolidating all those changes into one file that just grows at the
> end. Treat them the way you would a tape (which is what tar was
> designed for).
>
> Nothing wrong with SMR drives per se - they can potentially be cheaper
> especially for backup (using an appropriate file format), and are just
> as fast for reading so they're also great for infrequently changing
> bulky data. However, random writes are very costly and you should be
> aware of that going in...
>


I had no idea this was that type of drive.  I wonder if any of my other
drives are this type as well.  If it matters, I use ext4 on my drives,
except /boot, but I've thought about using something that is also
compatible with windoze, just in case I need to copy some things to
someone else's system.  Thing is, I'm not real big on those file systems
since I lose some info such as permissions and all.  Does my using ext4
change this any?  Better or worse?

Should I change the mounting options for this drive?  I've read some
people use certain options for SSD and they are needed.  This is the
options being used according to mount:

/dev/sdj1 on /run/media/dale/8tb-backup type ext4
(rw,nosuid,nodev,relatime,uhelper=udisks2)

That I guess is the default way KDE/udisks/etc does it. 

One other question, what should I look for to avoid these types of
drives?  I'll go back and look but I don't recall it saying anything
about this.  I'm also trying to figure out if this is a good thing or not. 

Thanks much for the info.  I'll read this one again shortly. Lot of info
to absorb. 

Dale

:-)  :-) 

On 02/01/2020 19:12, Rich Freeman wrote:
> For reads they're completely normal. For sequential writes to unused
> space they're completely normal. For random writes or overwrites they
> are significantly different from traditional hard drives.

That's why the "S" stands for "shingled" (which means "overlapped",
think of a shingled roof - which is your normal overlapping tile jobbie).

The point of a shingled drive is that it has a narrow read head - let's
call it one unit wide. But a write head is larger, lets call it three
units wide. If each track is one unit wide - the correct size to be read
- then writing a track will destroy two tracks.

So the write head lays down data in concentric rings, and CAN'T
overwrite data - it has to wipe everything and rewrite it. So as
mentioned before you really do not want these drives for random writes -
think of them as a bit like a random-write tape drive ie you can get
away with it but you're better off not trying.

And yes - the 8TB capacity gave it away - I think the largest "normal"
drives available are 4TB at present ... anything bigger must be shingled.

Cheers,
Wol

On Thu, Jan 2, 2020 at 7:43 PM Wol <antlists@youngman.org.uk> wrote:
>
> And yes - the 8TB capacity gave it away - I think the largest "normal"
> drives available are 4TB at present ... anything bigger must be shingled.
>

I have several non-SMR drives in the 8-12TB range, shucked from those
WD external drives that occasionally are on sale at Best Buy. While
the drives inside are white-label people have managed to socially
engineer the support team at WD to reveal that they aren't SMR.

But, you definitely need to check.

--
Rich

Mick wrote:
> On Thursday, 2 January 2020 18:48:03 GMT Dale wrote:
>> Ian Zimmerman wrote:
>>> On 2020-01-01 18:09, Dale wrote:
>>>> As some may recall, I have a 8TB external SATA hard drive that I do
>>>> back ups on. Usually, I back up once a day, more often if
>>>> needed. Usually I turn the power on, mount it, do the back ups,
>>>> unmount and turn the power back off. Usually it is powered up for 5
>>>> minutes or so. When I unmount it tho, I sometimes notice it is still
>>>> doing something. I can feel the mechanism for the heads moving. It has
>>>> a slight vibration to it. Questions are, what is it doing and should
>>>> I let it finish before powering it off? I'd assume that once it in
>>>> unmounted, the copy process is done so the files are safe. I guess it
>>>> is doing some sort of internal checks or something but I'm not sure.
>>> I have observed the same thing. But in my case, I also disconnect the
>>> _cable_ from the computer to the enclosure when I am done ... and still
>>> the drive activity goes on. From that I conclude that it is the drive
>>> circuitry itself doing some kind of internal housekeeping, and there is
>>> no point in worrying about it because one would wait forever for it to
>>> end.
>> That's one thing that makes it unnervey. I'll put my hand around to the
>> back and feel those little bumps. I wait until I think it is done but
>> just as I'm about to power it off, it bumps again. It's so
>> unpredictable, I never know if it is done doing its thing or not. Just
>> like now, it's unmounted, did that during last reply, it hasn't did the
>> bump thingy while reading your reply or me typing mine in so far. Now
>> as soon as I reach around to turn it off, it'll likely do the bump thing
>> again. lol
> I suspect it is now reading your mind! Is this a case of AI?!
>
> LOL!
>
>
>> One thing is for sure tho, if you unplug the cable, whatever it is
>> doing, it's internal. Sort of hard for the puter to be doing something
>> when it isn't connected. That narrows the options down a lot. That's a
>> good piece of info there.
>>
>> Thanks.
>>
>> Dale
>>
>> :-) :-)
> In case you missed it in a previous post - have you compiled CONFIG_DM_ZONED
> in your kernel to see if its performance changes?
>
> https://zonedstorage.io/linux/config/
>
> It's behaviour may not change whatsoever, if the drive only has an internal
> (SMR firmware) data write mechanism. However, if the drive is exposing an I/O
> API to the OS, then you could well see a difference to how its data storage
> bumps, spins and shakes as it flushes its journal and goes about its garbage
> collection process when the Linux kernel 'talks' to it.
>


It seems that is not enabled in my kernel. 


Symbol: DM_ZONED [=n]


It took some effort to find and enable everything to get that but I
finally found them all.  It seems three things has to be enabled in
order for that option to be visible.  Then after that, make wouldn't
work.  I checked the gcc settings, emerged libtool and checked on
everything else with no improvement.  Then I copied the .config file to
a safe spot and did a make clean.  When I copied the file back and ran
make again, it worked.  No clue on what was up with that.  You know my
luck tho.  ;-)

I'll try to reboot the new kernel in a bit.  It's building at the
moment.  Thanks for posting about this.  I did not see it in other
replies.  I thought it might be in Rich's but didn't see it.  The extra
nudge was helpful.

Dale

:-)  :-)