Mailing List Archive

backend storage performance needs?
Just an opinion request here.  When I built my current backend it was an
old motherboard with lots of SATA II (3Gb/s) ports. I was worried about
the need to have 2 recording drives defined so 2 simultaneous recordings
would go to different drives.

I also wanted RAID mirrors so I used 4-2TB drives with mdadm RAID 1. A
lot to maintain as drives do fail and RAID volumes have to be replaced
and rebuilt. Replacing and rebuilding has occurred on average of once a
year.

So the question is, if I bought a new motherboard with SATA III (6Gb.s)
ports and new SATA III high performance hard drives would I really need
to define 2 recording drives? i.e. use 2-4TB drives in RAID 1 mirror
configuration.

My tuner is a PCIe Hauppauge WinTV Quat HD. I have 4 tuners defined with
each tuner having a maximum recordings of 2. But I've never recorded
more than 4 channels at once with some back-to-back programs on the same
channel.

With the current speed of processors, serial connections, and hard
drives do I still need 2 recording drives defined?

Since SSDs and RAID is not recommended, I avoided that in the discussion.

Jim A


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On Tue, 19 May 2020 06:44:41 -0400, you wrote:

>Just an opinion request here.? When I built my current backend it was an
>old motherboard with lots of SATA II (3Gb/s) ports. I was worried about
>the need to have 2 recording drives defined so 2 simultaneous recordings
>would go to different drives.
>
>I also wanted RAID mirrors so I used 4-2TB drives with mdadm RAID 1. A
>lot to maintain as drives do fail and RAID volumes have to be replaced
>and rebuilt. Replacing and rebuilding has occurred on average of once a
>year.
>
>So the question is, if I bought a new motherboard with SATA III (6Gb.s)
>ports and new SATA III high performance hard drives would I really need
>to define 2 recording drives? i.e. use 2-4TB drives in RAID 1 mirror
>configuration.
>
>My tuner is a PCIe Hauppauge WinTV Quat HD. I have 4 tuners defined with
>each tuner having a maximum recordings of 2. But I've never recorded
>more than 4 channels at once with some back-to-back programs on the same
>channel.
>
>With the current speed of processors, serial connections, and hard
>drives do I still need 2 recording drives defined?
>
>Since SSDs and RAID is not recommended, I avoided that in the discussion.
>
>Jim A

There is no exact answer to just how many recordings at once you can
do with one hard drive. There are quite a number of factors to
consider. For example, if you are recording from all four tuners, and
you have back-to-back programmes scheduled for all four tuners, you
can suddenly be recording eight programmes at once for the overlap
period. On top of that, you may also be playing back one recording
per frontend. And as you get more locations on the disk where there
is active writing (or reading), that means the heads will be spending
little time on station before they have to move again to the next
active location. It is not a linear problem - going from four to five
active locations uses far more disk resource (available head movement)
than going from one to two active locations.

Then you get the problem of the filesystem. Each recording will get
to a point where it needs more space to be allocated. So the heads
need to move from where the recording is to the system areas to find
and allocate that space. That may be fine when it happens for one
recording, but if it happens for all four or all eight at once,
suddenly the heads are moving great distances back and forth and may
be too late getting back to where the recording is being written.
Filesystems vary as to how they handle that - some have the system
areas spread out across the disk to allow shorter head movement. Some
have all the system areas at one end of the disk.

Then there is the organisation of the disk into tracks and cylinders.
A disk with more platters and fewer tracks per platter has less
distance to move the heads between things.

Then you need to consider what may happen when the drive is getting
full. If the filesystem is not good at keeping the free space
contiguous, then the heads will have to move much longer distances
between the files.

Modern fast hard drives can have tremendous sequential write speeds -
those have been increasing all the time as the rotational bit
densities increase. But the speeds of head movement have not
increased in proportion. So they tend to be the limiting factor, not
the write speed. And the drive manufacturers are much less
forthcoming about stepping rates and the like - they like to advertise
the super sequential write speeds.

And then there is the cache - a large RAM cache and a good caching
algorithm (where the head movement is minimised) can make a big
difference when the heads have to move away from a recording to update
the system areas. The updates for all the open files can be delayed
and then all done at once, rather than the heads moving back and
forth.

Personally, I am quite conservative with the number of drives versus
number of simultaneous recordings. Part of that is because I have
several older drives (3 x WD Green/Red, 6-7 years old), but also
because when I started out doing lots of recordings at once, I did get
occasions where I had all the recordings I was doing simultaneously
fail or be corrupted due to exceeding the ability of my (then two)
drives to handle the load. So now I have seven recording drives, and
am upgrading them slowly to much more modern and higher capacity. I
plan on no more than two recordings at once per drive. I do regularly
(at least once a week) do 10-12 recordings at once, and with that
setup I have never had a problem, even when the load increases sharply
(potentially doubles) during back-to-back overlaps. I only use normal
drives, not RAID. I can not afford the extra costs of doing RAID.

Except for their noise, my favourite drive is now the Seagate Exos
series enterprise helium drives. I have an ST12000NM0007 (12 Tbytes)
and an ST14000NM0018 (14 Tbytes) in my main MythTV box now. They are
amazingly fast, and are good at handling transaction processing where
the data can be spread across the entire disk. And they are much
cheaper than anything else with the same performance - they are
actually cheaper here than the high grade NAS rated drives. As
enterprise drives, I expect them to have a long lifetime, but I have
yet to find out just how long they will last - only time will tell. I
have been very happy with all the WD drives just keeping on going for
8+ years. And Seagate did make some horribly bad drives with very
high failure rates for a while - I had 5 of those drives all die early
(< 2 years) and only one of them that lasted, and that only for about
5 years. Fortunately, they were mostly so bad that they died while
still under warranty, and under New Zealand law I was able to refuse
to accept a replacement with the same type of drive as they clearly
were not of an acceptable standard of durability. I got very good at
recovering the data onto the replacement drive, so I only lost maybe
10-15 recordings from all of that, as the mode of death was to get
more and more difficult to read sectors, rather than the entire drive
just collapsing and all the data being gone.

If you can afford them, even better than the Exos drives are the
Hitachi (now WD) helium drives. Unfortunately, their price is
significantly higher for the same capacity - maybe 40% more. But the
Hitachi engineers have always made very reliable drives. The two
three Tbyte ones I retired when I upgraded to my Exos drives had been
in service 24/7 for eight and nine years respectively. And one is
back in service on my mother's MythTV box as she ran out of space.

Do not buy the cheap "desktop" drives - they are unsuitable for 24/7
operation and will die early if used like that. You want at least NAS
rated drives.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On Tue, May 19, 2020 at 10:04 AM Stephen Worthington <
stephen_agent@jsw.gen.nz> wrote:

> On Tue, 19 May 2020 06:44:41 -0400, you wrote:
>
> >Just an opinion request here. When I built my current backend it was an
> >old motherboard with lots of SATA II (3Gb/s) ports. I was worried about
> >the need to have 2 recording drives defined so 2 simultaneous recordings
> >would go to different drives.
> >
> >I also wanted RAID mirrors so I used 4-2TB drives with mdadm RAID 1. A
> >lot to maintain as drives do fail and RAID volumes have to be replaced
> >and rebuilt. Replacing and rebuilding has occurred on average of once a
> >year.
> >
> >So the question is, if I bought a new motherboard with SATA III (6Gb.s)
> >ports and new SATA III high performance hard drives would I really need
> >to define 2 recording drives? i.e. use 2-4TB drives in RAID 1 mirror
> >configuration.
> >
> >My tuner is a PCIe Hauppauge WinTV Quat HD. I have 4 tuners defined with
> >each tuner having a maximum recordings of 2. But I've never recorded
> >more than 4 channels at once with some back-to-back programs on the same
> >channel.
> >
> >With the current speed of processors, serial connections, and hard
> >drives do I still need 2 recording drives defined?
> >
> >Since SSDs and RAID is not recommended, I avoided that in the discussion.
> >
> >Jim A
>
> There is no exact answer to just how many recordings at once you can
> do with one hard drive. There are quite a number of factors to
> consider. For example, if you are recording from all four tuners, and
> you have back-to-back programmes scheduled for all four tuners, you
> can suddenly be recording eight programmes at once for the overlap
> period. On top of that, you may also be playing back one recording
> per frontend. And as you get more locations on the disk where there
> is active writing (or reading), that means the heads will be spending
> little time on station before they have to move again to the next
> active location. It is not a linear problem - going from four to five
> active locations uses far more disk resource (available head movement)
> than going from one to two active locations.
>
> Then you get the problem of the filesystem. Each recording will get
> to a point where it needs more space to be allocated. So the heads
> need to move from where the recording is to the system areas to find
> and allocate that space. That may be fine when it happens for one
> recording, but if it happens for all four or all eight at once,
> suddenly the heads are moving great distances back and forth and may
> be too late getting back to where the recording is being written.
> Filesystems vary as to how they handle that - some have the system
> areas spread out across the disk to allow shorter head movement. Some
> have all the system areas at one end of the disk.
>
> Then there is the organisation of the disk into tracks and cylinders.
> A disk with more platters and fewer tracks per platter has less
> distance to move the heads between things.
>
> Then you need to consider what may happen when the drive is getting
> full. If the filesystem is not good at keeping the free space
> contiguous, then the heads will have to move much longer distances
> between the files.
>
> Modern fast hard drives can have tremendous sequential write speeds -
> those have been increasing all the time as the rotational bit
> densities increase. But the speeds of head movement have not
> increased in proportion. So they tend to be the limiting factor, not
> the write speed. And the drive manufacturers are much less
> forthcoming about stepping rates and the like - they like to advertise
> the super sequential write speeds.
>
> And then there is the cache - a large RAM cache and a good caching
> algorithm (where the head movement is minimised) can make a big
> difference when the heads have to move away from a recording to update
> the system areas. The updates for all the open files can be delayed
> and then all done at once, rather than the heads moving back and
> forth.
>
> Personally, I am quite conservative with the number of drives versus
> number of simultaneous recordings. Part of that is because I have
> several older drives (3 x WD Green/Red, 6-7 years old), but also
> because when I started out doing lots of recordings at once, I did get
> occasions where I had all the recordings I was doing simultaneously
> fail or be corrupted due to exceeding the ability of my (then two)
> drives to handle the load. So now I have seven recording drives, and
> am upgrading them slowly to much more modern and higher capacity. I
> plan on no more than two recordings at once per drive. I do regularly
> (at least once a week) do 10-12 recordings at once, and with that
> setup I have never had a problem, even when the load increases sharply
> (potentially doubles) during back-to-back overlaps. I only use normal
> drives, not RAID. I can not afford the extra costs of doing RAID.
>
> Except for their noise, my favourite drive is now the Seagate Exos
> series enterprise helium drives. I have an ST12000NM0007 (12 Tbytes)
> and an ST14000NM0018 (14 Tbytes) in my main MythTV box now. They are
> amazingly fast, and are good at handling transaction processing where
> the data can be spread across the entire disk. And they are much
> cheaper than anything else with the same performance - they are
> actually cheaper here than the high grade NAS rated drives. As
> enterprise drives, I expect them to have a long lifetime, but I have
> yet to find out just how long they will last - only time will tell. I
> have been very happy with all the WD drives just keeping on going for
> 8+ years. And Seagate did make some horribly bad drives with very
> high failure rates for a while - I had 5 of those drives all die early
> (< 2 years) and only one of them that lasted, and that only for about
> 5 years. Fortunately, they were mostly so bad that they died while
> still under warranty, and under New Zealand law I was able to refuse
> to accept a replacement with the same type of drive as they clearly
> were not of an acceptable standard of durability. I got very good at
> recovering the data onto the replacement drive, so I only lost maybe
> 10-15 recordings from all of that, as the mode of death was to get
> more and more difficult to read sectors, rather than the entire drive
> just collapsing and all the data being gone.
>
> If you can afford them, even better than the Exos drives are the
> Hitachi (now WD) helium drives. Unfortunately, their price is
> significantly higher for the same capacity - maybe 40% more. But the
> Hitachi engineers have always made very reliable drives. The two
> three Tbyte ones I retired when I upgraded to my Exos drives had been
> in service 24/7 for eight and nine years respectively. And one is
> back in service on my mother's MythTV box as she ran out of space.
>
> Do not buy the cheap "desktop" drives - they are unsuitable for 24/7
> operation and will die early if used like that. You want at least NAS
> rated drives.
>

Thanks for your thoughts.

I'm wondering if instead of RAID 1, I should just use 1-2TB SSD as a
recording drive and then once a night do a cronjob that rsync's the
recordings to an external drive for backup. I really only have about 800GB
of actively use space for recordings because once my wife and I have seen
them we delete them. Not building a library.

I was doing the rsync part in preparation to upgrading to mythtv V31 in
case I could not just move the current RAID.

Also I know that I can record 4 HD programs while watching a recording all
at once because I did this as a test on my RPi4, HDHR Quatro, and
USB3-to-SATAIII 1TB SSD.

The RAID needs are because I also use my backend as a NAS for other data,
mostly backups.
Then my drive list would be:

- 1-2TB SSD as boot drive and Mythtv Recordings
- 2-2TB HDD as RAID 1 mirror for general NAS use for home computers.
- 1 external 2TB HDD drive to backup recordings daily.

That would take head movement out of the equation and at most I'd lose one
day's recordings.

Jim A
Re: backend storage performance needs? [ In reply to ]
---- On Tue, 19 May 2020 11:44:41 +0100 Jim Abernathy <jfabernathy@gmail.com> wrote ----
> Just an opinion request here. When I built my current backend it was an
> old motherboard with lots of SATA II (3Gb/s) ports. I was worried about
> the need to have 2 recording drives defined so 2 simultaneous recordings
> would go to different drives.
>
> I also wanted RAID mirrors so I used 4-2TB drives with mdadm RAID 1. A
> lot to maintain as drives do fail and RAID volumes have to be replaced
> and rebuilt. Replacing and rebuilding has occurred on average of once a
> year.
>
> So the question is, if I bought a new motherboard with SATA III (6Gb.s)
> ports and new SATA III high performance hard drives would I really need
> to define 2 recording drives? i.e. use 2-4TB drives in RAID 1 mirror
> configuration.
>
> My tuner is a PCIe Hauppauge WinTV Quat HD. I have 4 tuners defined with
> each tuner having a maximum recordings of 2. But I've never recorded
> more than 4 channels at once with some back-to-back programs on the same
> channel.
>
> With the current speed of processors, serial connections, and hard
> drives do I still need 2 recording drives defined?
>
> Since SSDs and RAID is not recommended, I avoided that in the discussion.
>
> Jim A
>
>
i run dual DVB-S2 tuner to a 2.5" 500gb laptop drive. plus playback on 1 tv. it handles things fine although thats only upto 1080i broadcast content (UK freesat). I think i estimate around 10MB/s per video feed as the absolute minimum although more performance is always experienced. take a look at the size of one of your recordings divide by runtime in seconds, and thats your file MB/sec. times that by tuners, plus some overhead (say 20%) and that'll be your minimum requirement.

i'm currently moving off the 2.5" to raid1 WD Red's for my recordings - i found some old un-used disks that are bigger than what i currently have plus i need to expand on disk space too.

any new 3.5" drive (WD Reds for example) should be seeing around 140MB/sec peak. my (i think) 1st gen 4TB WD Red drives can handle 1gig NIC throughput maxed out and anything local on the server sees higher than that by 20%.
while red's are rated for NAS, i dont beleive mdadm requires such disks but would definately recommend disks rated for 24/7 use reguardless of what they are. Raid-rated disks are more for hardware raid rather than mdadm's software implimentation.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
---- On Tue, 19 May 2020 16:03:36 +0100 James Abernathy <jfabernathy@gmail.com> wrote ----
>
> Thanks for your thoughts.
> I'm wondering if instead of RAID 1, I should just use 1-2TB SSD as a recording drive and then once a night do a cronjob that rsync's the recordings to an external drive for backup. I really only have about 800GB of actively use space for recordings because once my wife and I have seen them we delete them. Not building a library.
> I was doing the rsync part in preparation to upgrading to mythtv V31 in case I could not just move the current RAID.
> Also I know that I can record 4 HD programs while watching a recording all at once because I did this as a test on my RPi4, HDHR Quatro, and USB3-to-SATAIII 1TB SSD.
> The RAID needs are because I also use my backend as a NAS for other data, mostly backups.Then my drive list would be:1-2TB SSD as boot drive and Mythtv Recordings
> 2-2TB HDD as RAID 1 mirror for general NAS use for home computers.
> 1 external 2TB HDD drive to backup recordings daily.
> That would take head movement out of the equation and at most I'd lose one day's recordings.
> Jim A

depends what your aim is. Raid is not backup. and backup is not raid. Raid covers for disk failures, backups cover for deletions and oversights; the non-hardware side.
personally, i'd recommend going raid for your requirements.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On Tue, 19 May 2020 11:03:36 -0400, you wrote:


>Thanks for your thoughts.
>
>I'm wondering if instead of RAID 1, I should just use 1-2TB SSD as a
>recording drive and then once a night do a cronjob that rsync's the
>recordings to an external drive for backup. I really only have about 800GB
>of actively use space for recordings because once my wife and I have seen
>them we delete them. Not building a library.
>
>I was doing the rsync part in preparation to upgrading to mythtv V31 in
>case I could not just move the current RAID.
>
>Also I know that I can record 4 HD programs while watching a recording all
>at once because I did this as a test on my RPi4, HDHR Quatro, and
>USB3-to-SATAIII 1TB SSD.
>
>The RAID needs are because I also use my backend as a NAS for other data,
>mostly backups.
>Then my drive list would be:
>
> - 1-2TB SSD as boot drive and Mythtv Recordings
> - 2-2TB HDD as RAID 1 mirror for general NAS use for home computers.
> - 1 external 2TB HDD drive to backup recordings daily.
>
>That would take head movement out of the equation and at most I'd lose one
>day's recordings.
>
>Jim A

An SSD's lifetime is normally measured in Tbytes written. If you want
to record to an SSD, you need to do the calculation of how much data
you will be writing to the SSD to do the recordings, versus its rated
lifetime. I did that for an NVMe M.2 SSD when I was getting ready to
install one, but the numbers then were not good. If I were to do my
recording to my SSD, it would expire in less than a year. As it is,
with the system and database on the SSD, it is going to be 20-25 years
before it will die, which means it will likely be replaced long before
then. That was nearly four years ago - your needs are smaller and
SSDs are better (and much bigger, which gives a larger Tbytes written
value). So that calculation may say that it is a good idea now.
Getting rid of the head movement problem is great, but SSDs have their
own limitations.

If you are going for a new motherboard, then get an NVMe SSD. Almost
all motherboards now have at least one, often two M.2 NVMe slots. The
maximum speed of a SATA interface is way below what an SSD's hardware
can do. Good NVMe SSDs are 10 times faster than a SATA one, and do
not cost that much more. And they seem to be cheaper every month.
With a SATA SSD, you would also have to calculate whether multiple
recordings plus database activity plus the usual system activity would
be too much for a single SATA 6 Gbit/s interface. There are no such
worries with NVMe.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On 19/05/2020 12:44, Jim Abernathy wrote:
> Since SSDs and RAID is not recommended, I avoided that in the discussion.

Not sure why SSDs aren't recommended. That's the way I've gone and it
sidesteps many (all?) of the issues Stephen listed.

They of course introduce other issues, such as cost, but they also have
tremendous benefits such as no noise, low power consumption, low heat
dissipation, etc. I'm not listing raw sequential speed here because
that's not needed for a MythTV backend, but essentially-zero seek time is.

It does make sense to limit needless writing to SSDs, but their
endurance has improved to the point where that may also no longer be
necessary. Things like mounting with noatime,nodiratime options. The
machine is also on a UPS so I furthermore bunch my writes (minimising
the probability that slow sequential writes to a file result in blocks
being written and rewritten) by mounting with barrier=0,commit=60.

My backend contains a load of SSDs (14TB worth), which are
SATA-connected to the motherboard and to an additional PCIE SATA
controller. That's another SSD disadvantage: they don't come in high
capacities (at least not at affordable prices).

The database resides on a RAID1 pair consisting of partitions that sit
on SSDs from different manufacturers (thereby hopefully minimising the
probability of near-simultaneous failure).

I have six tuners (two HVR1950s and one 4-tuner HDHR).

I built this machine about a year ago, and so far only 1% of the SSDs's
write endurance has been consumed. That includes all of the writes that
were necessary to migrate content from the predecessor server to these
SSDs. At this rate write endurance won't be the reason why I retire the
machine.

HTH, Jan
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:

> There is no exact answer to just how many recordings at once you can
> do with one hard drive. There are quite a number of factors to
> consider.

More of a dark art than a science :D


> And then there is the cache - a large RAM cache and a good caching
> algorithm (where the head movement is minimised) can make a big
> difference when the heads have to move away from a recording to update
> the system areas.

Doesn't Myth effectively disable caching for recordings. AIUI, the recording loop constantly writes out a chunk of data then fsyncs the file so it's written out to disk. In some ways that's terrible for performance for the reasons you cite - but it avoids the "big system cache gets full, system effectively pauses for several seconds while it's written out".
But it does mean that *as a minimum*, each recording requires two disjoint writes per second - one to write the last second (or more) of recording, the other to update the filesystem (which itself may involve more than one write).

As another data point, I have two recording drives - a 4TB WD Purple and a 3TB Toshiba. I can record something like five streams (that's the highest I generally see, usually a lot lower) while watching back another one or two - without any noticeable issues.


Some time ago (as in quite a few years) when I did have a very constrained system, I did consider playing with the caching arrangement in the recorder loop. It has a fixed ring buffer size, and writes out the data once/second. Problems happen if the hardware can't keep up, and the buffer overflows - then you lose a chunk (I suspect about the buffer size) of the recording.
I suspect that in a system with poor sync performance, making the buffer larger and lengthening the sync time would help.

Also, separating the OS & database from the recording drives make a huge difference. There's a lot of database activity - particularly at the start and end of each recording, and any time you trigger a reschedule. Splitting those made a huge difference when I was able to - having originally not had that luxury. On my new backend, I have the OS & DB on SSDs which make a great improvement.

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On Tue, 19 May 2020 17:53:36 +0100, you wrote:

>Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:

>> And then there is the cache - a large RAM cache and a good caching
>> algorithm (where the head movement is minimised) can make a big
>> difference when the heads have to move away from a recording to update
>> the system areas.
>
>Doesn't Myth effectively disable caching for recordings. AIUI, the recording loop constantly writes out a chunk of data then fsyncs the file so it's written out to disk. In some ways that's terrible for performance for the reasons you cite - but it avoids the "big system cache gets full, system effectively pauses for several seconds while it's written out".
>But it does mean that *as a minimum*, each recording requires two disjoint writes per second - one to write the last second (or more) of recording, the other to update the filesystem (which itself may involve more than one write).

You are thinking about a different cache. The RAM cache on a hard
drive is not software controllable. The firmware on the drive uses it
according to the drive's algorithms. The cache you are thinking of is
a software one in the PC's RAM. The hard drive cache is used for
things like an "elevator" algorithm. Data is normally received into
the cache and then written to disk when the drive can arrange to have
its heads in the right place. With an elevator algorithm, the drive
orders the writes so that the heads move all the way towards the
centre of the platters, writing whatever the heads can move to next in
that direction, then the heads reverse and move towards the outside of
the platters, writing whatever they come to next while moving in that
direction. So instead of the heads having to move long distances, as
long as there is enough cache, they can move much shorter distances
between writes (or reads). This is dramatically more efficient than a
first in first out queue, but does have the downside that a power
failure with lots of data left in the cache can cause a lot of damage.
But good drives can often manage to write everything in the cache to
disk in a reserved journal area between the time they lose power and
when they have to retract the heads to prevent them being damaged as
the platters spin down. Then when the drive is powered up again, the
journalled data gets written to where it is supposed to be. They do
this by using the drive motor as a generator taking energy out of the
platters' rotation to give them enough power to do the journalling and
shut down safely. The journalling area is normally next to the head
park position, so not too much energy is needed to move the heads from
the journal tracks to a safe park.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:

> You are thinking about a different cache. The RAM cache on a hard
> drive is not software controllable. The firmware on the drive uses it
> according to the drive's algorithms. The cache you are thinking of is
> a software one in the PC's RAM. The hard drive cache is used for
> things like an "elevator" algorithm.

I was wondering about that - and the inherent reliability problems it introduces.

> This is dramatically more efficient than a
> first in first out queue, but does have the downside that a power
> failure with lots of data left in the cache can cause a lot of damage.
> But good drives can often manage to write everything in the cache to
> disk in a reserved journal area between the time they lose power and
> when they have to retract the heads to prevent them being damaged as
> the platters spin down.

There's a few ifs and maybes in that little bit of text.

Logically, if the host tells the drive "write this and tell me when it's written" then I'd expect the data to be written to the platter before the drive signals that it's been written. If not, then it completely fubars anything the OS does - such as writing file data, FS data, and journal data, in a specific order to ensure data integrity no matter when the failure occurs. It's no good if the disk re-orders your writes !

So I'd assumed that when an fsync was performed, that would to all effects override (or at least significantly reduce the effectiveness of) the drives re-ordering as it's been told to flush those blocks "now".

Even assuming everything works all hunky dory - I would not trust my data integrity to "some drives will manage to save the cache some of the time" which is what "good drives can often manage to write everything in the cache to disk in a reserved journal area between the time they lose power and" effectively says. I understand what you are saying, but really a drive should not be caching anything it cannot guarantee to save no matter how sudden or otherwise the power loss may be. Anything else is just pulling the pin on your data loss grenade and hoping the drive keeps a good hold of it !

But as it's been a while since I used to follow the in depth technical stuff in this area, I could be spouting rubbish :-(

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On Tue, 19 May 2020 11:03:36 -0400
James Abernathy <jfabernathy@gmail.com> wrote:

> On Tue, May 19, 2020 at 10:04 AM Stephen Worthington <
> stephen_agent@jsw.gen.nz> wrote:
>
> > On Tue, 19 May 2020 06:44:41 -0400, you wrote:
> >
> > >Just an opinion request here. When I built my current backend it
> > >was an old motherboard with lots of SATA II (3Gb/s) ports. I was
> > >worried about the need to have 2 recording drives defined so 2
> > >simultaneous recordings would go to different drives.
> > >
> > >I also wanted RAID mirrors so I used 4-2TB drives with mdadm RAID
> > >1. A lot to maintain as drives do fail and RAID volumes have to be
> > >replaced and rebuilt. Replacing and rebuilding has occurred on
> > >average of once a year.
> > >
> > >So the question is, if I bought a new motherboard with SATA III
> > >(6Gb.s) ports and new SATA III high performance hard drives would
> > >I really need to define 2 recording drives? i.e. use 2-4TB drives
> > >in RAID 1 mirror configuration.
> > >
> > >My tuner is a PCIe Hauppauge WinTV Quat HD. I have 4 tuners
> > >defined with each tuner having a maximum recordings of 2. But I've
> > >never recorded more than 4 channels at once with some back-to-back
> > >programs on the same channel.
> > >
> > >With the current speed of processors, serial connections, and hard
> > >drives do I still need 2 recording drives defined?
> > >
> > >Since SSDs and RAID is not recommended, I avoided that in the
> > >discussion.
> > >
> > >Jim A
> >
> > There is no exact answer to just how many recordings at once you can
> > do with one hard drive. There are quite a number of factors to
> > consider. For example, if you are recording from all four tuners,
> > and you have back-to-back programmes scheduled for all four tuners,
> > you can suddenly be recording eight programmes at once for the
> > overlap period. On top of that, you may also be playing back one
> > recording per frontend. And as you get more locations on the disk
> > where there is active writing (or reading), that means the heads
> > will be spending little time on station before they have to move
> > again to the next active location. It is not a linear problem -
> > going from four to five active locations uses far more disk
> > resource (available head movement) than going from one to two
> > active locations.
> >
> > Then you get the problem of the filesystem. Each recording will get
> > to a point where it needs more space to be allocated. So the heads
> > need to move from where the recording is to the system areas to find
> > and allocate that space. That may be fine when it happens for one
> > recording, but if it happens for all four or all eight at once,
> > suddenly the heads are moving great distances back and forth and may
> > be too late getting back to where the recording is being written.
> > Filesystems vary as to how they handle that - some have the system
> > areas spread out across the disk to allow shorter head movement.
> > Some have all the system areas at one end of the disk.
> >
> > Then there is the organisation of the disk into tracks and
> > cylinders. A disk with more platters and fewer tracks per platter
> > has less distance to move the heads between things.
> >
> > Then you need to consider what may happen when the drive is getting
> > full. If the filesystem is not good at keeping the free space
> > contiguous, then the heads will have to move much longer distances
> > between the files.
> >
> > Modern fast hard drives can have tremendous sequential write speeds
> > - those have been increasing all the time as the rotational bit
> > densities increase. But the speeds of head movement have not
> > increased in proportion. So they tend to be the limiting factor,
> > not the write speed. And the drive manufacturers are much less
> > forthcoming about stepping rates and the like - they like to
> > advertise the super sequential write speeds.
> >
> > And then there is the cache - a large RAM cache and a good caching
> > algorithm (where the head movement is minimised) can make a big
> > difference when the heads have to move away from a recording to
> > update the system areas. The updates for all the open files can be
> > delayed and then all done at once, rather than the heads moving
> > back and forth.
> >
> > Personally, I am quite conservative with the number of drives versus
> > number of simultaneous recordings. Part of that is because I have
> > several older drives (3 x WD Green/Red, 6-7 years old), but also
> > because when I started out doing lots of recordings at once, I did
> > get occasions where I had all the recordings I was doing
> > simultaneously fail or be corrupted due to exceeding the ability of
> > my (then two) drives to handle the load. So now I have seven
> > recording drives, and am upgrading them slowly to much more modern
> > and higher capacity. I plan on no more than two recordings at once
> > per drive. I do regularly (at least once a week) do 10-12
> > recordings at once, and with that setup I have never had a problem,
> > even when the load increases sharply (potentially doubles) during
> > back-to-back overlaps. I only use normal drives, not RAID. I can
> > not afford the extra costs of doing RAID.
> >
> > Except for their noise, my favourite drive is now the Seagate Exos
> > series enterprise helium drives. I have an ST12000NM0007 (12
> > Tbytes) and an ST14000NM0018 (14 Tbytes) in my main MythTV box
> > now. They are amazingly fast, and are good at handling transaction
> > processing where the data can be spread across the entire disk.
> > And they are much cheaper than anything else with the same
> > performance - they are actually cheaper here than the high grade
> > NAS rated drives. As enterprise drives, I expect them to have a
> > long lifetime, but I have yet to find out just how long they will
> > last - only time will tell. I have been very happy with all the WD
> > drives just keeping on going for 8+ years. And Seagate did make
> > some horribly bad drives with very high failure rates for a while -
> > I had 5 of those drives all die early (< 2 years) and only one of
> > them that lasted, and that only for about 5 years. Fortunately,
> > they were mostly so bad that they died while still under warranty,
> > and under New Zealand law I was able to refuse to accept a
> > replacement with the same type of drive as they clearly were not of
> > an acceptable standard of durability. I got very good at
> > recovering the data onto the replacement drive, so I only lost
> > maybe 10-15 recordings from all of that, as the mode of death was
> > to get more and more difficult to read sectors, rather than the
> > entire drive just collapsing and all the data being gone.
> >
> > If you can afford them, even better than the Exos drives are the
> > Hitachi (now WD) helium drives. Unfortunately, their price is
> > significantly higher for the same capacity - maybe 40% more. But
> > the Hitachi engineers have always made very reliable drives. The
> > two three Tbyte ones I retired when I upgraded to my Exos drives
> > had been in service 24/7 for eight and nine years respectively.
> > And one is back in service on my mother's MythTV box as she ran out
> > of space.
> >
> > Do not buy the cheap "desktop" drives - they are unsuitable for 24/7
> > operation and will die early if used like that. You want at least
> > NAS rated drives.
> >
>
> Thanks for your thoughts.
>
> I'm wondering if instead of RAID 1, I should just use 1-2TB SSD as a
> recording drive and then once a night do a cronjob that rsync's the
> recordings to an external drive for backup. I really only have about
> 800GB of actively use space for recordings because once my wife and I
> have seen them we delete them. Not building a library.
>
> I was doing the rsync part in preparation to upgrading to mythtv V31
> in case I could not just move the current RAID.
>
> Also I know that I can record 4 HD programs while watching a
> recording all at once because I did this as a test on my RPi4, HDHR
> Quatro, and USB3-to-SATAIII 1TB SSD.
>
> The RAID needs are because I also use my backend as a NAS for other
> data, mostly backups.
> Then my drive list would be:
>
> - 1-2TB SSD as boot drive and Mythtv Recordings
> - 2-2TB HDD as RAID 1 mirror for general NAS use for home
> computers.
> - 1 external 2TB HDD drive to backup recordings daily.
>
> That would take head movement out of the equation and at most I'd
> lose one day's recordings.
>
> Jim A

If you want to add another layer of complexity, set up an LVM and use
one of the SSDs as a cache for the RAID array. My suspicion is that
even if you manage to thrash that SSD to death, it will take you ages to
notice.

--
sig goes here...
bP
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On Wed, 20 May 2020 20:17:21 +0100, you wrote:

>Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:
>
>> You are thinking about a different cache. The RAM cache on a hard
>> drive is not software controllable. The firmware on the drive uses it
>> according to the drive's algorithms. The cache you are thinking of is
>> a software one in the PC's RAM. The hard drive cache is used for
>> things like an "elevator" algorithm.
>
>I was wondering about that - and the inherent reliability problems it introduces.
>
>> This is dramatically more efficient than a
>> first in first out queue, but does have the downside that a power
>> failure with lots of data left in the cache can cause a lot of damage.
>> But good drives can often manage to write everything in the cache to
>> disk in a reserved journal area between the time they lose power and
>> when they have to retract the heads to prevent them being damaged as
>> the platters spin down.
>
>There's a few ifs and maybes in that little bit of text.
>
>Logically, if the host tells the drive "write this and tell me when it's written" then I'd expect the data to be written to the platter before the drive signals that it's been written. If not, then it completely fubars anything the OS does - such as writing file data, FS data, and journal data, in a specific order to ensure data integrity no matter when the failure occurs. It's no good if the disk re-orders your writes !
>
>So I'd assumed that when an fsync was performed, that would to all effects override (or at least significantly reduce the effectiveness of) the drives re-ordering as it's been told to flush those blocks "now".
>
>Even assuming everything works all hunky dory - I would not trust my data integrity to "some drives will manage to save the cache some of the time" which is what "good drives can often manage to write everything in the cache to disk in a reserved journal area between the time they lose power and" effectively says. I understand what you are saying, but really a drive should not be caching anything it cannot guarantee to save no matter how sudden or otherwise the power loss may be. Anything else is just pulling the pin on your data loss grenade and hoping the drive keeps a good hold of it !
>
>But as it's been a while since I used to follow the in depth technical stuff in this area, I could be spouting rubbish :-(

And I was deliberately using weasel words as I have not been following
drive technology well enough to know for sure that drives will
guarantee being able to write their cache to journal space before
losing the power to do that. But that certainly seems to be what they
are aiming for. And I do not know at all whether drives receive and
act on fsync, or whether they choose to ignore that signal and keep on
doing their own caching. Or maybe they have an option you can set to
tell them which way to do things. Enterprise drives are more
configurable than desktop drives.

>Simon
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On 21/05/2020 05:11, Stephen Worthington wrote:
> And I was deliberately using weasel words as I have not been following
> drive technology well enough to know for sure that drives will
> guarantee being able to write their cache to journal space before
> losing the power to do that. But that certainly seems to be what they
> are aiming for. And I do not know at all whether drives receive and
> act on fsync, or whether they choose to ignore that signal and keep on
> doing their own caching. Or maybe they have an option you can set to
> tell them which way to do things. Enterprise drives are more
> configurable than desktop drives.

I suspect that on journaling filesystems fsync only guarantees that the
data has been committed to the journal (which is sufficient for ensuring
data integrity).

Unless you disable barriers.

http://fibrevillage.com/storage/565-what-s-barriers-how-to-enable-disable-it-on-linux

As I wrote before I run with disabled barriers. I can do that because my
server is powered through a UPS which gives it a few minutes of autonomy
in case of a power cut. In order to mitigate the risk that the server
might not shut down in time (which is a very small risk indeed) I enable
write barriers from apcupsd's onbattery script thusly:

while fs=`findmnt -l -t ext4 -O nobarrier --output=target --noheadings`
do
mount -o remount,barrier $fs
done

and disable them again if power returns before shutdown by means of the
following in the offbattery script:

while fs=`findmnt -s -l -t ext4 -O nobarrier --output=target --noheadings`
do
mount -o remount,barrier=0 $fs
done

Cheers, Jan
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: backend storage performance needs? [ In reply to ]
On 5/21/20 3:08 AM, Jan Ceuleers wrote:
> On 21/05/2020 05:11, Stephen Worthington wrote:
>> And I was deliberately using weasel words as I have not been following
>> drive technology well enough to know for sure that drives will
>> guarantee being able to write their cache to journal space before
>> losing the power to do that. But that certainly seems to be what they
>> are aiming for. And I do not know at all whether drives receive and
>> act on fsync, or whether they choose to ignore that signal and keep on
>> doing their own caching. Or maybe they have an option you can set to
>> tell them which way to do things. Enterprise drives are more
>> configurable than desktop drives.
> I suspect that on journaling filesystems fsync only guarantees that the
> data has been committed to the journal (which is sufficient for ensuring
> data integrity).
>
> Unless you disable barriers.
>
> http://fibrevillage.com/storage/565-what-s-barriers-how-to-enable-disable-it-on-linux
>
> As I wrote before I run with disabled barriers. I can do that because my
> server is powered through a UPS which gives it a few minutes of autonomy
> in case of a power cut. In order to mitigate the risk that the server
> might not shut down in time (which is a very small risk indeed) I enable
> write barriers from apcupsd's onbattery script thusly:
>
> while fs=`findmnt -l -t ext4 -O nobarrier --output=target --noheadings`
> do
> mount -o remount,barrier $fs
> done
>
> and disable them again if power returns before shutdown by means of the
> following in the offbattery script:
>
> while fs=`findmnt -s -l -t ext4 -O nobarrier --output=target --noheadings`
> do
> mount -o remount,barrier=0 $fs
> done
>
> Cheers, Jan


A quick summary of what I got from this great discussion.

1. Having a UPS on a Backend can make up for a lot of potential
issues.  I have one.
2. WD Red NAS drives are a good blend of reliability and cost.
3. SSDs might be more reliable than we think, but when they fail .....

When I was testing and setting up a Backend for my RV/camper, I used a
RPi4 and SATA SSD and got into a discussion about drive performance on
the Raspberry.  I added Quirks in my boot command line and got
performance like 155MB/s writes and 290MB/s reads on very large
sequential files. So SSD's in Backend use would be overkill on
performance for a 4-8 tuner setup. Key for the RV backend is if it fails
the recordings are also being done at home also. The RV backend is just
for convenience when on the road for a few months. (I'll do anything to
skip commercials when watching)

On my current home Backend using 2 sets of WD Red HDD mirrors I have not
seen any recordings tagged with recording problems because of drive
performance. It's always been tuner/antenna issues. So I conclude that
my normal record pattern isn't stressing any hardware.

In the past 2 years I've only replaced 2 drives before switching to WD
Red when SMART warned me via email or mdadm said a drive fell out of a
mirror. The replacing of the drive and the resync was painless. So I
have never loss a recording due to hardware failure, except when power
was lost for a day due to a hurricane.

So I can:

1. If it ain't broke, don't fix it.  or

2. Redesign and rebuild everything to keep busy during Covid-19 stay at
home orders.

Thanks,

Jim A


Jim A
Re: backend storage performance needs? [ In reply to ]
Jan Ceuleers <jan.ceuleers@gmail.com> wrote:

> As I wrote before I run with disabled barriers. I can do that because my
> server is powered through a UPS which gives it a few minutes of autonomy
> in case of a power cut.

Does your server also have redundant PSUs with at least one PSU *not* on the UPS ?
Over the years I've seen as many, possibly more, issues caused by UPS failure as I have seen them saved. Of course, you tend not to see all the brief interruptions the UPS rides through - so the numbers will be skewed by perception. It also depends on your power quality - at my last place I had systems with uptimes of over a year and no UPS, very reliable mains :-) At another place before that we had many short cuts due to long runs of overhead supplies.

It's "really annoying" to get your nice secure setup with UPS - only to have the UPS fall over and take everything down :-(


As to redundant PSUs, a friend told me a good story a while ago. He had an HP server with (IIRC) 4 PSUs - one of which failed. The server carried on working fine, until for whatever reason it got shut down - but would not power up again. It turned out that it would not startup with a missing or failed PSU.
There's always "interesting" ways the IT will turn around and bite your backside :D

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org