Mailing List Archive

understanding storage group balancing
I've been observing that no recordings are being written to one of my
directories in my Storage Group - Default.  I have 2 directories defined
but only one has any files in it. The volume that is getting all the
recordings has  about 300GB of free space more than the other volume.

df -h shows:

/dev/md1        1.8T  402G  1.4T  24% /mnt/md1
/dev/md0        1.8T  702G  1.1T  41% /mnt/md0

jim@mythbuntu:~$ ls /mnt/md1/recorded-tv2/
10404_20200806103000.ts      11701_20200806110000.ts
10404_20200806103000.ts.png  11701_20200806110000.ts.png
jim@mythbuntu:~$ ls /mnt/md0/recorded-tv1/
lost+found
jim@mythbuntu:~$

So my question: is mythtv using the storage group directory that is on
the volume with the most free disk space?

I was kind of hoping that it would use the 2 directories on different
volumes to balance the writes for multiple recordings at the same time.

Jim A

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
On 8/6/20 11:07 AM, Jim Abernathy wrote:
> I've been observing that no recordings are being written to one of my directories in my Storage Group - Default.  I have 2 directories defined
> but only one has any files in it. The volume that is getting all the recordings has  about 300GB of free space more than the other volume.
>
> df -h shows:
>
> /dev/md1        1.8T  402G  1.4T  24% /mnt/md1
> /dev/md0        1.8T  702G  1.1T  41% /mnt/md0
>
> jim@mythbuntu:~$ ls /mnt/md1/recorded-tv2/
> 10404_20200806103000.ts      11701_20200806110000.ts
> 10404_20200806103000.ts.png  11701_20200806110000.ts.png
> jim@mythbuntu:~$ ls /mnt/md0/recorded-tv1/
> lost+found
> jim@mythbuntu:~$
>
> So my question: is mythtv using the storage group directory that is on the volume with the most free disk space?

Check which "Storage Group Disk Scheduler" is in use.

You probably want "Balanced Free Space" 'til the 2nd disk fills.

Then switch to your preferred scheduler, e.g."Balanced Disk I/O"

Ref: https://www.mythtv.org/wiki/Setup_Storage_Directories#Storage_Group_Disk_Scheduler

> I was kind of hoping that it would use the 2 directories on different volumes to balance the writes for multiple recordings at the same time.

It can.

--
Bill
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
On Thu, 6 Aug 2020 11:28:26 -0500, you wrote:

>On 8/6/20 11:07 AM, Jim Abernathy wrote:
>> I've been observing that no recordings are being written to one of my directories in my Storage Group - Default.? I have 2 directories defined
>> but only one has any files in it. The volume that is getting all the recordings has? about 300GB of free space more than the other volume.
>>
>> df -h shows:
>>
>> /dev/md1??????? 1.8T? 402G? 1.4T? 24% /mnt/md1
>> /dev/md0??????? 1.8T? 702G? 1.1T? 41% /mnt/md0
>>
>> jim@mythbuntu:~$ ls /mnt/md1/recorded-tv2/
>> 10404_20200806103000.ts????? 11701_20200806110000.ts
>> 10404_20200806103000.ts.png? 11701_20200806110000.ts.png
>> jim@mythbuntu:~$ ls /mnt/md0/recorded-tv1/
>> lost+found
>> jim@mythbuntu:~$
>>
>> So my question: is mythtv using the storage group directory that is on the volume with the most free disk space?
>
>Check which "Storage Group Disk Scheduler" is in use.
>
>You probably want "Balanced Free Space" 'til the 2nd disk fills.
>
>Then switch to your preferred scheduler, e.g."Balanced Disk I/O"
>
>Ref: https://www.mythtv.org/wiki/Setup_Storage_Directories#Storage_Group_Disk_Scheduler
>
>> I was kind of hoping that it would use the 2 directories on different volumes to balance the writes for multiple recordings at the same time.
>
>It can.

If you want to correct the balance immediately, I have a script that
does that:

http://www.jsw.gen.nz/mythtv/balance_storage.sh

It will move files from the most full drive to the most empty one
until all directories in the Default storage group are within 20
Gibytes of the same amount of free space. It stops automatically when
it gets too close to the time for a recording to start, or you can
stop it manually at any time. The files to move are selected
randomly. See the comments at the beginning of the file for how to
use it.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
On 8/6/20 12:28 PM, Bill Meek wrote:
> On 8/6/20 11:07 AM, Jim Abernathy wrote:
>> I've been observing that no recordings are being written to one of my
>> directories in my Storage Group - Default.  I have 2 directories
>> defined but only one has any files in it. The volume that is getting
>> all the recordings has about 300GB of free space more than the other
>> volume.
>>
>> df -h shows:
>>
>> /dev/md1        1.8T  402G  1.4T  24% /mnt/md1
>> /dev/md0        1.8T  702G  1.1T  41% /mnt/md0
>>
>> jim@mythbuntu:~$ ls /mnt/md1/recorded-tv2/
>> 10404_20200806103000.ts      11701_20200806110000.ts
>> 10404_20200806103000.ts.png  11701_20200806110000.ts.png
>> jim@mythbuntu:~$ ls /mnt/md0/recorded-tv1/
>> lost+found
>> jim@mythbuntu:~$
>>
>> So my question: is mythtv using the storage group directory that is
>> on the volume with the most free disk space?
>
> Check which "Storage Group Disk Scheduler" is in use.
>
> You probably want "Balanced Free Space" 'til the 2nd disk fills.
>
> Then switch to your preferred scheduler, e.g."Balanced Disk I/O"
>
> Ref:
> https://www.mythtv.org/wiki/Setup_Storage_Directories#Storage_Group_Disk_Scheduler
>
>> I was kind of hoping that it would use the 2 directories on different
>> volumes to balance the writes for multiple recordings at the same time.
>
> It can.
>
> --
> Bill


So I have the default disk scheduler of "Balanced Free Space". That
explains the problem.  Since the backend is also a NAS for the home, I
just remembered I run Macrium Reflect to put backup images files on the
NAS and that was going to only one of the volumes.  So it was growing
outside of Mythtv and it was trying to balance by putting recordings on
the other one.

I think I want to get it balanced first and then change to "Balanced
Disk I/O". Since the disk I'm using are four 2TB drives in two RAID-1
mirrors on a 3Gb/s SATA interface, balanced I/O might be better.

I should probably move the Macrium Reflect image backups to a new USB3.1
external case I have with a Samsung EV0 970 NVMe m2-2280 SSD in it. It
does the backup averaging over 1.4Gb/s; way higher than a Gbit enet
connection to the NAS; 10 minutes vs. 30 minutes.

Jim A


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
On 8/6/20 1:23 PM, Stephen Worthington wrote:
> On Thu, 6 Aug 2020 11:28:26 -0500, you wrote:
>
>> On 8/6/20 11:07 AM, Jim Abernathy wrote:
>>> I've been observing that no recordings are being written to one of my directories in my Storage Group - Default.  I have 2 directories defined
>>> but only one has any files in it. The volume that is getting all the recordings has  about 300GB of free space more than the other volume.
>>>
>>> df -h shows:
>>>
>>> /dev/md1        1.8T  402G  1.4T  24% /mnt/md1
>>> /dev/md0        1.8T  702G  1.1T  41% /mnt/md0
>>>
>>> jim@mythbuntu:~$ ls /mnt/md1/recorded-tv2/
>>> 10404_20200806103000.ts      11701_20200806110000.ts
>>> 10404_20200806103000.ts.png  11701_20200806110000.ts.png
>>> jim@mythbuntu:~$ ls /mnt/md0/recorded-tv1/
>>> lost+found
>>> jim@mythbuntu:~$
>>>
>>> So my question: is mythtv using the storage group directory that is on the volume with the most free disk space?
>> Check which "Storage Group Disk Scheduler" is in use.
>>
>> You probably want "Balanced Free Space" 'til the 2nd disk fills.
>>
>> Then switch to your preferred scheduler, e.g."Balanced Disk I/O"
>>
>> Ref: https://www.mythtv.org/wiki/Setup_Storage_Directories#Storage_Group_Disk_Scheduler
>>
>>> I was kind of hoping that it would use the 2 directories on different volumes to balance the writes for multiple recordings at the same time.
>> It can.
> If you want to correct the balance immediately, I have a script that
> does that:
>
> http://www.jsw.gen.nz/mythtv/balance_storage.sh
>
> It will move files from the most full drive to the most empty one
> until all directories in the Default storage group are within 20
> Gibytes of the same amount of free space. It stops automatically when
> it gets too close to the time for a recording to start, or you can
> stop it manually at any time. The files to move are selected
> randomly. See the comments at the beginning of the file for how to
> use it.

I'm going to check out the script for later use, but I think my problem
is outside use of the drives and not a mythtv problem.

Jim A


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
On Thu, 6 Aug 2020 13:41:15 -0400, you wrote:

>
>On 8/6/20 1:23 PM, Stephen Worthington wrote:
>> On Thu, 6 Aug 2020 11:28:26 -0500, you wrote:
>>
>>> On 8/6/20 11:07 AM, Jim Abernathy wrote:
>>>> I've been observing that no recordings are being written to one of my directories in my Storage Group - Default.? I have 2 directories defined
>>>> but only one has any files in it. The volume that is getting all the recordings has? about 300GB of free space more than the other volume.
>>>>
>>>> df -h shows:
>>>>
>>>> /dev/md1??????? 1.8T? 402G? 1.4T? 24% /mnt/md1
>>>> /dev/md0??????? 1.8T? 702G? 1.1T? 41% /mnt/md0
>>>>
>>>> jim@mythbuntu:~$ ls /mnt/md1/recorded-tv2/
>>>> 10404_20200806103000.ts????? 11701_20200806110000.ts
>>>> 10404_20200806103000.ts.png? 11701_20200806110000.ts.png
>>>> jim@mythbuntu:~$ ls /mnt/md0/recorded-tv1/
>>>> lost+found
>>>> jim@mythbuntu:~$
>>>>
>>>> So my question: is mythtv using the storage group directory that is on the volume with the most free disk space?
>>> Check which "Storage Group Disk Scheduler" is in use.
>>>
>>> You probably want "Balanced Free Space" 'til the 2nd disk fills.
>>>
>>> Then switch to your preferred scheduler, e.g."Balanced Disk I/O"
>>>
>>> Ref: https://www.mythtv.org/wiki/Setup_Storage_Directories#Storage_Group_Disk_Scheduler
>>>
>>>> I was kind of hoping that it would use the 2 directories on different volumes to balance the writes for multiple recordings at the same time.
>>> It can.
>> If you want to correct the balance immediately, I have a script that
>> does that:
>>
>> http://www.jsw.gen.nz/mythtv/balance_storage.sh
>>
>> It will move files from the most full drive to the most empty one
>> until all directories in the Default storage group are within 20
>> Gibytes of the same amount of free space. It stops automatically when
>> it gets too close to the time for a recording to start, or you can
>> stop it manually at any time. The files to move are selected
>> randomly. See the comments at the beginning of the file for how to
>> use it.
>
>I'm going to check out the script for later use, but I think my problem
>is outside use of the drives and not a mythtv problem.
>
>Jim A

That is one of the reasons to use the script. I run it when I put
heaps of new music files in my music directory, which is on one of my
recording drives. The other main use I have for it is when I add or
upgrade a recording drive. Recently I upgraded an old 3 Tbyte drive
to a new 16 Tbyte one, so I had to run the script over several days to
balance the free space again.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
Bill Meek <keemllib@gmail.com> wrote:

> Check which "Storage Group Disk Scheduler" is in use.
>
> You probably want "Balanced Free Space" 'til the 2nd disk fills.
>
> Then switch to your preferred scheduler, e.g."Balanced Disk I/O"
>
> Ref: https://www.mythtv.org/wiki/Setup_Storage_Directories#Storage_Group_Disk_Scheduler

As I read the link above, specifically the advanced notes, it would appear that Balanced Disk I/O is now equivalent to Combined if either all storage is local or all is remote. If everything is either local or remote, then the decision tree would appear to boil down to :
> if fs1 or fs2 has a lower weight, use the one with the lowest weight
> else if both have equal weights then use the one with the most free space

Is that right ?

If so, then I guess for many (most ?) of us, Balanced Disk I/O makes the most sense - I have Combination sets as that's what came over from my older system when I upgraded.

In my experience, as long as you are watching and deleting stuff, then that seems to keep the free space fairly balanced. If recording one thing, it's going to go to the fs with the most free space (eights would be equal, so choose most free space). Two recordings would be split across the two fs. Three would be split with 2 to the fs with the most free space and 1 to the other, and so on.
Of course, if you add an empty drive then it's still going to use the fuller drives - but as long as you are watching and deleting stuff faster than it's recording to those then you'll still be OK.

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
Simon Hobson says:
> As I read the link above, specifically the advanced notes, it would
> appear that Balanced Disk I/O is now equivalent to Combined if
> either all storage is local or all is remote.

Author of the balanced disk I/O scheduler here. The answer to your question is "No". That you thought "Yes" is quite understandable, though, because I've been where you are.

The short version is that combination did not (and does not, unless it has since been fixed) work the way the wiki and documentation describes it, causing big problems for anyone with multiple storage directories in both local and remote locations. If you want the whole sordid story, read <https://lists.archive.carbon60.com/mythtv/users/346903/?page=1;mh=-1;> from back when combination was the only algorithm (and thus not called "combination").

Balanced free space and balanced percent free space were created after the above thread to fix the blatantly broken behavior of combination, with balanced free space being made default as it is the one that makes the most intuitive sense for most people. I submitted balanced disk I/O later, after experiencing problems with balanced free space when multiple tuners started writing to the same RAID 6 filesystem at the same time.

> If so, then I guess for many (most ?) of us, Balanced Disk I/O makes
> the most sense

One can make the case for either it or balanced free space as the best default choice. Balanced free space is the best fit for the typical user with one or two tuners who occasionally adds a new drive, and expects that one to be used for all new recordings until it is as full as the older filesystems. This also eliminates expiration of older programs on older filesystems to make room before free space parity occurs. Balanced disk I/O does not behave so intuitively, but is preferable for those using more than (say) four or five tuners and/or RAID 5/6 (inherently slower than single drives). In practice with balanced disk I/O, any new drive will get used more often than older filesystems and thus eventually also reach parity in terms of free space, albeit more slowly than with balanced free space.

--
Frontend: Apple MacBook Pro 2012, Nvidia Shield 2017
Backend: HP Microserver N40L 1.5GHz with 4x3TB HDDs
Tuners: Two over-the-air ATSC inputs with multirec
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
Yeechang Lee <ylee@columbia.edu> wrote:

>> As I read the link above, specifically the advanced notes, it would
>> appear that Balanced Disk I/O is now equivalent to Combined if
>> either all storage is local or all is remote.
>
> Author of the balanced disk I/O scheduler here. The answer to your question is "No". That you thought "Yes" is quite understandable, though, because I've been where you are.
>
> The short version is that combination did not (and does not, unless it has since been fixed) work the way the wiki and documentation describes it, causing big problems for anyone with multiple storage directories in both local and remote locations.

I did say "if all storage is local, or all is remote" ;-)
TBH, the description in the wiki is incredibly hard to follow with no visual clues as to which "else" goes with which "if"

> If you want the whole sordid story, read <https://lists.archive.carbon60.com/mythtv/users/346903/?page=1;mh=-1;> from back when combination was the only algorithm (and thus not called "combination").

Yes, I can see the problem there. Ouch. Doesn't take much thought to see how weighting on local vs remote is always going to create such a problem in the absence of some other mechanism (e.g. manual or scripted moving of files from local to remote storage). IMO there's an argument that if there's a performance problem with remote storage, the answer would have to be to put new recordings onto local storage and then move them when the system is quiet.

> One can make the case for either it or balanced free space as the best default choice. Balanced free space is the best fit for the typical user with one or two tuners who occasionally adds a new drive, and expects that one to be used for all new recordings until it is as full as the older filesystems. This also eliminates expiration of older programs on older filesystems to make room before free space parity occurs. Balanced disk I/O does not behave so intuitively, but is preferable for those using more than (say) four or five tuners and/or RAID 5/6 (inherently slower than single drives). In practice with balanced disk I/O, any new drive will get used more often than older filesystems and thus eventually also reach parity in terms of free space, albeit more slowly than with balanced free space.

Yup, that all makes sense.
I have two tuners (UK DVB-T). With multirec I can get 5 or 6 recordings going at once during the pre/post roll overlaps - but mostly have only 1 or 2. It's been working fine with 2 drives (local, SATA connected), currently about 100G free out of 6T and it's keeping free space nicely balanced.

And thank you (and all the other devs) for your contributions. Since I upgraded everything, the WAF is such that she's happy to use it herself now - though I have to setup any recording rules, she's not that adventurous !

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
On Sun, 9 Aug 2020 09:50:58 +0100, you wrote:

>Yes, I can see the problem there. Ouch. Doesn't take much thought to see how weighting on local vs remote is always going to create such a problem in the absence of some other mechanism (e.g. manual or scripted moving of files from local to remote storage). IMO there's an argument that if there's a performance problem with remote storage, the answer would have to be to put new recordings onto local storage and then move them when the system is quiet.

My mythsgu program should be able to move recordings when the system
is idle. It is not currently set up to do this exact job, but would
only take a few tweaks, if anyone wants to try it. But I still have
to convert it to Python 3 - I am still on v30 until I can do that.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: understanding storage group balancing [ In reply to ]
Simon Hobson says:
> And thank you (and all the other devs) for your contributions.

I am merely a user who has been around long enough, and motivated enough, to have contributed patches and bug reports. Balanced disk I/O existed before my contribution in that area, but by adding the free space routine I like to think that I helped make it actually useful, as opposed to ... well, I can't think of a use case for the previous version at all, but it did exist.

--
Frontend: Apple MacBook Pro 2012, Nvidia Shield 2017
Backend: HP Microserver N40L 1.5GHz with 4x3TB HDDs
Tuners: Two over-the-air ATSC inputs with multirec
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org