Mailing List Archive

Managing storage groups intelligently
I come from mythtv 0.27/0.28. So, I apologize if this is already handled
in later versions.

Is it possible to prioritize storage groups when recording? I plan to
add a 1TB SSD to the storage groups and would like recordings to choose
that over spinning disk whenever possible.

If possible, I like to try this after the above is resolved. Manually
move files between SSD and spinning disk (both are storage groups) to
keep only recently created/accessed file in SSD and also make sure at
lease 100GB space is left for any new recording. I will do this using
overnight cron job.?? Since mythtv finds recorded programs as long as
they are present in one of the storage groups, file moves should not
affect look up for media files.?? Do you see issue with my approach?

Yes, I already investigated bcache/dmcache/lvmcache etc., and was
advised that they are either overkill or unreliable or ineffective in
mythtv environment.

Regards
Ramesh
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Managing storage groups intelligently [ In reply to ]
On 01/09/2020 17:34, R. Ramesh wrote:
> I come from mythtv 0.27/0.28. So, I apologize if this is already handled
> in later versions.
>
> Is it possible to prioritize storage groups when recording? I plan to
> add a 1TB SSD to the storage groups and would like recordings to choose
> that over spinning disk whenever possible.
>
> If possible, I like to try this after the above is resolved. Manually
> move files between SSD and spinning disk (both are storage groups) to
> keep only recently created/accessed file in SSD and also make sure at
> lease 100GB space is left for any new recording. I will do this using
> overnight cron job.?? Since mythtv finds recorded programs as long as
> they are present in one of the storage groups, file moves should not
> affect look up for media files.?? Do you see issue with my approach?
>
> Yes, I already investigated bcache/dmcache/lvmcache etc., and was
> advised that they are either overkill or unreliable or ineffective in
> mythtv environment.
>
> Regards
> Ramesh

You have a few options.

1) different storage groups for the different disks
- setup recording rules to use this SG

2) both disks in 1 SG and tune the disk selection algorithm
- balanced free space, immediately comes to mind.
- the backend will prefer to use the disk with the
most free space.
- this is a user selectable configuration item.


Regards
Stuart

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Managing storage groups intelligently [ In reply to ]
On Tue, 1 Sep 2020 11:34:44 -0500, you wrote:

>I come from mythtv 0.27/0.28. So, I apologize if this is already handled
>in later versions.
>
>Is it possible to prioritize storage groups when recording? I plan to
>add a 1TB SSD to the storage groups and would like recordings to choose
>that over spinning disk whenever possible.
>
>If possible, I like to try this after the above is resolved. Manually
>move files between SSD and spinning disk (both are storage groups) to
>keep only recently created/accessed file in SSD and also make sure at
>lease 100GB space is left for any new recording. I will do this using
>overnight cron job.?? Since mythtv finds recorded programs as long as
>they are present in one of the storage groups, file moves should not
>affect look up for media files.?? Do you see issue with my approach?
>
>Yes, I already investigated bcache/dmcache/lvmcache etc., and was
>advised that they are either overkill or unreliable or ineffective in
>mythtv environment.
>
>Regards
>Ramesh.

You can do what I do, which is to have my recordings go to the
partitions in my "Default" storage group, but also have another
storage group I call "Archives". In my case, the "Default" drives CMR
drives and most of the "Archives" drives are shingled (SMR) drives,
which are unsuitable for recording to. I currently have 7 drives on
"Default" so I am safe to do up to 14 recordings at once before I have
to worry about getting damaged recordings. There are 5 drives on
"Archives". When playing back recordings, MythTV will find the files
in any storage group, but will only record to the storage group
specified in the recording rule. The default storage group used when
creating new recording rules is set in the "Default (Template)"
template rule. So when my recording drives are getting full, I move
recordings from them to my "Archives" drives until I have enough free
space again.

It is safe to move files about in the storage groups as long as they
are not in use by MythTV. Since it takes a while to move recording
files as they are relatively large, you can have a problem if you
start moving a file and then MythTV needs to use it during the move.
So the safe way to do a move is to copy the file to its new location
using a different name that MythTV will not recognise as a recording
file, and then when it has been copied, delete the source file and
rename the destination file to the correct name. If you want to
prevent MythTV from using the file while it is being copied, you can
rename the source file before moving it, so MythTV will think it is
just a missing recording file.

You also need to worry about performance problems. When MythTV is
recording or playing back, it is using much less than the maximum
sequential speed of a drive. When you are copying or moving a file,
that operation normally takes the full sequential bandwidth available
on the slower of the two drives. So if MythTV is using that drive at
the same time, it may not get access to the file(s) it is using fast
enough. On playback, that results in stuttering. For recordings, it
causes damaged recordings due to buffers failing to be written to disk
in time.

As a result of these problems and probably some others I have not
thought about, it is better to not do non-MythTV use of its recording
drives at the same time as MythTV is using them, if you want to do
anything using high bandwidth.

In my case, I have written myself a Python program (mythsgu) that
moves recordings from "Default" to "Archives". I just run it manually
when I need to, but you could run it automatically if you want. It
moves the oldest recordings first. It is set up so that it gets
events from MythTV and checks the status of MythTV to ensure that it
will not move files when MythTV is busy. If it is moving a file and
it gets an event and subsequently discovers (by checking the
inuseprograms table) that MythTV has become busy (perhaps someone
started to play a recording), it will pause its move operation until
MythTV is no longer busy. It keeps track of when the next scheduled
recording will happen and pauses operations when it gets near to that
time. It is only capable of discovering MythTV activity for
recordings, as MythTV does not report when it is playing video files
as it does for recordings. That does not matter for me as all my
video files (and pictures and music files) are on a different set of
drives from my recording files.

Currently mythsgu is Python 2 - I have not upgraded to MythTV v31 yet.
I am just in the process of converting it to Python 3 after which I
will be doing the upgrade. The latest Python 2 version (0.7) is here:

http://www.jsw.gen.nz/mythtv/mythsgu

It is not very well documented as to how to set it up, so if you would
like to try it, please let me know and I can help with that.

I believe I also saw in a recent post here that it is possible to
assign priorities to the various drives in storage groups. This does
not have a proper user interface and does not seem to be very well
documented, but there was detail about how to set it up in that post.
I only skimmed over the details, but I think you could tell the
scheduler to record only up to a certain number of recordings at once
on a drive and after that it would use a lower priority drive.

One other consideration which probably does not apply to you is that
when mythsgu finishes, my recording drives normally have very
unbalanced amounts of free space. That means that the drives that
have the most free space will be preferentially chosen to be used for
new recordings. I prefer that all my recording drives have roughly
equal free space to prevent this, so after I run mythsgu, I run an
older script I wrote that balances the free space by moving randomly
chosen recordings from the recording drive with the least free space
to the one with the most free space:

http://www.jsw.gen.nz/mythtv/balance_storage.sh

This script stops itself ahead of the next scheduled recording, but is
not as sophisticated as mythsgu in being able to detect most MythTV
activity and pause when it happens. I am intending eventually to move
the balancing function to become part of mythsgu so that it will have
the same level of sophistication.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Managing storage groups intelligently [ In reply to ]
> You can do what I do, which is to have my recordings go to the
> partitions in my "Default" storage group, but also have another
> storage group I call "Archives". In my case, the "Default" drives CMR
> drives and most of the "Archives" drives are shingled (SMR) drives,
> which are unsuitable for recording to. I currently have 7 drives on
> "Default" so I am safe to do up to 14 recordings at once before I have
> to worry about getting damaged recordings. There are 5 drives on
> "Archives". When playing back recordings, MythTV will find the files
> in any storage group, but will only record to the storage group
> specified in the recording rule. The default storage group used when
> creating new recording rules is set in the "Default (Template)"
> template rule. So when my recording drives are getting full, I move
> recordings from them to my "Archives" drives until I have enough free
> space again.

Thanks for the details. It appears like I could put SSD in Default group
and nothing else. I can create HD storage group and put all the other
spinning disks there. Since recording rules usually choose Default, it
will always go to SSD.

In the night when the system is at rest, I will execute cache management
type copying from SSD?? to HD.?? I will keep the most recently accessed
files in SSD and others will go to HD. Like you suggest, I will copy xxx
as backup_xxx and delete xxx and rename backup_xxx as xxx.

Do you think this will work based on your knowledge?

Regards
Ramesh

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Managing storage groups intelligently [ In reply to ]
> You have a few options.
>
> 1) different storage groups for the different disks
> - setup recording rules to use this SG
>
> 2) both disks in 1 SG and tune the disk selection algorithm
> - balanced free space, immediately comes to mind.
> - the backend will prefer to use the disk with the
> most free space.
> - this is a user selectable configuration item.
>
>
> Regards
> Stuart
>
Thanks. It appears that putting SSD in "Default group" and moving all
spinning HDDs to another storage group should do the trick.

Disk selection algorithms are all based on space usage and my SSD will
fail next to large HD pool that I have as 12TB RAID. So, your first
method is more suitable, I think. I will copy overnight between SSD and
HD making sure that I leave more recent recordings in the SSD and still
leave enough space to record.

Regards
Ramesh

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Managing storage groups intelligently [ In reply to ]
On Tue, 1 Sep 2020 17:08:18 -0500, you wrote:

>> You can do what I do, which is to have my recordings go to the
>> partitions in my "Default" storage group, but also have another
>> storage group I call "Archives". In my case, the "Default" drives CMR
>> drives and most of the "Archives" drives are shingled (SMR) drives,
>> which are unsuitable for recording to. I currently have 7 drives on
>> "Default" so I am safe to do up to 14 recordings at once before I have
>> to worry about getting damaged recordings. There are 5 drives on
>> "Archives". When playing back recordings, MythTV will find the files
>> in any storage group, but will only record to the storage group
>> specified in the recording rule. The default storage group used when
>> creating new recording rules is set in the "Default (Template)"
>> template rule. So when my recording drives are getting full, I move
>> recordings from them to my "Archives" drives until I have enough free
>> space again.
>
>Thanks for the details. It appears like I could put SSD in Default group
>and nothing else. I can create HD storage group and put all the other
>spinning disks there. Since recording rules usually choose Default, it
>will always go to SSD.
>
>In the night when the system is at rest, I will execute cache management
>type copying from SSD?? to HD.?? I will keep the most recently accessed
>files in SSD and others will go to HD. Like you suggest, I will copy xxx
>as backup_xxx and delete xxx and rename backup_xxx as xxx.
>
>Do you think this will work based on your knowledge?
>
>Regards
>Ramesh

Yes. That is analogous to what I am doing, but you will be doing it
daily and I only do it whenever my "Default" storage group is running
low on free space.

As you will be working the SSD hard, you will need to run TRIM
(fstrim) more often than the once a week that is standard. In Ubuntu
this is now done from a systemd timer, but it used to be done from
/etc/cron.weekly. I have changed it to daily, but you would need to
do some calculations to see how often you should be doing it. Or you
could add the "discard" option to the SSD in fstab, so that TRIM will
be done every time space from a file is freed by the system. The
problem here is that an SSD takes up to several seconds to erase one
of its storage blocks. So you need to make sure it has already erased
blocks available, otherwise all write operations will be so slow as to
cause recordings to fail (and other problems for the system too). The
way that SSD users are told to arrange this is to have 10% spare
blocks on the SSD that are not ever allocated to a partition, so that
even if the SSD is full, every time fstrim runs at least 10% of the
SSD's blocks will be then on the erased list and will be available for
writing. But if that 10% gets used up and fstrim has not been run
again, then you will get the big erase times happening on every write.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Managing storage groups intelligently [ In reply to ]
On Tue, 1 Sep 2020 17:41:55 -0500
"R. Ramesh" <rramesh@verizon.net> wrote:

> > You have a few options.
> >
> > 1) different storage groups for the different disks
> > - setup recording rules to use this SG
> >
> > 2) both disks in 1 SG and tune the disk selection algorithm
> > - balanced free space, immediately comes to mind.
> > - the backend will prefer to use the disk with the
> > most free space.
> > - this is a user selectable configuration item.
> >
> >
> > Regards
> > Stuart
> >
> Thanks. It appears that putting SSD in "Default group" and moving all
> spinning HDDs to another storage group should do the trick.
>
> Disk selection algorithms are all based on space usage and my SSD
> will fail next to large HD pool that I have as 12TB RAID. So, your
> first method is more suitable, I think. I will copy overnight between
> SSD and HD making sure that I leave more recent recordings in the SSD
> and still leave enough space to record.
>
> Regards
> Ramesh

There is an example script to move files as each recording is finished
here; http://www.mythtv.org/wiki/MoveRecording.sh maybe you could add
an "ionice", but it probably does more than what you want.

--
sig goes here...
bP
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org