Mailing List Archive

On 02/03/2024 17:35, Ian Evans wrote:
> Just drinking coffee and a random mythtv musing struck me: is it
> possible to run mythcommflag on another PC so as not to bog down a PC
> doing multiple recordings.
>
> Say you have your main mythbox with its recording drives, but you
> archive older recordings to a NAS with storage directories. And you
> have a small computer on your network that does odd jobs like DNS,
> Home Assistant, etc. Could you run mythcommflag on just the archived
> recordings on that pc at a relaxed pace? :)
>
> Again, random coffee musing. Maybe the answer is "a X Gen intel/amd
> won't break a sweat mythcommflagging dozens of recordings, just limit
> the time to overnight."

Yes.

Run a slave backend on the other computer. It doesn't need any tuners,
but it does need access to the recording drives. You can configure the
kinds of jobs it is capable of running, how many of them at the same
time etc. If you want commflagging to run only on that other machine
then disallow commflagging on the main backend.

HTH, Jan
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sat, Mar 2, 2024, 12:09?p.m. Jan Ceuleers <jan.ceuleers@gmail.com> wrote:

> On 02/03/2024 17:35, Ian Evans wrote:
> > Just drinking coffee and a random mythtv musing struck me: is it
> > possible to run mythcommflag on another PC so as not to bog down a PC
> > doing multiple recordings.
> >
> > Say you have your main mythbox with its recording drives, but you
> > archive older recordings to a NAS with storage directories. And you
> > have a small computer on your network that does odd jobs like DNS,
> > Home Assistant, etc. Could you run mythcommflag on just the archived
> > recordings on that pc at a relaxed pace? :)
> >
> > Again, random coffee musing. Maybe the answer is "a X Gen intel/amd
> > won't break a sweat mythcommflagging dozens of recordings, just limit
> > the time to overnight."
>
> Yes.
>
> Run a slave backend on the other computer. It doesn't need any tuners,
> but it does need access to the recording drives. You can configure the
> kinds of jobs it is capable of running, how many of them at the same
> time etc. If you want commflagging to run only on that other machine
> then disallow commflagging on the main backend.
>
> Thanks. I see that it can be run with a - -file input. So I'm assuming if
you run it from a script against just the files in the archive storage it
finds the recording in the database that way? Is there also a way a script
can detect if a file has already been Flagged? (Barring that, I guess the
script could always move files from /storagedirunflagged to
/storagedirflagged.



>

On Sat, 2 Mar 2024 13:48:13 -0500, you wrote:

>On Sat, Mar 2, 2024, 12:09?p.m. Jan Ceuleers <jan.ceuleers@gmail.com> wrote:
>
>> On 02/03/2024 17:35, Ian Evans wrote:
>> > Just drinking coffee and a random mythtv musing struck me: is it
>> > possible to run mythcommflag on another PC so as not to bog down a PC
>> > doing multiple recordings.
>> >
>> > Say you have your main mythbox with its recording drives, but you
>> > archive older recordings to a NAS with storage directories. And you
>> > have a small computer on your network that does odd jobs like DNS,
>> > Home Assistant, etc. Could you run mythcommflag on just the archived
>> > recordings on that pc at a relaxed pace? :)
>> >
>> > Again, random coffee musing. Maybe the answer is "a X Gen intel/amd
>> > won't break a sweat mythcommflagging dozens of recordings, just limit
>> > the time to overnight."
>>
>> Yes.
>>
>> Run a slave backend on the other computer. It doesn't need any tuners,
>> but it does need access to the recording drives. You can configure the
>> kinds of jobs it is capable of running, how many of them at the same
>> time etc. If you want commflagging to run only on that other machine
>> then disallow commflagging on the main backend.
>>
>> Thanks. I see that it can be run with a - -file input. So I'm assuming if
>you run it from a script against just the files in the archive storage it
>finds the recording in the database that way? Is there also a way a script
>can detect if a file has already been Flagged? (Barring that, I guess the
>script could always move files from /storagedirunflagged to
>/storagedirflagged.

Mythcommflag puts its data in the recordedmarkup table, so if you
check for the commflag mark values in that table, you should be able
to work out if mythcommflag has been run or not.

https://www.mythtv.org/wiki/Recordedmarkup_table
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 02/03/2024 18:48, Ian Evans wrote:
> On Sat, Mar 2, 2024, 12:09?p.m. Jan Ceuleers <jan.ceuleers@gmail.com> wrote:
>
>> On 02/03/2024 17:35, Ian Evans wrote:
>>> Just drinking coffee and a random mythtv musing struck me: is it
>>> possible to run mythcommflag on another PC so as not to bog down a PC
>>> doing multiple recordings.
>>>
>>> Say you have your main mythbox with its recording drives, but you
>>> archive older recordings to a NAS with storage directories. And you
>>> have a small computer on your network that does odd jobs like DNS,
>>> Home Assistant, etc. Could you run mythcommflag on just the archived
>>> recordings on that pc at a relaxed pace? :)
>>>
>>> Again, random coffee musing. Maybe the answer is "a X Gen intel/amd
>>> won't break a sweat mythcommflagging dozens of recordings, just limit
>>> the time to overnight."
>>
>> Yes.
>>
>> Run a slave backend on the other computer. It doesn't need any tuners,
>> but it does need access to the recording drives. You can configure the
>> kinds of jobs it is capable of running, how many of them at the same
>> time etc. If you want commflagging to run only on that other machine
>> then disallow commflagging on the main backend.
>>
>> Thanks. I see that it can be run with a - -file input. So I'm assuming if
> you run it from a script against just the files in the archive storage it
> finds the recording in the database that way? Is there also a way a script
> can detect if a file has already been Flagged? (Barring that, I guess the
> script could always move files from /storagedirunflagged to
> /storagedirflagged.
>
You wouldn't need to move actual files from one directory to another. All you need do is for your
script to touch a new file somewhere or append a line to a file - that can be easily searched
for/tested and doesn't involve direct database access and also saves large video files being
shuffled around, possibly while myth is trying to record something else!

--

Mike Perkins


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sat, 2 Mar 2024 22:46:04 +0000, you wrote:

>On 02/03/2024 18:48, Ian Evans wrote:
>> On Sat, Mar 2, 2024, 12:09?p.m. Jan Ceuleers <jan.ceuleers@gmail.com> wrote:
>>
>>> On 02/03/2024 17:35, Ian Evans wrote:
>>>> Just drinking coffee and a random mythtv musing struck me: is it
>>>> possible to run mythcommflag on another PC so as not to bog down a PC
>>>> doing multiple recordings.
>>>>
>>>> Say you have your main mythbox with its recording drives, but you
>>>> archive older recordings to a NAS with storage directories. And you
>>>> have a small computer on your network that does odd jobs like DNS,
>>>> Home Assistant, etc. Could you run mythcommflag on just the archived
>>>> recordings on that pc at a relaxed pace? :)
>>>>
>>>> Again, random coffee musing. Maybe the answer is "a X Gen intel/amd
>>>> won't break a sweat mythcommflagging dozens of recordings, just limit
>>>> the time to overnight."
>>>
>>> Yes.
>>>
>>> Run a slave backend on the other computer. It doesn't need any tuners,
>>> but it does need access to the recording drives. You can configure the
>>> kinds of jobs it is capable of running, how many of them at the same
>>> time etc. If you want commflagging to run only on that other machine
>>> then disallow commflagging on the main backend.
>>>
>>> Thanks. I see that it can be run with a - -file input. So I'm assuming if
>> you run it from a script against just the files in the archive storage it
>> finds the recording in the database that way? Is there also a way a script
>> can detect if a file has already been Flagged? (Barring that, I guess the
>> script could always move files from /storagedirunflagged to
>> /storagedirflagged.
>>
>You wouldn't need to move actual files from one directory to another. All you need do is for your
>script to touch a new file somewhere or append a line to a file - that can be easily searched
>for/tested and doesn't involve direct database access and also saves large video files being
>shuffled around, possibly while myth is trying to record something else!

Moving a file from one directory to another on the same partition is
virtually instantaneous - only the filesystem pointer is changed, the
actual file does not get moved. This does depend on the particular
filesystem though - I know this to be true for ext[234] and JFS. I
would be less sure for ZFS though, as I have never used that and it is
more complicated than most filesystems. So having two directories
side by side and moving the files from one to the other is actually a
good solution, as long as both directories are in a storage group. I
believe that such "rename only" moves are also atomic at the
filesystem level, so they should not have any impact on MythTV.

As for mythcommflag performance, for processors 10 years old or
younger and not the super cheap variety, then they can normally do
commflagging on one recording per CPU core/thread in real time. So
for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
simultaneously in real time as 8 recordings happen together. The
trick with real time commflagging is that it gets done in RAM, on the
buffered recording data before it gets written to disk. Once the data
has been written to disk, reading it back again slows down
commflagging, as there is then the problem of contention for the disk,
with the heads having to move between recordings. So you also need
enough RAM, as well as the CPU resources.

I gave up doing commflagging a long time ago as here in New Zealand,
it never gave useful results. So I have not tried commflagging with
the current generations of CPU, which are much faster again than 10
year old ones. They may well be able to do more than one mythcommflag
operation per CPU thread, maybe even three or four.

I would recommend a starting position of one commflag operation per
CPU core/thread being run on the main recording box, starting the
commflagging when the recording starts. Then just let any more
simultaneous recordings that go over that limit have their
commflagging queued to be done later. There should not be any need to
have another box doing commflagging unless you are recording 20 things
at once all day.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sat, Mar 2, 2024 at 7:02?PM Stephen Worthington <stephen_agent@jsw.gen.nz>
wrote:

> As for mythcommflag performance, for processors 10 years old or
> younger and not the super cheap variety, then they can normally do
> commflagging on one recording per CPU core/thread in real time. So
> for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
> simultaneously in real time as 8 recordings happen together. The
> trick with real time commflagging is that it gets done in RAM, on the
> buffered recording data before it gets written to disk. Once the data
> has been written to disk, reading it back again slows down
> commflagging, as there is then the problem of contention for the disk,
> with the heads having to move between recordings. So you also need
> enough RAM, as well as the CPU resources.
>
> I gave up doing commflagging a long time ago as here in New Zealand,
> it never gave useful results. So I have not tried commflagging with
> the current generations of CPU, which are much faster again than 10
> year old ones. They may well be able to do more than one mythcommflag
> operation per CPU thread, maybe even three or four.
>
> I would recommend a starting position of one commflag operation per
> CPU core/thread being run on the main recording box, starting the
> commflagging when the recording starts. Then just let any more
> simultaneous recordings that go over that limit have their
> commflagging queued to be done later. There should not be any need to
> have another box doing commflagging unless you are recording 20 things
> at once all day.


Just a quick background. My wife and I are currently working on another
place and I've just MacGyver'd a temporary setup so we'd have a PVR and I
could tiner with v34.

The backend is currently running on an old Toshiba Satellite laptop. It has
8 gig of RAM and a i7-3720QM CPU.

According to Intel ARK:
- Total Cores 4
- Total Threads 8
- Max Turbo Frequency 3.60 GHz
- Intel® Turbo Boost Technology 2.0 Frequency‡ 3.60 GHz
- Processor Base Frequency 2.60 GHz
- Cache 6 MB Intel® Smart Cache

When I tried running commercial detection a few months ago, I remember the
load spiking to 2.00. Can't remember if I had that running post recording
while there were other recordings. I may have not set it for real-time
detection thinking a laptop couldn't handle it. I'll have to change the
setup when my the SVU marathon is over.

On Sat, Mar 2, 2024 at 11:31?PM Ian Evans <dheianevans@gmail.com> wrote:

>
>
> On Sat, Mar 2, 2024 at 7:02?PM Stephen Worthington <
> stephen_agent@jsw.gen.nz> wrote:
>
>> As for mythcommflag performance, for processors 10 years old or
>> younger and not the super cheap variety, then they can normally do
>> commflagging on one recording per CPU core/thread in real time. So
>> for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
>> simultaneously in real time as 8 recordings happen together. The
>> trick with real time commflagging is that it gets done in RAM, on the
>> buffered recording data before it gets written to disk. Once the data
>> has been written to disk, reading it back again slows down
>> commflagging, as there is then the problem of contention for the disk,
>> with the heads having to move between recordings. So you also need
>> enough RAM, as well as the CPU resources.
>>
>> I gave up doing commflagging a long time ago as here in New Zealand,
>> it never gave useful results. So I have not tried commflagging with
>> the current generations of CPU, which are much faster again than 10
>> year old ones. They may well be able to do more than one mythcommflag
>> operation per CPU thread, maybe even three or four.
>>
>> I would recommend a starting position of one commflag operation per
>> CPU core/thread being run on the main recording box, starting the
>> commflagging when the recording starts. Then just let any more
>> simultaneous recordings that go over that limit have their
>> commflagging queued to be done later. There should not be any need to
>> have another box doing commflagging unless you are recording 20 things
>> at once all day.
>
>
> Just a quick background. My wife and I are currently working on another
> place and I've just MacGyver'd a temporary setup so we'd have a PVR and I
> could tiner with v34.
>
> The backend is currently running on an old Toshiba Satellite laptop. It
> has 8 gig of RAM and a i7-3720QM CPU.
>
> According to Intel ARK:
>
> - Total Cores 4
> - Total Threads 8
> - Max Turbo Frequency 3.60 GHz
> - Intel® Turbo Boost Technology 2.0 Frequency‡ 3.60 GHz
> - Processor Base Frequency 2.60 GHz
> - Cache 6 MB Intel® Smart Cache
>
>
> When I tried running commercial detection a few months ago, I remember the
> load spiking to 2.00. Can't remember if I had that running post recording
> while there were other recordings. I may have not set it for real-time
> detection thinking a laptop couldn't handle it. I'll have to change the
> setup when my the SVU marathon is over.
>

An update. Went into setup and allowed commercial detection to start
immediately. CPU setting was Low.

Created a 15 minute manual recording a few minutes away and allowed
commercial detection for that recording. The recording started.

According to the queue it first looked up the metadata and then ran
mythcommflag. According to top, load spiked to 3.97, and mythcommflag's CPU
% was around 97%. During the whole 15 minutes of the recording it never got
past "looking for logo:

When the recording ended, it finally started processing. Load dropped down
to about 1.75 and CPU% for mythcommflag was around 37%. According to the
job queue it was running at about 97.0251 fps.

Why would the logo detection spike so high?

On Sun, Mar 3, 2024, 12:37?a.m. Ian Evans <dheianevans@gmail.com> wrote:

> On Sat, Mar 2, 2024 at 11:31?PM Ian Evans <dheianevans@gmail.com> wrote:
>
>>
>>
>> On Sat, Mar 2, 2024 at 7:02?PM Stephen Worthington <
>> stephen_agent@jsw.gen.nz> wrote:
>>
>>> As for mythcommflag performance, for processors 10 years old or
>>> younger and not the super cheap variety, then they can normally do
>>> commflagging on one recording per CPU core/thread in real time. So
>>> for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
>>> simultaneously in real time as 8 recordings happen together. The
>>> trick with real time commflagging is that it gets done in RAM, on the
>>> buffered recording data before it gets written to disk. Once the data
>>> has been written to disk, reading it back again slows down
>>> commflagging, as there is then the problem of contention for the disk,
>>> with the heads having to move between recordings. So you also need
>>> enough RAM, as well as the CPU resources.
>>>
>>> I gave up doing commflagging a long time ago as here in New Zealand,
>>> it never gave useful results. So I have not tried commflagging with
>>> the current generations of CPU, which are much faster again than 10
>>> year old ones. They may well be able to do more than one mythcommflag
>>> operation per CPU thread, maybe even three or four.
>>>
>>> I would recommend a starting position of one commflag operation per
>>> CPU core/thread being run on the main recording box, starting the
>>> commflagging when the recording starts. Then just let any more
>>> simultaneous recordings that go over that limit have their
>>> commflagging queued to be done later. There should not be any need to
>>> have another box doing commflagging unless you are recording 20 things
>>> at once all day.
>>
>>
>> Just a quick background. My wife and I are currently working on another
>> place and I've just MacGyver'd a temporary setup so we'd have a PVR and I
>> could tiner with v34.
>>
>> The backend is currently running on an old Toshiba Satellite laptop. It
>> has 8 gig of RAM and a i7-3720QM CPU.
>>
>> According to Intel ARK:
>>
>> - Total Cores 4
>> - Total Threads 8
>> - Max Turbo Frequency 3.60 GHz
>> - Intel® Turbo Boost Technology 2.0 Frequency‡ 3.60 GHz
>> - Processor Base Frequency 2.60 GHz
>> - Cache 6 MB Intel® Smart Cache
>>
>>
>> When I tried running commercial detection a few months ago, I remember
>> the load spiking to 2.00. Can't remember if I had that running post
>> recording while there were other recordings. I may have not set it for
>> real-time detection thinking a laptop couldn't handle it. I'll have to
>> change the setup when my the SVU marathon is over.
>>
>
> An update. Went into setup and allowed commercial detection to start
> immediately. CPU setting was Low.
>
> Created a 15 minute manual recording a few minutes away and allowed
> commercial detection for that recording. The recording started.
>
> According to the queue it first looked up the metadata and then ran
> mythcommflag. According to top, load spiked to 3.97, and mythcommflag's CPU
> % was around 97%. During the whole 15 minutes of the recording it never got
> past "looking for logo:
>
> When the recording ended, it finally started processing. Load dropped down
> to about 1.75 and CPU% for mythcommflag was around 37%. According to the
> job queue it was running at about 97.0251 fps.
>
> Why would the logo detection spike so high?
>
>
What would you guys say is the minimum Intel/AMD generation for real-time
commercial flagging, because this i7-3720QM isn't cutting it.

Thanks.

>
>

On Wed, Mar 6, 2024 at 1:39?PM Ian Evans <dheianevans@gmail.com> wrote:

> On Sun, Mar 3, 2024, 12:37?a.m. Ian Evans <dheianevans@gmail.com> wrote:
>
>> On Sat, Mar 2, 2024 at 11:31?PM Ian Evans <dheianevans@gmail.com> wrote:
>>
>>>
>>>
>>> On Sat, Mar 2, 2024 at 7:02?PM Stephen Worthington <
>>> stephen_agent@jsw.gen.nz> wrote:
>>>
>>>> As for mythcommflag performance, for processors 10 years old or
>>>> younger and not the super cheap variety, then they can normally do
>>>> commflagging on one recording per CPU core/thread in real time. So
>>>> for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
>>>> simultaneously in real time as 8 recordings happen together. The
>>>> trick with real time commflagging is that it gets done in RAM, on the
>>>> buffered recording data before it gets written to disk. Once the data
>>>> has been written to disk, reading it back again slows down
>>>> commflagging, as there is then the problem of contention for the disk,
>>>> with the heads having to move between recordings. So you also need
>>>> enough RAM, as well as the CPU resources.
>>>>
>>>> I gave up doing commflagging a long time ago as here in New Zealand,
>>>> it never gave useful results. So I have not tried commflagging with
>>>> the current generations of CPU, which are much faster again than 10
>>>> year old ones. They may well be able to do more than one mythcommflag
>>>> operation per CPU thread, maybe even three or four.
>>>>
>>>> I would recommend a starting position of one commflag operation per
>>>> CPU core/thread being run on the main recording box, starting the
>>>> commflagging when the recording starts. Then just let any more
>>>> simultaneous recordings that go over that limit have their
>>>> commflagging queued to be done later. There should not be any need to
>>>> have another box doing commflagging unless you are recording 20 things
>>>> at once all day.
>>>
>>>
>>> Just a quick background. My wife and I are currently working on another
>>> place and I've just MacGyver'd a temporary setup so we'd have a PVR and I
>>> could tiner with v34.
>>>
>>> The backend is currently running on an old Toshiba Satellite laptop. It
>>> has 8 gig of RAM and a i7-3720QM CPU.
>>>
>>> According to Intel ARK:
>>>
>>> - Total Cores 4
>>> - Total Threads 8
>>> - Max Turbo Frequency 3.60 GHz
>>> - Intel® Turbo Boost Technology 2.0 Frequency‡ 3.60 GHz
>>> - Processor Base Frequency 2.60 GHz
>>> - Cache 6 MB Intel® Smart Cache
>>>
>>>
>>> When I tried running commercial detection a few months ago, I remember
>>> the load spiking to 2.00. Can't remember if I had that running post
>>> recording while there were other recordings. I may have not set it for
>>> real-time detection thinking a laptop couldn't handle it. I'll have to
>>> change the setup when my the SVU marathon is over.
>>>
>>
>> An update. Went into setup and allowed commercial detection to start
>> immediately. CPU setting was Low.
>>
>> Created a 15 minute manual recording a few minutes away and allowed
>> commercial detection for that recording. The recording started.
>>
>> According to the queue it first looked up the metadata and then ran
>> mythcommflag. According to top, load spiked to 3.97, and mythcommflag's CPU
>> % was around 97%. During the whole 15 minutes of the recording it never got
>> past "looking for logo:
>>
>> When the recording ended, it finally started processing. Load dropped
>> down to about 1.75 and CPU% for mythcommflag was around 37%. According to
>> the job queue it was running at about 97.0251 fps.
>>
>> Why would the logo detection spike so high?
>>
>>
> What would you guys say is the minimum Intel/AMD generation for real-time
> commercial flagging, because this i7-3720QM isn't cutting it.
>
> Thanks.
>

I am stabbing in the dark here because I haven not used MythTV in a few
years, but real-time commflagging is very efficient (it comm flags as it
records). Your request to have commflagging on a separate machine (slave
backend) basically (I think) disables that capability because it relies on
the files to be written to disk before the slave actually has access to
it. This would greatly increase IO pressure too.

Did you every try the immediate commercial detection setting without the
slave? I had a very old i7 (older than yours) that could comm detect 4
ATSC recordings @ once without any load at all (well, not much).

When you measure the load, is it in IOWait, or what?

On Wed, Mar 6, 2024, 4:16?p.m. Greg Oliver <oliver.greg@gmail.com> wrote:

> On Wed, Mar 6, 2024 at 1:39?PM Ian Evans <dheianevans@gmail.com> wrote:
>
>> On Sun, Mar 3, 2024, 12:37?a.m. Ian Evans <dheianevans@gmail.com> wrote:
>>
>>> On Sat, Mar 2, 2024 at 11:31?PM Ian Evans <dheianevans@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Sat, Mar 2, 2024 at 7:02?PM Stephen Worthington <
>>>> stephen_agent@jsw.gen.nz> wrote:
>>>>
>>>>> As for mythcommflag performance, for processors 10 years old or
>>>>> younger and not the super cheap variety, then they can normally do
>>>>> commflagging on one recording per CPU core/thread in real time. So
>>>>> for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
>>>>> simultaneously in real time as 8 recordings happen together. The
>>>>> trick with real time commflagging is that it gets done in RAM, on the
>>>>> buffered recording data before it gets written to disk. Once the data
>>>>> has been written to disk, reading it back again slows down
>>>>> commflagging, as there is then the problem of contention for the disk,
>>>>> with the heads having to move between recordings. So you also need
>>>>> enough RAM, as well as the CPU resources.
>>>>>
>>>>> I gave up doing commflagging a long time ago as here in New Zealand,
>>>>> it never gave useful results. So I have not tried commflagging with
>>>>> the current generations of CPU, which are much faster again than 10
>>>>> year old ones. They may well be able to do more than one mythcommflag
>>>>> operation per CPU thread, maybe even three or four.
>>>>>
>>>>> I would recommend a starting position of one commflag operation per
>>>>> CPU core/thread being run on the main recording box, starting the
>>>>> commflagging when the recording starts. Then just let any more
>>>>> simultaneous recordings that go over that limit have their
>>>>> commflagging queued to be done later. There should not be any need to
>>>>> have another box doing commflagging unless you are recording 20 things
>>>>> at once all day.
>>>>
>>>>
>>>> Just a quick background. My wife and I are currently working on another
>>>> place and I've just MacGyver'd a temporary setup so we'd have a PVR and I
>>>> could tiner with v34.
>>>>
>>>> The backend is currently running on an old Toshiba Satellite laptop. It
>>>> has 8 gig of RAM and a i7-3720QM CPU.
>>>>
>>>> According to Intel ARK:
>>>>
>>>> - Total Cores 4
>>>> - Total Threads 8
>>>> - Max Turbo Frequency 3.60 GHz
>>>> - Intel® Turbo Boost Technology 2.0 Frequency‡ 3.60 GHz
>>>> - Processor Base Frequency 2.60 GHz
>>>> - Cache 6 MB Intel® Smart Cache
>>>>
>>>>
>>>> When I tried running commercial detection a few months ago, I remember
>>>> the load spiking to 2.00. Can't remember if I had that running post
>>>> recording while there were other recordings. I may have not set it for
>>>> real-time detection thinking a laptop couldn't handle it. I'll have to
>>>> change the setup when my the SVU marathon is over.
>>>>
>>>
>>> An update. Went into setup and allowed commercial detection to start
>>> immediately. CPU setting was Low.
>>>
>>> Created a 15 minute manual recording a few minutes away and allowed
>>> commercial detection for that recording. The recording started.
>>>
>>> According to the queue it first looked up the metadata and then ran
>>> mythcommflag. According to top, load spiked to 3.97, and mythcommflag's CPU
>>> % was around 97%. During the whole 15 minutes of the recording it never got
>>> past "looking for logo:
>>>
>>> When the recording ended, it finally started processing. Load dropped
>>> down to about 1.75 and CPU% for mythcommflag was around 37%. According to
>>> the job queue it was running at about 97.0251 fps.
>>>
>>> Why would the logo detection spike so high?
>>>
>>>
>> What would you guys say is the minimum Intel/AMD generation for real-time
>> commercial flagging, because this i7-3720QM isn't cutting it.
>>
>> Thanks.
>>
>
> I am stabbing in the dark here because I haven not used MythTV in a few
> years, but real-time commflagging is very efficient (it comm flags as it
> records). Your request to have commflagging on a separate machine (slave
> backend) basically (I think) disables that capability because it relies on
> the files to be written to disk before the slave actually has access to
> it. This would greatly increase IO pressure too.
>
> Did you every try the immediate commercial detection setting without the
> slave? I had a very old i7 (older than yours) that could comm detect 4
> ATSC recordings @ once without any load at all (well, not much).
>
> When you measure the load, is it in IOWait, or what?
>

Sorry... never actually got to using a slave. My temporary setup on
the i7-3720QM wasn't cutting it, so I was just tossing out an idea. The
spiked load was just the Load reading in top. The laptop was still pretty
responsive, but the fan was just howling like a banshee.

Of course, a laptop isn't the perfect airflow situation, so maybe I should
hold off on commflagging until Im using a better setup. : -)

I am curious why the real-time test took 15 high load minutes on "looking
for logo" before settling down and flagging at 90+ fps.

>
>

On Wed, 6 Mar 2024 16:31:49 -0500, you wrote:

>On Wed, Mar 6, 2024, 4:16?p.m. Greg Oliver <oliver.greg@gmail.com> wrote:
>
>> On Wed, Mar 6, 2024 at 1:39?PM Ian Evans <dheianevans@gmail.com> wrote:
>>
>>> On Sun, Mar 3, 2024, 12:37?a.m. Ian Evans <dheianevans@gmail.com> wrote:
>>>
>>>> On Sat, Mar 2, 2024 at 11:31?PM Ian Evans <dheianevans@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sat, Mar 2, 2024 at 7:02?PM Stephen Worthington <
>>>>> stephen_agent@jsw.gen.nz> wrote:
>>>>>
>>>>>> As for mythcommflag performance, for processors 10 years old or
>>>>>> younger and not the super cheap variety, then they can normally do
>>>>>> commflagging on one recording per CPU core/thread in real time. So
>>>>>> for a 4 core dual threaded CPU, you could do 8 mythcommflag processes
>>>>>> simultaneously in real time as 8 recordings happen together. The
>>>>>> trick with real time commflagging is that it gets done in RAM, on the
>>>>>> buffered recording data before it gets written to disk. Once the data
>>>>>> has been written to disk, reading it back again slows down
>>>>>> commflagging, as there is then the problem of contention for the disk,
>>>>>> with the heads having to move between recordings. So you also need
>>>>>> enough RAM, as well as the CPU resources.
>>>>>>
>>>>>> I gave up doing commflagging a long time ago as here in New Zealand,
>>>>>> it never gave useful results. So I have not tried commflagging with
>>>>>> the current generations of CPU, which are much faster again than 10
>>>>>> year old ones. They may well be able to do more than one mythcommflag
>>>>>> operation per CPU thread, maybe even three or four.
>>>>>>
>>>>>> I would recommend a starting position of one commflag operation per
>>>>>> CPU core/thread being run on the main recording box, starting the
>>>>>> commflagging when the recording starts. Then just let any more
>>>>>> simultaneous recordings that go over that limit have their
>>>>>> commflagging queued to be done later. There should not be any need to
>>>>>> have another box doing commflagging unless you are recording 20 things
>>>>>> at once all day.
>>>>>
>>>>>
>>>>> Just a quick background. My wife and I are currently working on another
>>>>> place and I've just MacGyver'd a temporary setup so we'd have a PVR and I
>>>>> could tiner with v34.
>>>>>
>>>>> The backend is currently running on an old Toshiba Satellite laptop. It
>>>>> has 8 gig of RAM and a i7-3720QM CPU.
>>>>>
>>>>> According to Intel ARK:
>>>>>
>>>>> - Total Cores 4
>>>>> - Total Threads 8
>>>>> - Max Turbo Frequency 3.60 GHz
>>>>> - Intel? Turbo Boost Technology 2.0 Frequency? 3.60 GHz
>>>>> - Processor Base Frequency 2.60 GHz
>>>>> - Cache 6 MB Intel? Smart Cache
>>>>>
>>>>>
>>>>> When I tried running commercial detection a few months ago, I remember
>>>>> the load spiking to 2.00. Can't remember if I had that running post
>>>>> recording while there were other recordings. I may have not set it for
>>>>> real-time detection thinking a laptop couldn't handle it. I'll have to
>>>>> change the setup when my the SVU marathon is over.
>>>>>
>>>>
>>>> An update. Went into setup and allowed commercial detection to start
>>>> immediately. CPU setting was Low.
>>>>
>>>> Created a 15 minute manual recording a few minutes away and allowed
>>>> commercial detection for that recording. The recording started.
>>>>
>>>> According to the queue it first looked up the metadata and then ran
>>>> mythcommflag. According to top, load spiked to 3.97, and mythcommflag's CPU
>>>> % was around 97%. During the whole 15 minutes of the recording it never got
>>>> past "looking for logo:
>>>>
>>>> When the recording ended, it finally started processing. Load dropped
>>>> down to about 1.75 and CPU% for mythcommflag was around 37%. According to
>>>> the job queue it was running at about 97.0251 fps.
>>>>
>>>> Why would the logo detection spike so high?
>>>>
>>>>
>>> What would you guys say is the minimum Intel/AMD generation for real-time
>>> commercial flagging, because this i7-3720QM isn't cutting it.
>>>
>>> Thanks.
>>>
>>
>> I am stabbing in the dark here because I haven not used MythTV in a few
>> years, but real-time commflagging is very efficient (it comm flags as it
>> records). Your request to have commflagging on a separate machine (slave
>> backend) basically (I think) disables that capability because it relies on
>> the files to be written to disk before the slave actually has access to
>> it. This would greatly increase IO pressure too.
>>
>> Did you every try the immediate commercial detection setting without the
>> slave? I had a very old i7 (older than yours) that could comm detect 4
>> ATSC recordings @ once without any load at all (well, not much).
>>
>> When you measure the load, is it in IOWait, or what?
>>
>
>Sorry... never actually got to using a slave. My temporary setup on
>the i7-3720QM wasn't cutting it, so I was just tossing out an idea. The
>spiked load was just the Load reading in top. The laptop was still pretty
>responsive, but the fan was just howling like a banshee.
>
>Of course, a laptop isn't the perfect airflow situation, so maybe I should
>hold off on commflagging until Im using a better setup. : -)
>
>I am curious why the real-time test took 15 high load minutes on "looking
>for logo" before settling down and flagging at 90+ fps.

The only problem there seems to be the "looking for logo" bit. I do
not remember that way back when I last used commercial flagging, so
there may be some new method that is being used that is causing your
problem. This may be working as designed, but it could also be a bug.
The rest of the flagging is going faster than real time, which is what
is expected. Does the logo problem happen only on specific channels,
or is it common to all channels?
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Wed, Mar 6, 2024, 6:10?p.m. Stephen Worthington <stephen_agent@jsw.gen.nz>
wrote:

> On Wed, 6 Mar 2024 16:31:49 -0500, you wrote:
>
> [snip]
> >Sorry... never actually got to using a slave. My temporary setup on
> >the i7-3720QM wasn't cutting it, so I was just tossing out an idea. The
> >spiked load was just the Load reading in top. The laptop was still pretty
> >responsive, but the fan was just howling like a banshee.
> >
> >Of course, a laptop isn't the perfect airflow situation, so maybe I should
> >hold off on commflagging until Im using a better setup. : -)
> >
> >I am curious why the real-time test took 15 high load minutes on "looking
> >for logo" before settling down and flagging at 90+ fps.
>
> The only problem there seems to be the "looking for logo" bit. I do
> not remember that way back when I last used commercial flagging, so
> there may be some new method that is being used that is causing your
> problem. This may be working as designed, but it could also be a bug.
> The rest of the flagging is going faster than real time, which is what
> is expected. Does the logo problem happen only on specific channels,
> or is it common to all channels?
>

I'll have to do some more tests on different channels and will report back.

>
>

On Wed, Mar 6, 2024, 6:47?p.m. Ian Evans <dheianevans@gmail.com> wrote:

> On Wed, Mar 6, 2024, 6:10?p.m. Stephen Worthington <
> stephen_agent@jsw.gen.nz> wrote:
>
>> On Wed, 6 Mar 2024 16:31:49 -0500, you wrote:
>>
>> [snip]
>> >Sorry... never actually got to using a slave. My temporary setup on
>> >the i7-3720QM wasn't cutting it, so I was just tossing out an idea. The
>> >spiked load was just the Load reading in top. The laptop was still pretty
>> >responsive, but the fan was just howling like a banshee.
>> >
>> >Of course, a laptop isn't the perfect airflow situation, so maybe I
>> should
>> >hold off on commflagging until Im using a better setup. : -)
>> >
>> >I am curious why the real-time test took 15 high load minutes on "looking
>> >for logo" before settling down and flagging at 90+ fps.
>>
>> The only problem there seems to be the "looking for logo" bit. I do
>> not remember that way back when I last used commercial flagging, so
>> there may be some new method that is being used that is causing your
>> problem. This may be working as designed, but it could also be a bug.
>> The rest of the flagging is going faster than real time, which is what
>> is expected. Does the logo problem happen only on specific channels,
>> or is it common to all channels?
>>
>
> I'll have to do some more tests on different channels and will report
> back.
>

Set up two recordings, staggered by about 10 minutes.

Both videos spent about 9-10 minutes "building logo detection buffer" with
high CPU% in top. After that, about another 5-10 minutes "looking for
logo." When that finished, CPU% dropped to about 45%. Load started about
3.95 and settled down to about 1.25. It was flagging between 96 to 115 fps.

One recording was ATSC 1, 1080, the other a 480 subchannel.

Has anyone else noticed that many American channels are no longer
superimposing network bugs on the lower right. Nothing. When did that stop?

Also, I'm in mythtv 34. Should I be enabling the "experimental" commercial
detection features?

On Wed, Mar 6, 2024 at 8:30?PM Ian Evans <dheianevans@gmail.com> wrote:

> On Wed, Mar 6, 2024, 6:47?p.m. Ian Evans <dheianevans@gmail.com> wrote:
>
>> On Wed, Mar 6, 2024, 6:10?p.m. Stephen Worthington <
>> stephen_agent@jsw.gen.nz> wrote:
>>
>>> On Wed, 6 Mar 2024 16:31:49 -0500, you wrote:
>>>
>>> [snip]
>>> >Sorry... never actually got to using a slave. My temporary setup on
>>> >the i7-3720QM wasn't cutting it, so I was just tossing out an idea. The
>>> >spiked load was just the Load reading in top. The laptop was still
>>> pretty
>>> >responsive, but the fan was just howling like a banshee.
>>> >
>>> >Of course, a laptop isn't the perfect airflow situation, so maybe I
>>> should
>>> >hold off on commflagging until Im using a better setup. : -)
>>> >
>>> >I am curious why the real-time test took 15 high load minutes on
>>> "looking
>>> >for logo" before settling down and flagging at 90+ fps.
>>>
>>> The only problem there seems to be the "looking for logo" bit. I do
>>> not remember that way back when I last used commercial flagging, so
>>> there may be some new method that is being used that is causing your
>>> problem. This may be working as designed, but it could also be a bug.
>>> The rest of the flagging is going faster than real time, which is what
>>> is expected. Does the logo problem happen only on specific channels,
>>> or is it common to all channels?
>>>
>>
>> I'll have to do some more tests on different channels and will report
>> back.
>>
>
> Set up two recordings, staggered by about 10 minutes.
>
> Both videos spent about 9-10 minutes "building logo detection buffer" with
> high CPU% in top. After that, about another 5-10 minutes "looking for
> logo." When that finished, CPU% dropped to about 45%. Load started about
> 3.95 and settled down to about 1.25. It was flagging between 96 to 115 fps.
>
> One recording was ATSC 1, 1080, the other a 480 subchannel.
>
> Has anyone else noticed that many American channels are no longer
> superimposing network bugs on the lower right. Nothing. When did that stop?
>
> Also, I'm in mythtv 34. Should I be enabling the "experimental" commercial
> detection features?
>

IIRC, v31 was the last version I used, but for several years prior and up
to, I always had "experimental" enabled.