Mailing List Archive

On Fri, Aug 23, 2019 at 05:31:27PM +0200, Klaas de Waal wrote:
> In a system that uses EIT for the guide data, if you schedule a single
> recording and the start time does later on change even as little as
> one minute then the entry will be shown in the "Upcoming Recordings"
> as "Not listed" and it will not be recorded.

This is why single recording rules are discouraged, especialy for
users with unreliable or changing guide data.

> This is something that I would like to fix.
>
> I am familiar with the EIT guide update code in programdata.cpp; I
> have the chanid, the original start date/time and the new start
> date/time of the program to be updated. I am not familiar with the
> scheduler code.

The current schedule can be retrieved by calling
Scheduler::GetAllPending().

> As I understand it the chanid and the start date/time does uniquely
> identify a recording and thus also a scheduled recording.

The manualid needs to be considered too.

> What needs to be done is then:
> - check if a program to be updated is to be recorded (based on chanid
> and start date/time)
> - if it is, update the start date/time.
>
> I would appreciate comments about this and if possible hints on where
> and how this could be implemented in an efficient manner.

If you do anything for EIT, you should likely handle it for
mythfilldatabase cases too.

David
--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 23/08/2019 17:07, David Engel wrote:
> On Fri, Aug 23, 2019 at 05:31:27PM +0200, Klaas de Waal wrote:
>> In a system that uses EIT for the guide data, if you schedule a single
>> recording and the start time does later on change even as little as
>> one minute then the entry will be shown in the "Upcoming Recordings"
>> as "Not listed" and it will not be recorded.
>
> This is why single recording rules are discouraged, especialy for
> users with unreliable or changing guide data.
>
>> This is something that I would like to fix.
>>
>> I am familiar with the EIT guide update code in programdata.cpp; I
>> have the chanid, the original start date/time and the new start
>> date/time of the program to be updated. I am not familiar with the
>> scheduler code.
>
> The current schedule can be retrieved by calling
> Scheduler::GetAllPending().
>
>> As I understand it the chanid and the start date/time does uniquely
>> identify a recording and thus also a scheduled recording.
>
> The manualid needs to be considered too.
>
>> What needs to be done is then:
>> - check if a program to be updated is to be recorded (based on chanid
>> and start date/time)
>> - if it is, update the start date/time.
>>
>> I would appreciate comments about this and if possible hints on where
>> and how this could be implemented in an efficient manner.
>
> If you do anything for EIT, you should likely handle it for
> mythfilldatabase cases too.
>
> David
>
I have just seen an example of this - not, of course, for the first
time. What usually works, in "Upcoming recordings" is to hit "o" to
show where it is now and then "r" to record that and "d" to cancel the
original. But I don't know how that would be coded.

John P
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

As discussed, the problem of missed single recordings is caused by the starttime
of programs being changed by the guide updates from the EIT.

I am now testing a fix that does the following.
In addition to updating the starttime in database table program (and in the
associated tables credits, programrating and programgenres) the starttime
is now also updated in table record when the program is present there.

However, there may be a better solution possible.
The underlying problem, as I understand it, is that the programs, i.e.
the records in table program, can only be identified by the fields
chanid plus starttime. This means there is no way to identify a
program when the starttime of the program can have been changed.

It is possible to give tables a separate field to be used as the
primary key, independent of the content of the rest of the record.
This is done in table record which has field recordid as primary key.
Same for table capturecard with primary key cardid.

If the table program would also have a independent primary key then it
would be possible to refer to programs (records in table program) using
that key instead of using the starttime. This then makes it possible to
have a reference to a program in which all fields including the starttime
can be changed.

The advantage of the fix currently being tested is that it does not require
a change in the database schema and that it comprises only a few lines of code.
Nevertheless, when and if there is a database schema change forthcoming it might
be a good moment to add a field programidx to table program and to all
other tables that contain references to table program so that the code
can be cleaned up.

Or maybe there is a very good reason why not to use a separate primary
key but use startttime pluls chanid instead. Performance?

Comments are very much appreciated. In the meantime I keep on testing the fix!

Cheers,
Klaas.

N.B. There is a field programid in table program but that is used
for other purposes so that fieldname cannot be used for the primary index.

N.B. The code in programdata.cpp that does the updates is used by
mythfilldatabase as well as far as I know so anything that works with the
guide updates from EIT should work for mythfilldatabase as well.

N.B. There is no field manualid in table record so I assume the manual records
are dealt with in another place.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Thu, 29 Aug 2019 at 21:52, Klaas de Waal <klaas.de.waal@gmail.com> wrote:
> N.B. The code in programdata.cpp that does the updates is used by
> mythfilldatabase as well as far as I know so anything that works with the
> guide updates from EIT should work for mythfilldatabase as well.
>
Update: this statement is NOT correct.
The guide updates done from mythfilldatabase are, although in the same
file programdata.cpp, largely independent from the guide updates done
from EIT events.

Klaas.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Thu, Aug 29, 2019 at 09:52:17PM +0200, Klaas de Waal wrote:
> As discussed, the problem of missed single recordings is caused by the starttime
> of programs being changed by the guide updates from the EIT.
>
> I am now testing a fix that does the following.
> In addition to updating the starttime in database table program (and in the
> associated tables credits, programrating and programgenres) the starttime
> is now also updated in table record when the program is present there.
>
> However, there may be a better solution possible.
> The underlying problem, as I understand it, is that the programs, i.e.
> the records in table program, can only be identified by the fields
> chanid plus starttime. This means there is no way to identify a
> program when the starttime of the program can have been changed.

I think you're heading down a rat hole where you're ultimately going
to be trading one problem for another.

It's reasonable to assume that the program with the same title that is
now 1 minute later than before is the same program. But what if it's
5 minutes later? Still probably the same. But what if it's 15, 30 or
60 minutes later? Maybe EIT provides information to reliably do that.
I don't know since I'm not familiar with it.

As I think your were implying in your followup message,
mythfilldatabase updates the program table differently. I don't know
that it has the ability to reliably determine that a program slipped
by even 1 minute.

You're also assuming that because the program and related tables are
indexed by chanid and starttime that the scheduler works similarly.
It doesn't. It actually uses starttime and callsign. What you
propose would work if there's only one channel on one videsource to
involved. If there are multiple, matching channels on one or more
videosources, it won't work.

> It is possible to give tables a separate field to be used as the
> primary key, independent of the content of the rest of the record.
> This is done in table record which has field recordid as primary key.
> Same for table capturecard with primary key cardid.
>
> If the table program would also have a independent primary key then it
> would be possible to refer to programs (records in table program) using
> that key instead of using the starttime. This then makes it possible to
> have a reference to a program in which all fields including the starttime
> can be changed.
>
> The advantage of the fix currently being tested is that it does not require
> a change in the database schema and that it comprises only a few lines of code.
> Nevertheless, when and if there is a database schema change forthcoming it might
> be a good moment to add a field programidx to table program and to all
> other tables that contain references to table program so that the code
> can be cleaned up.

I think you might be better off using find-one rules with the
this-episode filter enabled. They are easy to create initially (at
least using mythfrontend) but might need some TLC to better fit the
problems you are trying to solve.

> Or maybe there is a very good reason why not to use a separate primary
> key but use startttime pluls chanid instead. Performance?
>
> Comments are very much appreciated. In the meantime I keep on testing the fix!
>
> Cheers,
> Klaas.
>
> N.B. There is a field programid in table program but that is used
> for other purposes so that fieldname cannot be used for the primary index.
>
> N.B. The code in programdata.cpp that does the updates is used by
> mythfilldatabase as well as far as I know so anything that works with the
> guide updates from EIT should work for mythfilldatabase as well.
>
> N.B. There is no field manualid in table record so I assume the manual records
> are dealt with in another place.

For manual recording rules, record.search is set to "manual" search
and record.recordid is used to match program.manualid.

David
--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Thu, Aug 29, 2019 at 09:52:17PM +0200, Klaas de Waal wrote:
> As discussed, the problem of missed single recordings is caused by the starttime
> of programs being changed by the guide updates from the EIT.

Klaas, I think I have a better idea for you. In
Scheduler::AddNotListed(), the scheduler identifies single and
override rules that no longer match program in the guide. It should
probably include don't record rules too but that's a separate issue.

I suggest you hook into AddNotListed() and if a not listed rule is
found, then attempt to see if the program has moved slightly in time.
If a program has moved, update the recording rule according and
initiate a reschedule.

Ideally this should only be done the first time a not listed rule is
found. It would be wasteful to keep looking for a slightly moved
program when one doesn't exist. I'm not sure how easy that would be
to do but it should be the goal.

David

> I am now testing a fix that does the following.
> In addition to updating the starttime in database table program (and in the
> associated tables credits, programrating and programgenres) the starttime
> is now also updated in table record when the program is present there.
>
> However, there may be a better solution possible.
> The underlying problem, as I understand it, is that the programs, i.e.
> the records in table program, can only be identified by the fields
> chanid plus starttime. This means there is no way to identify a
> program when the starttime of the program can have been changed.
>
> It is possible to give tables a separate field to be used as the
> primary key, independent of the content of the rest of the record.
> This is done in table record which has field recordid as primary key.
> Same for table capturecard with primary key cardid.
>
> If the table program would also have a independent primary key then it
> would be possible to refer to programs (records in table program) using
> that key instead of using the starttime. This then makes it possible to
> have a reference to a program in which all fields including the starttime
> can be changed.
>
> The advantage of the fix currently being tested is that it does not require
> a change in the database schema and that it comprises only a few lines of code.
> Nevertheless, when and if there is a database schema change forthcoming it might
> be a good moment to add a field programidx to table program and to all
> other tables that contain references to table program so that the code
> can be cleaned up.
>
> Or maybe there is a very good reason why not to use a separate primary
> key but use startttime pluls chanid instead. Performance?
>
> Comments are very much appreciated. In the meantime I keep on testing the fix!
>
> Cheers,
> Klaas.
>
> N.B. There is a field programid in table program but that is used
> for other purposes so that fieldname cannot be used for the primary index.
>
> N.B. The code in programdata.cpp that does the updates is used by
> mythfilldatabase as well as far as I know so anything that works with the
> guide updates from EIT should work for mythfilldatabase as well.
>
> N.B. There is no field manualid in table record so I assume the manual records
> are dealt with in another place.
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org

--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

About:
> It's reasonable to assume that the program with the same title that is
> now 1 minute later than before is the same program. But what if it's
> 5 minutes later? Still probably the same. But what if it's 15, 30 or
> 60 minutes later? Maybe EIT provides information to reliably do that.

Yes, this is exactly what the program update code for EIT events does.
There is large amount of code that does this, by comparing titles, descriptions,
time overlap etc, it then computes a matching score to decide if this
is a new program or an update of an existing program.

About
> I suggest you hook into AddNotListed() and if a not listed rule is
> found, then attempt to see if the program has moved slightly in time.
> If a program has moved, update the recording rule according and
> initiate a reschedule.
I have been thinking about something along those lines but the problem
is then again to identify the moved program. Not only the start time may
have been changed but also the title and the description.
This kind of matching has already been done by the EIT update code
and it does not make sense to duplicate it.

About
> Ideally this should only be done the first time a not listed rule is
> found. It would be wasteful to keep looking for a slightly moved
> program when one doesn't exist. I'm not sure how easy that would be
> to do but it should be the goal.
This is also already solved by the EIT update code because the EIT update
code does this exactly once when the event is received.

About manualid
Manually created scheduled recordings will not be changed.

About mythfilldatabase.
The XMLTV update code called by mythfilldatabase is different, and much simpler,
than the EIT update code. In order to improve the "Single Record" behavior for
mythfilldatabase as well the XMLTV update code might be merged with
the EIT update code.
It might be a bit slower to pass all program updates through the EIT
update code due
to the additional processing but that should not be a problem.
However, with mythfilldatabase the problem of "Not Listed" programs
is, I think, much smaller.
When mythfilldatabase runs e.g. once a day early in the morning, then all the
"Single Record" type of recordings scheduled during that day and scheduled
for that day will stay in the schedule.
For recordings scheduled days ahead a check once a day is sufficient.
This is different for EIT where program guide updates come at any moment.

About the root cause
I would like to point out again that this bug is made possible by what
is to me a design flaw.
An entry in table program is identified by the chanid plus the starttime.
The starttime is part of the the data of the program (program as in TV
program....)
so if the starttime changes that entry in table program cannot be found anymore.
The solution for this, as implemented in the EIT update code in programdata.cpp,
is to modify the starttime everywhere there is a reference (chanid
plus starttime)
to an entry in table program when the starttime in that entry is changed.
This is done for a number of tables but not for table record.
The patch I wrote does this also for table record.
A better solution would be to add a unique key to table program and use that
in all references but that is rather more work and even worse, does
require a database schema change.

Next steps
I will create a ticket for this with a summary of this email thread so
it is recorded
for eternity (or as long as the trac server will live....) together
with the patch.
I intended to commit it to master a few days after that.
N.B. I have been doing lots of testing on this now, including test with two
video sources with overlapping channels as suggested, and for me it works OK.

Thanks for the feedback & Cheers!
Klaas.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Mon, Sep 02, 2019 at 09:11:46PM +0200, Klaas de Waal wrote:
> About:
> > It's reasonable to assume that the program with the same title that is
> > now 1 minute later than before is the same program. But what if it's
> > 5 minutes later? Still probably the same. But what if it's 15, 30 or
> > 60 minutes later? Maybe EIT provides information to reliably do that.
>
> Yes, this is exactly what the program update code for EIT events does.
> There is large amount of code that does this, by comparing titles, descriptions,
> time overlap etc, it then computes a matching score to decide if this
> is a new program or an update of an existing program.
>
> About
> > I suggest you hook into AddNotListed() and if a not listed rule is
> > found, then attempt to see if the program has moved slightly in time.
> > If a program has moved, update the recording rule according and
> > initiate a reschedule.
> I have been thinking about something along those lines but the problem
> is then again to identify the moved program. Not only the start time may
> have been changed but also the title and the description.
> This kind of matching has already been done by the EIT update code
> and it does not make sense to duplicate it.
>
> About
> > Ideally this should only be done the first time a not listed rule is
> > found. It would be wasteful to keep looking for a slightly moved
> > program when one doesn't exist. I'm not sure how easy that would be
> > to do but it should be the goal.
> This is also already solved by the EIT update code because the EIT update
> code does this exactly once when the event is received.

Doing this all the way the EIT code handles it seems grossly
inefficient to me. In the vast majority of cases, we don't care that
some random programs that we aren't going to record anyway shift by a
few minutes here and there. Simply and blindly replace the old,
program entries with the new ones and carry on.

It is only the relatively few cases (in my experience) where we setup
a program to record at a specific time and it gets shifted that
matter. We already detect those cases efficiently In AddNewRecords().
When and only when those cases occur do we need to do any extra
processing to see if the program got moved by a few minutes or
replaced by something completely different.

> About manualid
> Manually created scheduled recordings will not be changed.
>
> About mythfilldatabase.
> The XMLTV update code called by mythfilldatabase is different, and much simpler,
> than the EIT update code.

Exactly, handle the 99% of cases as simply as possible.

> In order to improve the "Single Record" behavior for
> mythfilldatabase as well the XMLTV update code might be merged with
> the EIT update code.
> It might be a bit slower to pass all program updates through the EIT
> update code due
> to the additional processing but that should not be a problem.

So slow everything down for the 1% (or less) cases? I don't think
that's the right solution. Or don't slow everything down and ignore
the mythfilldatabase users?

> However, with mythfilldatabase the problem of "Not Listed" programs
> is, I think, much smaller.
> When mythfilldatabase runs e.g. once a day early in the morning, then all the
> "Single Record" type of recordings scheduled during that day and scheduled
> for that day will stay in the schedule.

That's not necessarrily the case any more. The Schedule's Direct,
xmltv grabbers are efficient enough to run multiple times a day. I do
and I believe many other users do as well.

> For recordings scheduled days ahead a check once a day is sufficient.
> This is different for EIT where program guide updates come at any moment.
>
> About the root cause
> I would like to point out again that this bug is made possible by what
> is to me a design flaw.
> An entry in table program is identified by the chanid plus the starttime.
> The starttime is part of the the data of the program (program as in TV
> program....)
> so if the starttime changes that entry in table program cannot be found anymore.
> The solution for this, as implemented in the EIT update code in programdata.cpp,
> is to modify the starttime everywhere there is a reference (chanid
> plus starttime)
> to an entry in table program when the starttime in that entry is changed.
> This is done for a number of tables but not for table record.
> The patch I wrote does this also for table record.
> A better solution would be to add a unique key to table program and use that
> in all references but that is rather more work and even worse, does
> require a database schema change.

The poor design of the program* tables sounds like a red herring to me
unless I misunderstand EIT. Does it include cast, rating of genre
information? If not, none of those other tables beyond the program
table proper will matter -- they won't have anything in them to change
anyway.

> Next steps
> I will create a ticket for this with a summary of this email thread so
> it is recorded
> for eternity (or as long as the trac server will live....) together
> with the patch.
> I intended to commit it to master a few days after that.
> N.B. I have been doing lots of testing on this now, including test with two
> video sources with overlapping channels as suggested, and for me it works OK.
>
> Thanks for the feedback & Cheers!
> Klaas.
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org

--
David Engel
david@istwok.net
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Hi,

Am 02.09.19 um 21:11 schrieb Klaas de Waal:
> About the root cause
> I would like to point out again that this bug is made possible by what
> is to me a design flaw.
> An entry in table program is identified by the chanid plus the starttime.
> The starttime is part of the the data of the program (program as in TV
> program....)
> so if the starttime changes that entry in table program cannot be found anymore.
> The solution for this, as implemented in the EIT update code in programdata.cpp,
> is to modify the starttime everywhere there is a reference (chanid
> plus starttime)
> to an entry in table program when the starttime in that entry is changed.
> This is done for a number of tables but not for table record.
> The patch I wrote does this also for table record.
> A better solution would be to add a unique key to table program and use that
> in all references but that is rather more work and even worse, does
> require a database schema change.

Both MySQL and PostgreSQL support "ON UPDATE CASCADE", but the Schema
definition all lack the required "REFERENCES $table($column)" statements
for that to work. Then the DBMS would do the job for you, faster and
less error prone and guaranteed to follow the ACID properties.

I already added most 'REFERENCES'-statements to the table descriptions
in the Wiki? as I had to look up that information multiple times is the
past, so that might be an elegant alternative solution.

Thanks.
Philipp

?: <https://www.mythtv.org/wiki/Category:DB_Table>
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org