Mailing List Archive

mythfilldatabase is adding credits out of order
Hi Mythtics,

I've looked in the code and it's way beyond me so hopefully someone can
help me fix this:

I get listings from tv_grab_zz_sdjson and that has credits in billing
order. I use that and a bit of a database hack in an attempt to list the
actors in a tv progam in billing order. Most of the time it works but I
noticed this today.

XML from tv_grab_zz_sdjson:

<programme start="20181225232000 +0000" stop="20181226014500 +0000"
channel="48020">
<title>The Silence of the Lambs</title>
<desc>Multi-Oscar-winning thriller about a trainee female FBI agent
.....</desc>
<credits>
<director>Jonathan Demme</director>
<actor role="Clarice Starling">Jodie Foster</actor>
<actor role="Dr. Hannibal Lecter">Anthony Hopkins</actor>
<actor role="Jack Crawford">Scott Glenn</actor>
<actor role="Jame &quot;Buffalo Bill&quot; Gumb">Ted Levine</actor>
<actor role="Dr. Frederick Chilton">Anthony Heald</actor>
<actor role="Catherine Martin">Brooke Smith</actor>
<actor role="Senator Ruth Martin">Diane Baker</actor>
<actor role="Ardelia Mapp">Kasi Lemmons</actor>
<actor role="Lt. Boyle">Charles Napier</actor>
<actor role="FBI Director Hayden Burke">Roger Corman</actor>
<actor role="FBI Agent in Memphis">George A. Romero</actor>
<writer>Thomas Harris</writer>
<writer>Ted Tally</writer>
<producer>Grace Blake</producer>
<producer>Ron Bozman</producer>
<producer>Gary Goetzman</producer>
<producer>Edward Saxon</producer>
<producer>Kenneth Utt</producer>
</credits>
<date>1991</date>
<category>Thriller</category>
<category>Crime drama</category>
.....


That looks fine but my daily email has this:


The Silence of the Lambs (ITV4 Thriller film 1991 100%)
Charles Napier, Roger Corman, George A. Romero, Kasi Lemmons,
Brooke Smith
Multi-Oscar-winning thriller about a trainee female FBI agent who is
seconded to the hunt for a horrific serial killer. For insight into
the
workings of his diseased mind she turns to the psychotic yet urbane
Dr
Hannibal "the Cannibal" Lecter, with whom she forms a compelling and
dangerous relationship.

The hack I use to try to get a list of actors in order is this SQL:

"alter table credits add column localrank int auto_increment primary
key;"

I do this before fetching the listings, generate my list of new programs
into an email and then remove the localrank column so the credits table
is restored to standard format.

The SQL to extract program data is this:

"select localrank, role, name from credits, people where credits.person
= people.person and chanid = $chanid and starttime =
from_unixtime($unix_st) order by localrank;"

and I get this:

localrank role name
162737 writer Thomas Harris
162738 producer Gary Goetzman
162739 producer Ron Bozman
162740 producer Grace Blake
162741 producer Kenneth Utt
162742 producer Edward Saxon
162743 actor Charles Napier
162744 actor Roger Corman
162745 writer Ted Tally
162746 actor George A. Romero
162747 actor Kasi Lemmons
162748 actor Brooke Smith
162749 actor Anthony Hopkins
162750 actor Scott Glenn
162751 director Jonathan Demme
162752 actor Jodie Foster
162753 actor Diane Baker
162754 actor Ted Levine
162755 actor Anthony Heald

As you can see localrank is an incrementing integer but those actor
names are not in the order they came from the XML. I can only think that
mythfilldatabase is adding the credits from XMLTV to a internal buffer
in some way that does not preserve order before sending to the database.

Is it possible to fix this? I don't care if mythfilldatabase takes
longer to run and a patch will do if a real fix would be problematic.
BTW the credits on mythfrontend are not in order, I'm not worried about
that as it includes all the credits but my email only lists the first
five and having them in billing order makes it easy to spot interesting
films.

Cheers,
Tim.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase is adding credits out of order [ In reply to ]
On 12/11/2018 07:33 AM, TimP wrote:
> Hi Mythtics,
>
> I've looked in the code and it's way beyond me so hopefully someone
> can help me fix this:
>
> I get listings from tv_grab_zz_sdjson and that has credits in billing
> order. I use that and a bit of a database hack in an attempt to list
> the actors in a tv progam in billing order. Most of the time it works
> but I noticed this today.
>
> XML from tv_grab_zz_sdjson:
>
> <programme start="20181225232000 +0000" stop="20181226014500 +0000"
> channel="48020">
> <title>The Silence of the Lambs</title>
> <desc>Multi-Oscar-winning thriller about a trainee female FBI agent
> .....</desc>
> <credits>
> <director>Jonathan Demme</director>
> <actor role="Clarice Starling">Jodie Foster</actor>
> <actor role="Dr. Hannibal Lecter">Anthony Hopkins</actor>
> <actor role="Jack Crawford">Scott Glenn</actor>
> <actor role="Jame &quot;Buffalo Bill&quot; Gumb">Ted Levine</actor>
> <actor role="Dr. Frederick Chilton">Anthony Heald</actor>
> <actor role="Catherine Martin">Brooke Smith</actor>
> <actor role="Senator Ruth Martin">Diane Baker</actor>
> <actor role="Ardelia Mapp">Kasi Lemmons</actor>
> <actor role="Lt. Boyle">Charles Napier</actor>
> <actor role="FBI Director Hayden Burke">Roger Corman</actor>
> <actor role="FBI Agent in Memphis">George A. Romero</actor>
> <writer>Thomas Harris</writer>
> <writer>Ted Tally</writer>
> <producer>Grace Blake</producer>
> <producer>Ron Bozman</producer>
> <producer>Gary Goetzman</producer>
> <producer>Edward Saxon</producer>
> <producer>Kenneth Utt</producer>
> </credits>
> <date>1991</date>
> <category>Thriller</category>
> <category>Crime drama</category>
> .....
>
>
> That looks fine but my daily email has this:
>
>
> The Silence of the Lambs (ITV4 Thriller film 1991 100%)
> Charles Napier, Roger Corman, George A. Romero, Kasi Lemmons,
> Brooke Smith
> Multi-Oscar-winning thriller about a trainee female FBI agent who is
> seconded to the hunt for a horrific serial killer. For insight into
> the
> workings of his diseased mind she turns to the psychotic yet urbane Dr
> Hannibal "the Cannibal" Lecter, with whom she forms a compelling and
> dangerous relationship.
>
> The hack I use to try to get a list of actors in order is this SQL:
>
> "alter table credits add column localrank int auto_increment primary
> key;"
>
> I do this before fetching the listings, generate my list of new
> programs into an email and then remove the localrank column so the
> credits table is restored to standard format.
>
> The SQL to extract program data is this:
>
> "select localrank, role, name from credits, people where
> credits.person = people.person and chanid = $chanid and starttime =
> from_unixtime($unix_st) order by localrank;"
>
> and I get this:
>
> localrank role name
> 162737 writer Thomas Harris
> 162738 producer Gary Goetzman
> 162739 producer Ron Bozman
> 162740 producer Grace Blake
> 162741 producer Kenneth Utt
> 162742 producer Edward Saxon
> 162743 actor Charles Napier
> 162744 actor Roger Corman
> 162745 writer Ted Tally
> 162746 actor George A. Romero
> 162747 actor Kasi Lemmons
> 162748 actor Brooke Smith
> 162749 actor Anthony Hopkins
> 162750 actor Scott Glenn
> 162751 director Jonathan Demme
> 162752 actor Jodie Foster
> 162753 actor Diane Baker
> 162754 actor Ted Levine
> 162755 actor Anthony Heald
>
> As you can see localrank is an incrementing integer but those actor
> names are not in the order they came from the XML. I can only think
> that mythfilldatabase is adding the credits from XMLTV to a internal
> buffer in some way that does not preserve order before sending to the
> database.
>
> Is it possible to fix this? I don't care if mythfilldatabase takes
> longer to run and a patch will do if a real fix would be problematic.
> BTW the credits on mythfrontend are not in order, I'm not worried
> about that as it includes all the credits but my email only lists the
> first five and having them in billing order makes it easy to spot
> interesting films.
>
> Cheers,
> Tim.

I'm guessing you may be the reporter of the bug, but I'm referencing for
completeness:
https://code.mythtv.org/trac/ticket/12887

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase is adding credits out of order [ In reply to ]
On 2018-12-11 17:00, Michael T. Dean wrote:
> On 12/11/2018 07:33 AM, TimP wrote:
>
> I'm guessing you may be the reporter of the bug, but I'm referencing
> for completeness:
> https://code.mythtv.org/trac/ticket/12887
>

Problem identified: It's my script to email new programs, and it doesn't
always email them when they are added by mythfilldatabse. That means my
hack to the credits database to add a sort column fails as my script
removes that column every day and adds it just before mythfilldatabase.
The reason it doesn't always email on insertion is that the script also
filters out any programme title inserted in the last 11 months
(otherwise the email list is TLDR) and occaisionally that 11 month
filter elapses for a film a few days after insertion by
mythfilldatabase.

My fix is to not remove the sorting columns. I'll have to remember to to
remove it before an upgrade (which should also overcome the future
problem of the sort index hitting 4 billion, so please release an
upgrade at least once a decade).

The ticket is still valid asadding the sort order officially would allow
mythfrontend to list credits in order (with a fix to mythfrontend) but
is not so simple as it makes the table larger and using a small sized
token will mean it hits the max sooner without some code to repack the
credits order.

Cheers,m
Tim.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org