Mailing List Archive

Hi,

These all have "All New " in front for me (using tv_grab_nz-py),
except possibly repeats which have the normal titles. I don't do any
post processing. Something isn't removing "All New" and not removing
the trailing space on your end?

All New Dr. Ken
All New Mike & Molly
All New The Big Bang Theory
All New 2 Broke Girls
All New The Middle

Duke has "New: " just to be different. And of course "New Girl" is valid.

On Mon, Oct 17, 2016 at 4:19 PM, Stephen Worthington
<stephen_agent@jsw.gen.nz> wrote:
> It looks like TVNZ has invented a new way to create crappy EPG
> listings - adding a blank character at the start of the title for some
> programmes. This is what I found in my database when I looked at why
> the new series of The Middle was not recording:
>
> select
> chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
> from program where left(title,1)=' '
>
> +--------+---------------------+----------------------+------------+----------------------+--------+---------+----------+-----------+-----------+
> | chanid | starttime | title | subtitle | left(description,20) | season | episode | category | seriesid | programid |
> +--------+---------------------+----------------------+------------+----------------------+--------+---------+----------+-----------+-----------+
> | 1007 | 2016-10-18 07:30:00 | The Middle | New Season | S8 E1 Our favourite | 0 | 0 | | 220783125 | 10454729 |
> | 1002 | 2016-10-18 06:30:00 | The Middle | New Season | S8 E1 Our favourite | 0 | 0 | | 220783125 | 10454729 |
> | 1007 | 2016-10-19 07:30:00 | Dr. Ken | | S2 E2 Allison's firs | 0 | 0 | | 86298846 | 10447741 |
> | 1007 | 2016-10-19 08:00:00 | Mike & Molly | | S6 E4 Mike turns int | 0 | 0 | | 262741513 | 10421583 |
> | 1007 | 2016-10-19 08:30:00 | The Big Bang Theory | | S10 E5 Shamy navigat | 0 | 0 | | 139824009 | 10454708 |
> | 1007 | 2016-10-19 09:00:00 | 2 Broke Girls | | S5 E20 A law firm th | 0 | 0 | | 223419955 | 10421612 |
> | 1002 | 2016-10-19 06:30:00 | Dr. Ken | | S2 E2 Allison's firs | 0 | 0 | | 86298846 | 10447741 |
> | 1002 | 2016-10-19 07:00:00 | Mike & Molly | | S6 E4 Mike turns int | 0 | 0 | | 262741513 | 10421583 |
> | 1002 | 2016-10-19 07:30:00 | The Big Bang Theory | | S10 E5 Shamy navigat | 0 | 0 | | 139824009 | 10454708 |
> | 1002 | 2016-10-19 08:00:00 | 2 Broke Girls | | S5 E20 A law firm th | 0 | 0 | | 223419955 | 10421612 |
> +--------+---------------------+----------------------+------------+----------------------+--------+---------+----------+-----------+-----------+
> 10 rows in set (0.10 sec)
>
> So it is happening on TVNZ 2 (chanid 1002) and TVNZ 2+1 (chanid 1007).
>
> As a temporary fix, I have added this processing to the SQL I do
> automatically after an EPG update:
>
> update program set title=substr(title,2) where left(title,1)=' ';
>
> _______________________________________________
> mythtvnz mailing list
> mythtvnz@lists.linuxnut.co.nz
> http://lists.ourshack.com/mailman/listinfo/mythtvnz
> Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.linuxnut.co.nz
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

On Mon, 17 Oct 2016 17:44:04 +1300, you wrote:

>Hi,
>
>These all have "All New " in front for me (using tv_grab_nz-py),
>except possibly repeats which have the normal titles. I don't do any
>post processing. Something isn't removing "All New" and not removing
>the trailing space on your end?
>
>All New Dr. Ken
>All New Mike & Molly
>All New The Big Bang Theory
>All New 2 Broke Girls
>All New The Middle
>
>Duke has "New: " just to be different. And of course "New Girl" is valid.

Where are you getting your EPG from? The downloaded EPG does have
processing to remove the "All New " bit, but I think it does not
handle "All New: " and definitely does not handle "New: ". What I
have done is to do postprocessing on my database after
mythfilldatabase has been run. This is what I currently have in my
epg_fixes.sh script:

#!/bin/bash

##############################################################################
# epg_fixes.sh
#
# Fix EPG problems by editing the mythconverg.program table.
#
# Author: J S Worthington
# Version: 0.1 2016-08-30
#
##############################################################################

which xmlstarlet >/dev/null
if [ $? != 0 ] ; then
echo 'Error: xmlstarlet is not installed.'
echo 'On Ubuntu, you can use the command "sudo apt-get install
xmlstarlet" to install it.'
exit 1
fi

configxml=/etc/mythtv/config.xml

DBUserName=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBUserName $configxml`
if [ "$DBUserName" == "" ]; then
# Either config.xml does not exist, or it is in MythTV 0.26 or
greater
# format. Try 0.26+ format.

DBUserName=`xmlstarlet sel -t -v Configuration/Database/UserName
$configxml`
if [ "$DBUserName" == "" ]; then
echo "Error: Either $configxml does not exist or it is in an
unknown format."
exit 1
fi

DBPassword=`xmlstarlet sel -t -v Configuration/Database/Password
$configxml`
DBHostName=`xmlstarlet sel -t -v Configuration/Database/Host
$configxml`
DBName=`xmlstarlet sel -t -v Configuration/Database/DatabaseName
$configxml`
else
# The config.xml file is in 0.25 and before format.

DBPassword=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBPassword $configxml`
DBHostName=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBHostName $configxml`
DBName=`xmlstarlet sel -t -v
Configuration/UPnP/MythFrontend/DefaultBackend/DBName $configxml`
fi

mysql -vvv --show-warnings --local-infile -u $DBUserName -p$DBPassword
-h $DBHostName $DBName << EOF
select chanid,starttime,title,subtitle,programid from program where
title like 'All New %';
update program set title=substr(title,8) where title like 'All New %';
select chanid,starttime,title,substr(title,6) as
title6,subtitle,programid from program where title like 'New: %';
update program set title=substr(title,6) where title like 'New: %';
select chanid,starttime,title,subtitle,programid from program where
title like 'Cleverman %';
update program set title='Cleverman' where title like 'Cleverman %';
select
chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
from program where title like '%trans-si%';
update program set programid='' where title="Joanna Lumley's
Trans-Siberian";
select
chanid,starttime,title,subtitle,left(description,20),season,episode,category,seriesid,programid
from program where left(title,1)=' ';
update program set title=substr(title,2) where left(title,1)=' ';
EOF

mythutil --resched

If anyone wants a copy, I have it on my web server at:

http://www.jsw.gen.nz/mythtv/epg_fixes.sh

Most of the fixes are obvious, but the Joanna Lumley one is for a bad
programid value that tells MythTV that all the episodes are the same
episode, when in fact there is a new episode weekly. Without that
fix, the duplicate detection was using the programid and saying they
all matched, when the description was changing each week.

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.linuxnut.co.nz
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

On Mon, Oct 17, 2016 at 7:43 PM, Stephen Worthington
<stephen_agent@jsw.gen.nz> wrote:
> On Mon, 17 Oct 2016 17:44:04 +1300, you wrote:
>
>>Hi,
>>
>>These all have "All New " in front for me (using tv_grab_nz-py),
>>except possibly repeats which have the normal titles. I don't do any
>>post processing. Something isn't removing "All New" and not removing
>>the trailing space on your end?
>>
>>All New Dr. Ken
>>All New Mike & Molly
>>All New The Big Bang Theory
>>All New 2 Broke Girls
>>All New The Middle
>>
>>Duke has "New: " just to be different. And of course "New Girl" is valid.
>
> Where are you getting your EPG from? The downloaded EPG does have
> processing to remove the "All New " bit, but I think it does not
> handle "All New: " and definitely does not handle "New: ". What I
> have done is to do postprocessing on my database after
> mythfilldatabase has been run. This is what I currently have in my
> epg_fixes.sh script:

epg.org.nz/freeview.xml.gz

I haven't touched this in a couple of years and may way well have
something out of date and/or missing, but if I download the XML and
take a look I see:

<programme channel="tv2-plus1.freeviewnz.tv" start="20161018203000
+1300" stop="20161018210000 +1300">
<title>All New The Middle</title>
<sub-title>New Season</sub-title>
<desc>S8 E1 Our favourite family from Orson, Indiana are back
for an 8th season! In the season premiere Frankie and Mike prepare to
meet April - the love of Axl's life.</desc>
<episode-num system="dd_progid">10454729</episode-num>
<subtitles type="teletext" />
<rating system="Freeview">
<value>PGR</value>
</rating>

I've not seen "All New" get removed in the source.


> #!/bin/bash
>
> ##############################################################################
> # epg_fixes.sh

> select chanid,starttime,title,subtitle,programid from program where
> title like 'All New %';
> update program set title=substr(title,8) where title like 'All New %';

Shouldn't that be:

update program set title=substr(title,9) where title like 'All New %';

And another select and update for 'All New: %'? Not sure I've noticed
that syntax in the RPG but if not now then surely we'll see it in
time.

Thanks for the script, I'd been thinking about tidying up some of this myself.

> If anyone wants a copy, I have it on my web server at:
>
> http://www.jsw.gen.nz/mythtv/epg_fixes.sh
>
> Most of the fixes are obvious, but the Joanna Lumley one is for a bad
> programid value that tells MythTV that all the episodes are the same
> episode, when in fact there is a new episode weekly. Without that
> fix, the duplicate detection was using the programid and saying they
> all matched, when the description was changing each week.
>
> _______________________________________________
> mythtvnz mailing list
> mythtvnz@lists.linuxnut.co.nz
> http://lists.ourshack.com/mailman/listinfo/mythtvnz
> Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.linuxnut.co.nz
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

On Mon, 17 Oct 2016 22:26:38 +1300, you wrote:

>On Mon, Oct 17, 2016 at 7:43 PM, Stephen Worthington
><stephen_agent@jsw.gen.nz> wrote:
>> On Mon, 17 Oct 2016 17:44:04 +1300, you wrote:
>>
>>>Hi,
>>>
>>>These all have "All New " in front for me (using tv_grab_nz-py),
>>>except possibly repeats which have the normal titles. I don't do any
>>>post processing. Something isn't removing "All New" and not removing
>>>the trailing space on your end?
>>>
>>>All New Dr. Ken
>>>All New Mike & Molly
>>>All New The Big Bang Theory
>>>All New 2 Broke Girls
>>>All New The Middle
>>>
>>>Duke has "New: " just to be different. And of course "New Girl" is valid.
>>
>> Where are you getting your EPG from? The downloaded EPG does have
>> processing to remove the "All New " bit, but I think it does not
>> handle "All New: " and definitely does not handle "New: ". What I
>> have done is to do postprocessing on my database after
>> mythfilldatabase has been run. This is what I currently have in my
>> epg_fixes.sh script:
>
>epg.org.nz/freeview.xml.gz
>
>I haven't touched this in a couple of years and may way well have
>something out of date and/or missing, but if I download the XML and
>take a look I see:
>
> <programme channel="tv2-plus1.freeviewnz.tv" start="20161018203000
>+1300" stop="20161018210000 +1300">
> <title>All New The Middle</title>
> <sub-title>New Season</sub-title>
> <desc>S8 E1 Our favourite family from Orson, Indiana are back
>for an 8th season! In the season premiere Frankie and Mike prepare to
>meet April - the love of Axl's life.</desc>
> <episode-num system="dd_progid">10454729</episode-num>
> <subtitles type="teletext" />
> <rating system="Freeview">
> <value>PGR</value>
> </rating>
>
>I've not seen "All New" get removed in the source.

It is supposed to be, but as I am now doing my own fixes, it is a bit
hard to check.

>> #!/bin/bash
>>
>> ##############################################################################
>> # epg_fixes.sh
>
>> select chanid,starttime,title,subtitle,programid from program where
>> title like 'All New %';
>> update program set title=substr(title,8) where title like 'All New %';
>
>Shouldn't that be:
>
>update program set title=substr(title,9) where title like 'All New %';

Thank you, it was a bug in my code causing the problem with the blank
characters.

>And another select and update for 'All New: %'? Not sure I've noticed
>that syntax in the RPG but if not now then surely we'll see it in
>time.

Yes, it probably should be added to the list.

>Thanks for the script, I'd been thinking about tidying up some of this myself.
>
>> If anyone wants a copy, I have it on my web server at:
>>
>> http://www.jsw.gen.nz/mythtv/epg_fixes.sh
>>
>> Most of the fixes are obvious, but the Joanna Lumley one is for a bad
>> programid value that tells MythTV that all the episodes are the same
>> episode, when in fact there is a new episode weekly. Without that
>> fix, the duplicate detection was using the programid and saying they
>> all matched, when the description was changing each week.

I have updated the script on my web server.

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.linuxnut.co.nz
http://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/