Mailing List Archive

Python 3
Recently decided to make Python 3 the default on my system, since Python 2.7 is officially dead. The EPG update stopped working. Is there a Python 3 version of the MythTV module? Currently working round it by forcing mhegepgsnoop.py to run with python2, but would prefer a 'proper' fix.

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
?For whatever reason mhegepgsnoop.py stopped working quite a while ago on my Fedora MythTV system (even with the latest version at the time). I now just get the EPG listing from DVB-T directly. It is for 7+ days. Only thing I can’t do with it is easily edit out words like “Movie:” or “All New” from titles so I need to be bit more careful when setting up recordings where the title may change for the same actual program. For example I have two set up for NCIS: “NCIS” and “All New NCIS”.

On Wed, 10 Mar 2021 at 11:47 AM, Austin Green <austin.green@orcon.net.nz> wrote:
> Recently decided to make Python 3 the default on my system, since Python 2.7 is officially dead. The EPG update stopped working. Is there a Python 3 version of the MythTV module? Currently working round it by forcing mhegepgsnoop.py to run with python2, but would prefer a 'proper' fix.
>
> _______________________________________________
> mythtvnz mailing list
> mythtvnz@lists.ourshack.com
> https://lists.ourshack.com/mailman/listinfo/mythtvnz
> Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
On Wed, 10 Mar 2021 11:46:18 +1300, you wrote:

>Recently decided to make Python 3 the default on my system, since Python 2.7 is officially dead. The EPG update stopped working. Is there a Python 3 version of the MythTV module? Currently working round it by forcing mhegepgsnoop.py to run with python2, but would prefer a 'proper' fix.

That is what I am doing. I did try making a Python 3 version of
mhegepgsnoop.py but did not get it fully working as I do not
understand the MHEG5 protocol so I did not understand what some of the
code was supposed to be doing. The problem was that the MHEG-5
standards documents are paywalled - and cost heaps. But today I did
another search and found one that says it is the final draft of
MHEG-5, which would probably be good enough for me to use. So I will
have another look at getting mhegepgsnoop.py working with Python 3.

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
On Wed, 10 Mar 2021 15:10:28 +1300
Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:

> On Wed, 10 Mar 2021 11:46:18 +1300, Austin Green wrote:
> >Recently decided to make Python 3 the default on my system, since Python 2.7 is officially dead. The EPG update stopped working. Is there a Python 3 version of the MythTV module? Currently working round it by forcing mhegepgsnoop.py to run with python2, but would prefer a 'proper' fix.
>
> That is what I am doing. I did try making a Python 3 version of
> mhegepgsnoop.py but did not get it fully working as I do not
> understand the MHEG5 protocol so I did not understand what some of the
> code was supposed to be doing. The problem was that the MHEG-5
> standards documents are paywalled - and cost heaps. But today I did
> another search and found one that says it is the final draft of
> MHEG-5, which would probably be good enough for me to use. So I will
> have another look at getting mhegepgsnoop.py working with Python 3.

No big deal to convert mhegepgsnoop.py, just the format of 'print' calls and one IO exception; don't need to understand what it's trying to do (fortunately!). The problem arises when it tries to use the Python MythTV API - it complains "No module named 'MythTV'". There is indeed no Python3 version of the MythTV module, or if there is, the pip installer doesn't know where to find it (and nor do I). But surely someone will have done a Python 3 version of it by now?

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
On Wed, 10 Mar 2021 15:04:01 +1300, you wrote:

>?For whatever reason mhegepgsnoop.py stopped working quite a while ago on my Fedora MythTV system (even with the latest version at the time). I now just get the EPG listing from DVB-T directly. It is for 7+ days. Only thing I can?t do with it is easily edit out words like ?Movie:? or ?All New? from titles so I need to be bit more careful when setting up recordings where the title may change for the same actual program. For example I have two set up for NCIS: ?NCIS? and ?All New NCIS?.

It is not unlikely that your Fedora problem happened when they
upgraded their Python to Python 3 and made that the default. To run
mhegepgsnoop.py after that, you would have to change the first line of
the file (the #! line) to something that ran it using Python 2 instead
of the default Python 3. The exact format of the #! line needed can
vary depending on the distro, but this might work:

#!/usr/bin/env python2

To fix the problems with "Movie:" and "All New" you need to run
xmltv_proc_nz on the EPG data after it has been collected but before
mythfilldatabase puts it into the database. If you use the latest
version, this will also provide series/episode numbers for some
channels that put it in the description field. Or alternatively, you
can do fixes directly in the database by regularly running an SQL
script. So how exactly are you collecting your EPG now? Have you
just enabled the internal MythTV EIT collection option?

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
Yes, you are probably correct. I was last using mhegepgsnoop-0.6.0.py
and amended the following lines to clean up the titles:

parser.add_argument('-c', dest='clean_titles', nargs='?', const='All New
|Movie: |Prime Flicks: |New: ', default=defs.clean_titles,

options.clean_titles = 'All New |Movie: |Prime Flicks: |New: '

Don't know much about python coding, but this cleaned up the titles.

re: what am I using now? I am using the internal MythTV EIT collection
option. This does collect 7 days now (I think originally it was just now
and next few programs). The data collected isn't quite as informative
(it doesn't show for example the type of program beyond for the current
and next two programs) but for my purposes it 'solved' the problem I had
with mhegepgsnoop not working for me.


On 3/10/2021 5:02 PM, Stephen Worthington wrote:
> On Wed, 10 Mar 2021 15:04:01 +1300, you wrote:
>
>> ?For whatever reason mhegepgsnoop.py stopped working quite a while ago on my Fedora MythTV system (even with the latest version at the time). I now just get the EPG listing from DVB-T directly. It is for 7+ days. Only thing I can?t do with it is easily edit out words like ?Movie:? or ?All New? from titles so I need to be bit more careful when setting up recordings where the title may change for the same actual program. For example I have two set up for NCIS: ?NCIS? and ?All New NCIS?.
> It is not unlikely that your Fedora problem happened when they
> upgraded their Python to Python 3 and made that the default. To run
> mhegepgsnoop.py after that, you would have to change the first line of
> the file (the #! line) to something that ran it using Python 2 instead
> of the default Python 3. The exact format of the #! line needed can
> vary depending on the distro, but this might work:
>
> #!/usr/bin/env python2
>
> To fix the problems with "Movie:" and "All New" you need to run
> xmltv_proc_nz on the EPG data after it has been collected but before
> mythfilldatabase puts it into the database. If you use the latest
> version, this will also provide series/episode numbers for some
> channels that put it in the description field. Or alternatively, you
> can do fixes directly in the database by regularly running an SQL
> script. So how exactly are you collecting your EPG now? Have you
> just enabled the internal MythTV EIT collection option?
>
> _______________________________________________
> mythtvnz mailing list
> mythtvnz@lists.ourshack.com
> https://lists.ourshack.com/mailman/listinfo/mythtvnz
> Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
On Wed, 10 Mar 2021 18:17:55 +1300, you wrote:

>Yes, you are probably correct. I was last using mhegepgsnoop-0.6.0.py
>and amended the following lines to clean up the titles:
>
>parser.add_argument('-c', dest='clean_titles', nargs='?', const='All New
>|Movie: |Prime Flicks: |New: ', default=defs.clean_titles,
>
>options.clean_titles = 'All New |Movie: |Prime Flicks: |New: '
>
>Don't know much about python coding, but this cleaned up the titles.
>
>re: what am I using now? I am using the internal MythTV EIT collection
>option. This does collect 7 days now (I think originally it was just now
>and next few programs). The data collected isn't quite as informative
>(it doesn't show for example the type of program beyond for the current
>and next two programs) but for my purposes it 'solved' the problem I had
>with mhegepgsnoop not working for me.

Ok, so the best path forwards is probably to go back to using
mhegepgsnoop.py again, and then add in xmltv-proc-nz processing. Until
a Python 3 version of mhegepgsnoop.py is available, that means running
it with Python 2. So, to find out if your system still has Python 2
support, run this:

python2 --version

If that works, then Python 2 is installed but not the default and you
just need to change the #! line in mhegepgsnoop.py.

If that does not work, then you should still be able to install Python
2 in parallel with Python 3:

dnf install python2

That worked for me on my Fedora 33 virtual machine. Then you will
likely need to install several other Python 2 packages (sqlite3, zlib
and difflib at least), and there will probably be more that will be
needed when mhegepgsnoop.py fails to run because they are not present.
Look to see if they are available as system packages first, and if
not, install them with pip:

pip2 install <package name>

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
On Wed, 10 Mar 2021 16:48:55 +1300, you wrote:

>On Wed, 10 Mar 2021 15:10:28 +1300
>Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:
>
>> On Wed, 10 Mar 2021 11:46:18 +1300, Austin Green wrote:
>> >Recently decided to make Python 3 the default on my system, since Python 2.7 is officially dead. The EPG update stopped working. Is there a Python 3 version of the MythTV module? Currently working round it by forcing mhegepgsnoop.py to run with python2, but would prefer a 'proper' fix.
>>
>> That is what I am doing. I did try making a Python 3 version of
>> mhegepgsnoop.py but did not get it fully working as I do not
>> understand the MHEG5 protocol so I did not understand what some of the
>> code was supposed to be doing. The problem was that the MHEG-5
>> standards documents are paywalled - and cost heaps. But today I did
>> another search and found one that says it is the final draft of
>> MHEG-5, which would probably be good enough for me to use. So I will
>> have another look at getting mhegepgsnoop.py working with Python 3.
>
>No big deal to convert mhegepgsnoop.py, just the format of 'print' calls and one IO exception; don't need to understand what it's trying to do (fortunately!). The problem arises when it tries to use the Python MythTV API - it complains "No module named 'MythTV'". There is indeed no Python3 version of the MythTV module, or if there is, the pip installer doesn't know where to find it (and nor do I). But surely someone will have done a Python 3 version of it by now?

So do I understand from this that you have a Python 3 version of
mhegepgsnoop.py that runs and only fails on not finding a MythTV
module? I think that because it is not finding that module, it is
actually failing very early, before most of the problem code even gets
executed. You need to try running mhegepgsnoop.py in the mode where
it does not use the MythTV bindings. This is how I run it:

root@mypvr:/tmp# cat /usr/local/bin/do_mhegepgsnoop.sh
#!/bin/bash

# DVB-T multiplex frequency (kHz)
DVB_T_FREQ=610000

# Adapter number of DVB-T card.
ADAPTER=17

# Number of times to try mhegepgsnoop.py if it fails.
TRIES=5

# Amount of time to sleep between tries.
SLEEP_TIME=5m

# Output file to store the xmltv EPG data.
TEMP_FILE=/tmp/xmltv.xml
OUTPUT_FILE=/var/www/html/epg/listings-mheg5.xml

# Channel map file.
MAP_FILE=/home/stephen/bin/mhegepgsnoop_channel_map.txt


retval=0

#PWD=`pwd`

if [ -e "$TEMP_FILE" ] && [ -f "$TEMP_FILE" ] ; then
rm "$TEMP_FILE"
fi

for ((i=1; i<=$TRIES; i++)); do
# Get the MHEG5 EPG data from the DVT-T multiplex.
echo "Try $i, Running dvbtune"
dvbtune -f $DVB_T_FREQ -qam 64 -gi 16 -cr 3_4 -bw 8 -tm 8 -m -c
$ADAPTER 2>&1 &
DVBTUNE_PID=$!
sleep 1
echo "Try $i, running mhegepgsnoop.py now"
mhegepgsnoop.py2 -v -z -d /dev/dvb/adapter${ADAPTER}/demux0 -o
$TEMP_FILE -f $MAP_FILE
retval=$?
kill $DVBTUNE_PID
if [ $retval -eq 0 ]; then
break
fi
rm $TEMP_FILE
echo "Warning: mhegepgsnoop failed (retval=$retval), retrying
after sleep $SLEEP_TIME"
sleep $SLEEP_TIME
done
if [ $retval -ne 0 ]; then
echo "Error: mhegepgsnoop failed after $TRIES tries
(retval=$retval)!"
else
# Temporary fix for bad <channel id> values in the MHEG5 EPG data.
sed -f /usr/local/bin/mhegepgsnoop_sed_replacements --in-place
$TEMP_FILE

mv $TEMP_FILE $OUTPUT_FILE
chmod a+rw $OUTPUT_FILE
fi

exit $retval


For me, the conversion to Python 3 was not simple - mhegepgsnoop.py
uses some nice coding tricks to extract the data structures in the
data it downloads, prior to outputting the data as XML. Those coding
tricks did not convert at all to Python 3 as they involve strings and
bytes and that is a place where there are big changes between Python 2
and Python 3. So my Python 3 version works up until the point where
it has completed downloading the data, and then crashes as it tries to
convert the data to XML. The proper fix would be a rewrite to use the
new Python 3 features that handle structures in data, but I would be
happy if I could just work out how to adjust the exiting code for
Python 3.

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
Hi Stephen,

On Wed, 10 Mar 2021 23:01:26 +1300
Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:

> So do I understand from this that you have a Python 3 version of
> mhegepgsnoop.py that runs and only fails on not finding a MythTV
> module? I think that because it is not finding that module, it is
> actually failing very early, before most of the problem code even gets
> executed. You need to try running mhegepgsnoop.py in the mode where
> it does not use the MythTV bindings.

You're right, it was trying to load the MythTV module, and failing because there is no such. I was puzzled, because the default operation is NOT to use MythTV; I tracked that down to the options parser, which had
action='store_true'
where it should have been
action='store_false'
After fixing that, it ran into the bytes-versus-string problems you mentioned.

I have been working through those, and I have a version that works, or at least runs to completion without crashing; need to test some more.

BTW, the version I have is called 0.6.2-jsw -- is that you, by any chance?

Cheers,
Austin.

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/
Re: Python 3 [ In reply to ]
On Sat, 13 Mar 2021 17:21:48 +1300, you wrote:

>Hi Stephen,
>
>On Wed, 10 Mar 2021 23:01:26 +1300
>Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:
>
>> So do I understand from this that you have a Python 3 version of
>> mhegepgsnoop.py that runs and only fails on not finding a MythTV
>> module? I think that because it is not finding that module, it is
>> actually failing very early, before most of the problem code even gets
>> executed. You need to try running mhegepgsnoop.py in the mode where
>> it does not use the MythTV bindings.
>
>You're right, it was trying to load the MythTV module, and failing because there is no such. I was puzzled, because the default operation is NOT to use MythTV; I tracked that down to the options parser, which had
> action='store_true'
>where it should have been
> action='store_false'
>After fixing that, it ran into the bytes-versus-string problems you mentioned.
>
>I have been working through those, and I have a version that works, or at least runs to completion without crashing; need to test some more.
>
>BTW, the version I have is called 0.6.2-jsw -- is that you, by any chance?
>
>Cheers,
>Austin.

Yes, 0.6.2 JSW has my fix for the buffer overflow problems when
reading from the tuner.

Let me know if you want me to test it for your. My version is not as
far advanced as yours, so if you are making good progress, I may as
well wait for yours to be ready. But my version is working with -p (=
use MythTV Python bindings).

_______________________________________________
mythtvnz mailing list
mythtvnz@lists.ourshack.com
https://lists.ourshack.com/mailman/listinfo/mythtvnz
Archives http://www.gossamer-threads.com/lists/mythtv/mythtvnz/