Mailing List Archive: Errors from myhproto when using mythvidexport.py

Errors from myhproto when using mythvidexport.py

Aug 24, 2020, 10:25 AM

Post #1 of 9 (835 views)

Greetings!

I just upgraded from v30 to v31, and have run in to a problem as described in the subject. I haven’t had any other problems yet surface from my upgrade (knock on wood!).

I have tweaked the mythvidexport.py script to use python3, and I am getting an error from mythproto when trying to open the movie file to copy to the videos directory.

2020-08-24 11:24:53.249257 I [11137] mythvidexport.py Using recording -- b'The Closer' - b'Mom Duty'
2020-08-24 11:24:53.321444 I [11137] mythvidexport.py Forcing TV export with local data.
2020-08-24 11:24:53.648925 I [11137] mythvidexport.py Import complete
2020-08-24 11:24:53.649381 I [11137] mythvidexport.py Copying myth://Default@mythpc/44303_20200811220000.mp4 to myth://Videos@mythpc/Television/The Closer/Season 02/The Closer - S02E02 - Mom Duty-44303-20200811180000.mp4
2020-08-24 11:24:53.969356 C [11137] mythvidexport.py
Traceback (most recent call last):
File "/usr/local/bin/mythvidexport.py", line 438, in main
export = VIDEO(opts,int(args[0]))
File "/usr/local/bin/mythvidexport.py", line 77, in __init__
self.copy()
File "/usr/local/bin/mythvidexport.py", line 269, in copy
dstfp.write(srcfp.read(tsize))
File "/usr/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 41: invalid start byte

I haven’t been able to track through all the steps as to where the file is being opened as I think that is where the problem is according to my search abilities.

Any ideas where I need to check? Or?

Thanks!
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

stephen_agent at jsw

Aug 25, 2020, 2:46 AM

Post #2 of 9 (829 views)

Permalink

On Mon, 24 Aug 2020 13:25:27 -0400, you wrote:

>Greetings!
>
>I just upgraded from v30 to v31, and have run in to a problem as described in the subject. I haven?t had any other problems yet surface from my upgrade (knock on wood!).
>
>
>I have tweaked the mythvidexport.py script to use python3, and I am getting an error from mythproto when trying to open the movie file to copy to the videos directory.
>
>2020-08-24 11:24:53.249257 I [11137] mythvidexport.py Using recording -- b'The Closer' - b'Mom Duty'
>2020-08-24 11:24:53.321444 I [11137] mythvidexport.py Forcing TV export with local data.
>2020-08-24 11:24:53.648925 I [11137] mythvidexport.py Import complete
>2020-08-24 11:24:53.649381 I [11137] mythvidexport.py Copying myth://Default@mythpc/44303_20200811220000.mp4 to myth://Videos@mythpc/Television/The Closer/Season 02/The Closer - S02E02 - Mom Duty-44303-20200811180000.mp4
>2020-08-24 11:24:53.969356 C [11137] mythvidexport.py
> Traceback (most recent call last):
> File "/usr/local/bin/mythvidexport.py", line 438, in main
> export = VIDEO(opts,int(args[0]))
> File "/usr/local/bin/mythvidexport.py", line 77, in __init__
> self.copy()
> File "/usr/local/bin/mythvidexport.py", line 269, in copy
> dstfp.write(srcfp.read(tsize))
> File "/usr/lib/python3.6/codecs.py", line 321, in decode
> (result, consumed) = self._buffer_decode(data, self.errors, final)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 41: invalid start byte
>
>
>I haven?t been able to track through all the steps as to where the file is being opened as I think that is where the problem is according to my search abilities.
>
>Any ideas where I need to check? Or?
>
>Thanks!

I have never used mythvidexport.py, but I have been converting my
Python 2 programs to Python 3. One of the things you have to watch
out for is that Python 3 needs to have things either using unicode
strings (the default), or byte data. Your problem looks like a case
where unicode strings were used when it should have been byte data
(type "bytes"). I am just guessing from the code in the traceback,
but it looks like it is copying a video file. A video file is byte
data and will occasionally contain data that does not represent valid
unicode, causing an error as above. So you need to find where the
source (srcfp) and destination (dstfp) files are opened and change the
open mode to binary ('rb' instead of 'r', 'wb' instead of 'w'). That
may be the entire fix that is needed as that will cause strings read
from the file to be created as "bytes" type, but if you still get
errors, ensure that "bytes" types are used for the buffers used for
the copying.

See here for more about the problem:

https://medium.com/better-programming/strings-unicode-and-bytes-in-python-3-everything-you-always-wanted-to-know-27dc02ff2686
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

rcrernst at gmail

Aug 25, 2020, 3:18 AM

Post #3 of 9 (829 views)

Permalink

On Tue, Aug 25, 2020 at 11:48 AM Stephen Worthington <
stephen_agent@jsw.gen.nz> wrote:

> On Mon, 24 Aug 2020 13:25:27 -0400, you wrote:
>
> >
> >Any ideas where I need to check? Or?
> >
>
>
This seems to be a leftover from the python3 conversion
for mythtv v31+.

If the storage groups of the recording and video is local,
i.e.: on the same PC mythvidexport is running, please try
the following:

- locate the installed file mythproto.py from the Python
Bindings (mine is in the location below)

- in terminal, run the following commands
(change the path to mythproto.patch)

{{{
$ cd /usr/lib/python3/dist-packages/MythTV/
$ ls ./mythproto.py
$ sudo cp ./mythproto.py ./mythproto.org
$ sudo patch < /path/to/mythproto.patch
$ sudo python3 -m compileall ./mythproto.py
}}}

Note, that this will be overwritten again when updating
mythtv.

Please report back the result.
On success, I will update the Python Bindings asap.

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

jharbestonus at gmail

Aug 25, 2020, 7:47 AM

Post #4 of 9 (825 views)

Permalink

> On Aug 25, 2020, at 6:18 AM, Roland Ernst <rcrernst@gmail.com> wrote:
>
>
> On Tue, Aug 25, 2020 at 11:48 AM Stephen Worthington <stephen_agent@jsw.gen.nz <mailto:stephen_agent@jsw.gen.nz>> wrote:
> On Mon, 24 Aug 2020 13:25:27 -0400, you wrote:
>
> >
> >Any ideas where I need to check? Or?
> >
>
>
> This seems to be a leftover from the python3 conversion
> for mythtv v31+.
>
> If the storage groups of the recording and video is local,
> i.e.: on the same PC mythvidexport is running, please try
> the following:
>
> - locate the installed file mythproto.py from the Python
> Bindings (mine is in the location below)
>
> - in terminal, run the following commands
> (change the path to mythproto.patch)
>
> {{{
> $ cd /usr/lib/python3/dist-packages/MythTV/
> $ ls ./mythproto.py
> $ sudo cp ./mythproto.py ./mythproto.org <http://mythproto.org/>
> $ sudo patch < /path/to/mythproto.patch
> $ sudo python3 -m compileall ./mythproto.py
> }}}
>
> Note, that this will be overwritten again when updating
> mythtv.
>
> Please report back the result.
> On success, I will update the Python Bindings asap.

I found earlier that I needed to add a ‘/‘ between the path and the file name to avoid the error below. I tried setting the path with a leading ‘/‘ to avoid the issue, but that didn’t work as the ‘/‘ was stripped off earlier.

FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/mythtv/recordings44303_20200811220000.mp4’

I tweaked the open statement to put in +’/‘ immediately after the sg.dirname to add the ‘/‘ separator and it works great!

If you can add the directory separator into the code somewhere, all will be good!

Thanks!

Jay

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

rcrernst at gmail

Aug 25, 2020, 1:04 PM

Post #5 of 9 (823 views)

Permalink

On Tue, Aug 25, 2020 at 4:49 PM Jay Harbeston <jharbestonus@gmail.com>
wrote:

>
>
> On Aug 25, 2020, at 6:18 AM, Roland Ernst <rcrernst@gmail.com> wrote:
>
>
> On Tue, Aug 25, 2020 at 11:48 AM Stephen Worthington <
> stephen_agent@jsw.gen.nz> wrote:
>
>> On Mon, 24 Aug 2020 13:25:27 -0400, you wrote:
>>
>> >
>> >Any ideas where I need to check? Or?
>> >
>>
>>
> This seems to be a leftover from the python3 conversion
> for mythtv v31+.
>
> If the storage groups of the recording and video is local,
> i.e.: on the same PC mythvidexport is running, please try
> the following:
>
> - locate the installed file mythproto.py from the Python
> Bindings (mine is in the location below)
>
> - in terminal, run the following commands
> (change the path to mythproto.patch)
>
> {{{
> $ cd /usr/lib/python3/dist-packages/MythTV/
> $ ls ./mythproto.py
> $ sudo cp ./mythproto.py ./mythproto.org
> $ sudo patch < /path/to/mythproto.patch
> $ sudo python3 -m compileall ./mythproto.py
> }}}
>
> Note, that this will be overwritten again when updating
> mythtv.
>
> Please report back the result.
> On success, I will update the Python Bindings asap.
>
>
>
> I found earlier that I needed to add a ‘/‘ between the path and the file
> name to avoid the error below. I tried setting the path with a leading ‘/‘
> to avoid the issue, but that didn’t work as the ‘/‘ was stripped off
> earlier.
>
> FileNotFoundError: [Errno 2] No such file or directory:
> '/var/lib/mythtv/recordings44303_20200811220000.mp4’
>
>
> I tweaked the open statement to put in +’/‘ immediately after the
> sg.dirname to add the ‘/‘ separator and it works great!
>
> If you can add the directory separator into the code somewhere, all will
> be good!
>
> Thanks!
>
> Jay
>
>
You are describing two different failures / bugs:

1) The trailing path separator is missing on some entries in the
storagegroup :
Please post the output of
{{{
MariaDB [mythconverg]> select * from storagegroup;
}}}
Until v.30, all storage group entries had a trailing slash ('/') in the
database.
My test setup for v31 shows a mixed scenario, with and without trailing
slashes.
Everything was set up by the 'mythtv-setup' program, no direct sql edits.
This would affect all mythtv versions, regardless if they use python2 or
python3.
Finding all occurrences of this missing separator inside the Python
Bindings could be a lot of work.
I do not know if php/mythweb and perl bindings are affected here, too.
The correct python fix will be something like
{{{
os.path.join(sg.dirname, filename)
}}}

2) Opening a file with the MythTV Python Bindings is not done in binary
mode.
This only affects python3 and mythtv v31+

Roland

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

jharbestonus at gmail

Aug 25, 2020, 1:47 PM

Post #6 of 9 (823 views)

Permalink

> On Aug 25, 2020, at 4:04 PM, Roland Ernst <rcrernst@gmail.com> wrote:
>
>
>
> On Tue, Aug 25, 2020 at 4:49 PM Jay Harbeston <jharbestonus@gmail.com <mailto:jharbestonus@gmail.com>> wrote:
>
>
>> On Aug 25, 2020, at 6:18 AM, Roland Ernst <rcrernst@gmail.com <mailto:rcrernst@gmail.com>> wrote:
>>
>>
>> On Tue, Aug 25, 2020 at 11:48 AM Stephen Worthington <stephen_agent@jsw.gen.nz <mailto:stephen_agent@jsw.gen.nz>> wrote:
>> On Mon, 24 Aug 2020 13:25:27 -0400, you wrote:
>>
>> >
>> >Any ideas where I need to check? Or?
>> >
>>
>>
>> This seems to be a leftover from the python3 conversion
>> for mythtv v31+.
>>
>> If the storage groups of the recording and video is local,
>> i.e.: on the same PC mythvidexport is running, please try
>> the following:
>>
>> - locate the installed file mythproto.py from the Python
>> Bindings (mine is in the location below)
>>
>> - in terminal, run the following commands
>> (change the path to mythproto.patch)
>>
>> {{{
>> $ cd /usr/lib/python3/dist-packages/MythTV/
>> $ ls ./mythproto.py
>> $ sudo cp ./mythproto.py ./mythproto.org <http://mythproto.org/>
>> $ sudo patch < /path/to/mythproto.patch
>> $ sudo python3 -m compileall ./mythproto.py
>> }}}
>>
>> Note, that this will be overwritten again when updating
>> mythtv.
>>
>> Please report back the result.
>> On success, I will update the Python Bindings asap.
>
>
> I found earlier that I needed to add a ‘/‘ between the path and the file name to avoid the error below. I tried setting the path with a leading ‘/‘ to avoid the issue, but that didn’t work as the ‘/‘ was stripped off earlier.
>
> FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/mythtv/recordings44303_20200811220000.mp4’
>
>
> I tweaked the open statement to put in +’/‘ immediately after the sg.dirname to add the ‘/‘ separator and it works great!
>
> If you can add the directory separator into the code somewhere, all will be good!
>
> Thanks!
>
> Jay
>
>
> You are describing two different failures / bugs:
>
> 1) The trailing path separator is missing on some entries in the storagegroup :
> Please post the output of
> {{{
> MariaDB [mythconverg]> select * from storagegroup;
> }}}

+----+-------------+----------+------------------------------+
| id | groupname | hostname | dirname |
+----+-------------+----------+------------------------------+
| 1 | Default | mythpc | /var/lib/mythtv/recordings |
| 3 | Fanart | mythpc | /var/lib/mythtv/fanart/ |
| 4 | Trailers | mythpc | /var/lib/mythtv/trailers/ |
| 5 | Coverart | mythpc | /var/lib/mythtv/coverart/ |
| 7 | Screenshots | mythpc | /var/lib/mythtv/screenshots/ |
| 8 | Banners | mythpc | /var/lib/mythtv/banners/ |
| 9 | DB Backups | mythpc | /var/lib/mythtv/db_backups/ |
| 10 | LiveTV | mythpc | /var/lib/mythtv/livetv/ |
| 11 | Streaming | mythpc | /var/lib/mythtv/streaming/ |
| 19 | Videos | mythpc | /var/lib/mythtv/videos/ |
+----+-------------+----------+------------------------------+
10 rows in set (0.00 sec)

> Until v.30, all storage group entries had a trailing slash ('/') in the database.
> My test setup for v31 shows a mixed scenario, with and without trailing slashes.
> Everything was set up by the 'mythtv-setup' program, no direct sql edits.
> This would affect all mythtv versions, regardless if they use python2 or python3.
> Finding all occurrences of this missing separator inside the Python Bindings could be a lot of work.
> I do not know if php/mythweb and perl bindings are affected here, too.
> The correct python fix will be something like
> {{{
> os.path.join(sg.dirname, filename)
> }}}
>
> 2) Opening a file with the MythTV Python Bindings is not done in binary mode.
> This only affects python3 and mythtv v31+
>

I can see now that the path needs to be updated for the default, and once I update the default path,
the current code with the mods for the binary, but without the +’/‘ should be fine.

Thanks!

Jay

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

rcrernst at gmail

Aug 25, 2020, 2:03 PM

Post #7 of 9 (823 views)

Permalink

On Tue, Aug 25, 2020 at 10:49 PM Jay Harbeston <jharbestonus@gmail.com>
wrote:

>
>
> On Aug 25, 2020, at 4:04 PM, Roland Ernst <rcrernst@gmail.com> wrote:
>
>
>
> On Tue, Aug 25, 2020 at 4:49 PM Jay Harbeston <jharbestonus@gmail.com>
> wrote:
>
>>
>>
>> On Aug 25, 2020, at 6:18 AM, Roland Ernst <rcrernst@gmail.com> wrote:
>>
>>
>> On Tue, Aug 25, 2020 at 11:48 AM Stephen Worthington <
>> stephen_agent@jsw.gen.nz> wrote:
>>
>>> On Mon, 24 Aug 2020 13:25:27 -0400, you wrote:
>>>
>>> >
>>> >Any ideas where I need to check? Or?
>>> >
>>>
>>>
>> This seems to be a leftover from the python3 conversion
>> for mythtv v31+.
>>
>> If the storage groups of the recording and video is local,
>> i.e.: on the same PC mythvidexport is running, please try
>> the following:
>>
>> - locate the installed file mythproto.py from the Python
>> Bindings (mine is in the location below)
>>
>> - in terminal, run the following commands
>> (change the path to mythproto.patch)
>>
>> {{{
>> $ cd /usr/lib/python3/dist-packages/MythTV/
>> $ ls ./mythproto.py
>> $ sudo cp ./mythproto.py ./mythproto.org
>> $ sudo patch < /path/to/mythproto.patch
>> $ sudo python3 -m compileall ./mythproto.py
>> }}}
>>
>> Note, that this will be overwritten again when updating
>> mythtv.
>>
>> Please report back the result.
>> On success, I will update the Python Bindings asap.
>>
>>
>>
>> I found earlier that I needed to add a ‘/‘ between the path and the file
>> name to avoid the error below. I tried setting the path with a leading ‘/‘
>> to avoid the issue, but that didn’t work as the ‘/‘ was stripped off
>> earlier.
>>
>> FileNotFoundError: [Errno 2] No such file or directory:
>> '/var/lib/mythtv/recordings44303_20200811220000.mp4’
>>
>>
>> I tweaked the open statement to put in +’/‘ immediately after the
>> sg.dirname to add the ‘/‘ separator and it works great!
>>
>> If you can add the directory separator into the code somewhere, all will
>> be good!
>>
>> Thanks!
>>
>> Jay
>>
>>
> You are describing two different failures / bugs:
>
> 1) The trailing path separator is missing on some entries in the
> storagegroup :
> Please post the output of
> {{{
> MariaDB [mythconverg]> select * from storagegroup;
> }}}
>
>
> mysql> select * from storagegroup;
> +----+-------------+----------+------------------------------+
> | id | groupname | hostname | dirname |
> +----+-------------+----------+------------------------------+
> | 1 | Default | mythpc | /var/lib/mythtv/recordings |
> | 3 | Fanart | mythpc | /var/lib/mythtv/fanart/ |
> | 4 | Trailers | mythpc | /var/lib/mythtv/trailers/ |
> | 5 | Coverart | mythpc | /var/lib/mythtv/coverart/ |
> | 7 | Screenshots | mythpc | /var/lib/mythtv/screenshots/ |
> | 8 | Banners | mythpc | /var/lib/mythtv/banners/ |
> | 9 | DB Backups | mythpc | /var/lib/mythtv/db_backups/ |
> | 10 | LiveTV | mythpc | /var/lib/mythtv/livetv/ |
> | 11 | Streaming | mythpc | /var/lib/mythtv/streaming/ |
> | 19 | Videos | mythpc | /var/lib/mythtv/videos/ |
> +----+-------------+----------+------------------------------+
> 10 rows in set (0.00 sec)
>
> mysql>
>
>
> Until v.30, all storage group entries had a trailing slash ('/') in the
> database.
> My test setup for v31 shows a mixed scenario, with and without trailing
> slashes.
> Everything was set up by the 'mythtv-setup' program, no direct sql edits.
> This would affect all mythtv versions, regardless if they use python2 or
> python3.
> Finding all occurrences of this missing separator inside the Python
> Bindings could be a lot of work.
> I do not know if php/mythweb and perl bindings are affected here, too.
> The correct python fix will be something like
> {{{
> os.path.join(sg.dirname, filename)
> }}}
>
> 2) Opening a file with the MythTV Python Bindings is not done in binary
> mode.
> This only affects python3 and mythtv v31+
>
>
>
> I can see now that the path needs to be updated for the default, and once
> I update the default path,
> the current code with the mods for the binary, but without the +’/‘
> should be fine.
>
> Thanks!
>
> Jay
>
>
Jay,

I have a mixed set of paths in my storage group table as well, and I did a
fresh install.
So I need to investigate this further.

Anyway, I've created a patch that should work with and without trailing
slashes.

Could you please try this patch?

Thx
Roland

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

jharbestonus at gmail

Aug 26, 2020, 4:56 AM

Post #8 of 9 (807 views)

Permalink

>
> Jay,
>
> I have a mixed set of paths in my storage group table as well, and I did a fresh install.
> So I need to investigate this further.
>
> Anyway, I've created a patch that should work with and without trailing slashes.
>
> Could you please try this patch?

I have applied the patch; removed the trailing ‘/‘ on the default recordings setting to restore to original state; and run mythvidexport.py successfully.

The v2 patch is a good fix!

Thanks much!

Jay

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Re: Errors from myhproto when using mythvidexport.py [ In reply to ]

rcrernst at gmail

Aug 27, 2020, 2:51 PM

Post #9 of 9 (784 views)

Permalink

On Wed, Aug 26, 2020 at 1:58 PM Jay Harbeston <jharbestonus@gmail.com>
wrote:

> >
> > Jay,
> >
> > I have a mixed set of paths in my storage group table as well, and I did
> a fresh install.
> > So I need to investigate this further.
> >
> > Anyway, I've created a patch that should work with and without trailing
> slashes.
> >
> > Could you please try this patch?
>
>
> I have applied the patch; removed the trailing ‘/‘ on the default
> recordings setting to restore to original state; and run mythvidexport.py
> successfully.
>
> The v2 patch is a good fix!
>
> Thanks much!
>
> Jay
>
>
Hi
I have pushed this fix now to fixes/31 as well.
Once finished with the mythvidexport.py conversion to python3, could you
please share it?
The MythTV wiki has a section about mythvidexport.py and should be expanded
with a version for v31+.

Regards,
Roland

Mailing List Archive

Attached Files:

Attached Files: