Mailing List Archive

MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR)
Hi folks,

I've finally gotten around to fixing mythburn.py in 0.27 to support
cutlists on HD-PVR recordings. My prior patch
<http://www.mythtv.org/pipermail/mythtv-users/2012-October/341879.html>
did an extra (redundant) pass of transcoding to MPEG2 before proceeding,
but it seems that this isn't required anymore. But commercial cutting
still didn't work right out of the box. If you want, go right to the
patch. If you're interested about what it does and how I got there,
read on...

I had to change how the cutlist was obtained, as it didn't seem to get
it right off the bat. It now calls mythutil --getcutlist to get the
cutlist and parse it for passing to the mythtranscode commandline.

The first problem I ran into was that mythtranscode would fail if the
cutlist excluded any part of the beginning of the recording. In my
short example, if mythburn.py did:
mythtranscode --mpeg2 --infile
"/var/lib/mytharchive/temp/work/1/2251_20140302230641.mpg" --outfile
"/var/lib/mytharchive/temp/work/1/newfile3.mpg" --honorcutlist "0-254
992-1209"

mythreplex would:
STARTING DEMUX
Video: aspect ratio: 16:9
starting with video PTS:
0:00:00.8938
Can't find all required streams
Please check if audio and video have standard IDs (0xc0 or 0xe0)

Depending on how many frames were cut from the beginning (e.g. 0-10
instead), I might end up with something like:
STARTING DEMUX
Video: aspect ratio: 16:9
starting with video PTS:
0:00:00.6269
starting audio PTS:
0:00:26.4240
Wrong audio frame size: 1438
ringbuffer overflow 250<2016 629145
video ring buffer overrun error

However, if I removed the cutlist from the beginning (0-254):
mythtranscode --mpeg2 --infile
"/var/lib/mytharchive/temp/work/1/2251_20140302230641.mpg" --outfile
"/var/lib/mytharchive/temp/work/1/newfile3.mpg" --honorcutlist "992-1209"

mythreplex would succeed. This seems like a bug, but I didn't want to
dig into mythreplex. So on to projectx.

Projectx couldn't find the recording details via:
rec = DB.searchRecorded(chanid=chanid, starttime=starttime).next()
The chanid and starttime variables seemed OK and matched what's in my
recorded database. But the above line fails with a StopIteration error.
So if finding the recording via the channel and time fails, I'm falling
back to finding it via the recording filename, which was already stored
in streaminfo_orig.xml.

Now projectx fails because the "-set ExternPanel.appendPidToFileName=1"
option is not valid with "recent" projectx versions. This also means
that the filenames produced by projectx aren't what mythburn.py is
expecting. There's also some ".mv2" extensions in there, which should
be ".m2v"

The last thing I noticed was that despite the hd-pvr recording in AC3
already (and potentially in 5.1), mythburn.py was re-encoding the audio
down to stereo. I made a custom encoding profile
<http://www.mythtv.org/wiki/MythArchive#How_do_I_add_my_own_encoding_parameters_to_the_encoding_profiles.3F>
where I made a copy of "HQ" and renamed it. And in the existing HQ
profile I changed the "-acodec" option to "copy" and removed the "-b:a"
and "-ac" options. Note that if not all of your recordings are in AC3,
you may have DVD compatibility issues. My older recordings are in AAC
because of this bug: <https://code.mythtv.org/trac/ticket/2077>
mythburn.py could probably be modified to use lossless conversion only
when AC3 is detected, but I didn't feel like getting into that.

Finally, I ended up swapping out occurrences of "mythffmpeg" with
"ffmpeg". mythffmpeg hung up on an ATSC recording, but the ffmpeg that
comes with Mythbuntu 12.04 seems to work fine. I suppose changing the
mythffmpeg symlink would suffice, but I wanted to keep all modifications
to just mythburn.py.

That's about it. The patch is a bit hacky, but it works for me.


-WD
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 3/3/14, 9:27 PM, Will Dormann wrote:
> My prior patch
> <http://www.mythtv.org/pipermail/mythtv-users/2012-October/341879.html>
> did an extra (redundant) pass of transcoding to MPEG2 before proceeding,
> but it seems that this isn't required anymore.


Just to clarify the above statement:
It seems that the extra transcoding pass is indeed required if projectx
isn't used (and mythtranscode is run in its place). So it's not that
the extra pass isn't required anymore because of a change to
mythburn.py. But rather that it isn't required when projectx is used.

Because the non-projectx stuff wasn't working quite right, I left the
extra mythtranscode pass (using "newfile3.mpg") out of the patch.

If anybody wants to investigate why the mythtranscode method doesn't
work (if there's a cutlist that removes the beginning of the file), feel
free to try the attached patch. Though since projectx works, I'm not
sure of the value of that work. What's the practical difference between
the projectx and mythtranscode methods? I'm guessing that the latter
is the default just to minimize the reliance on external 3rd party code?


-WD
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
> What's the practical difference between
> the projectx and mythtranscode methods? I'm guessing that the latter
> is the default just to minimize the reliance on external 3rd party code?
>
>
> -WD
> <mythburn27_mythtranscode.patch>
> _______________________________________________
I don't know mythburn.py at all but I know when I was looking at transcoding heavily a while ago one of the key differences between ProjectX and Mythtranscode was that Mythtranscode was frame accurate cutting and ProjectX was keyframe accurate cutting only. Depending on your recordings that might make a difference of a second or two at the cut boundaries (where you either lose a bit more than you intended or keep a bit more than you intended).

IIRC there was also some fancy audio processing (as a command line option) at the cut boundaries with Mythtranscode that could prevent a small audio chirp at the cut points.

But I'm just guessing, someone in the know will confirm.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 04/03/14 02:27, Will Dormann wrote:
<snip>
>
> Projectx couldn't find the recording details via:
> rec = DB.searchRecorded(chanid=chanid, starttime=starttime).next()
> The chanid and starttime variables seemed OK and matched what's in my
> recorded database. But the above line fails with a StopIteration error.
> So if finding the recording via the channel and time fails, I'm falling
> back to finding it via the recording filename, which was already stored
> in streaminfo_orig.xml.

I suspect a utc-related cause for this - but you've found a workaround.
Fine.
>
> Now projectx fails because the "-set ExternPanel.appendPidToFileName=1"
> option is not valid with "recent" projectx versions. This also means
> that the filenames produced by projectx aren't what mythburn.py is
> expecting.

I'm still using this with ProjectX 0.91.0.09/25.01.2014 TEST PROJECT
ONLY , User: John, - from a recent tarball, without obvious problems.

Java Environment
03 March 2014 23:36:41 GMT
java.version 1.7.0_51
java.vendor Oracle Corporation
java.home
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.4.5.0.fc19.x86_64/jre
java.vm.version 24.51-b03
java.vm.vendor Oracle Corporation
java.vm.name OpenJDK 64-Bit Server VM
java.class.vers 51.0
java.class.path /home/John/projectx_link
os.name Linux
os.arch amd64
os.version 3.13.5-101.fc19.x86_64


There's also some ".mv2" extensions in there, which should
> be ".m2v"

<snip>
>
> Finally, I ended up swapping out occurrences of "mythffmpeg" with
> "ffmpeg". mythffmpeg hung up on an ATSC recording, but the ffmpeg that
> comes with Mythbuntu 12.04 seems to work fine. I suppose changing the
> mythffmpeg symlink would suffice, but I wanted to keep all modifications
> to just mythburn.py.

Yes. I've found a few end-of-schedule recordings where mythffmpeg
doesn't identify the wanted streams, but I deal with this, and the
cutlist, outside mythburn.py
>
> That's about it. The patch is a bit hacky, but it works for me.
>
>
> -WD
>

I have a procedure and a version of mythburn.py that works for me - for
DVB-T, and as long as the calling parameters remain unchanged - so I may
not rush to try yours; but thanks for posting it.

Cheers,

John P



_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 3/4/14, 5:59 AM, John Pilkington wrote:
>
> I suspect a utc-related cause for this - but you've found a workaround.
> Fine.

Yeah, I read about some UTC issues, but I also thought that they were
addressed in recent builds. I also tried doing a query directly in
mysql for "chanid" and "starttime" values that are passed to
generateProjectXCutlist(), and it does show a recording for me. I didn't
dig into the internals of DB.searchRecorded(), though, which is why I
use the workaround.

The mytharchive package I'm using is (MythBuntu 12.04):
2:0.27.0+fixes.20140226.28b768a-0ubuntu0mythbuntu2


> I'm still using this with ProjectX 0.91.0.09/25.01.2014 TEST PROJECT
> ONLY , User: John, - from a recent tarball, without obvious problems.

OK, looking at mythburn.py I see that ProjectX 0.91 or later is
required. I suppose I also assumed that if Ubuntu packages mytharchive,
they would also package a projectx version that is required by it. I have:

ProjectX 0.90.4.00/30.03.2006 TEST PROJECT ONLY


> I have a procedure and a version of mythburn.py that works for me - for
> DVB-T, and as long as the calling parameters remain unchanged - so I may
> not rush to try yours; but thanks for posting it.

I don't blame you. It's likely that if I had a more recent (supported)
version of projectx, my patch would have been much more trivial. And
for the very same reason (what I have now works) I'm unlikely to muck
with it more. In hindsight, I should have been working with a supported
projectx version from the start, but what's done is done. :)


-WD



_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 3/4/14, 8:37 AM, Will Dormann wrote:
> OK, looking at mythburn.py I see that ProjectX 0.91 or later is
> required.

It looks like the MythTV documentation needs to be updated to reflect
this:
<http://www.mythtv.org/wiki/Installing_MythArchive_Dependencies#ProjectX>

I don't have a wiki account and have not gone through the steps of
installing or using 0.91, but maybe somebody who has can update the wiki
with correct instructions.


-WD

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 04/03/14 13:44, Will Dormann wrote:
> On 3/4/14, 8:37 AM, Will Dormann wrote:
>> OK, looking at mythburn.py I see that ProjectX 0.91 or later is
>> required.
>
> It looks like the MythTV documentation needs to be updated to reflect
> this:
> <http://www.mythtv.org/wiki/Installing_MythArchive_Dependencies#ProjectX>
>
> I don't have a wiki account and have not gone through the steps of
> installing or using 0.91, but maybe somebody who has can update the wiki
> with correct instructions.
>
>
> -WD
>

I don't recollect seeing that; it's full of detailed info that is way
out of date, and updating it would be a challenge.

I suspect that major distros will be largely ok except for the optional
m2vrequantiser and ProjectX. These work for me; I put the links in a
patch posted some time ago but they didn't get in.

https://launchpad.net/m2vrequantiser/trunk/1.1/+download/M2VRequantiser-v1.1.tar.gz

http://project-x.cvs.sourceforge.net/viewvc/project-x/Project-X/

Download GNU tarball from the bottom of the page.

HTH

John P


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
> On Tuesday, 4 March 2014, 15:25, John Pilkington <J.Pilk@tesco.net> wrote:
> > On 04/03/14 13:44, Will Dormann wrote:
>> On 3/4/14, 8:37 AM, Will Dormann wrote:
>>> OK, looking at mythburn.py I see that ProjectX 0.91 or later is
>>> required.
>>
>> It looks like the MythTV documentation needs to be updated to reflect
>> this:
>>
> <http://www.mythtv.org/wiki/Installing_MythArchive_Dependencies#ProjectX>
>>
>> I don't have a wiki account and have not gone through the steps of
>> installing or using 0.91, but maybe somebody who has can update the wiki
>> with correct instructions.
>>
>>
>> -WD
>>
>
> I don't recollect seeing that;  it's full of detailed info that is way
> out of date, and updating it would be a challenge.
>
> I suspect that major distros will be largely ok except for the optional
> m2vrequantiser and ProjectX.  These work for me;  I put the links in a
> patch posted some time ago but they didn't get in.
>
> https://launchpad.net/m2vrequantiser/trunk/1.1/+download/M2VRequantiser-v1.1.tar.gz
>
> http://project-x.cvs.sourceforge.net/viewvc/project-x/Project-X/
>
> Download GNU tarball from the bottom of the page.
>
> HTH
>
> John P
>
>

Has anybody experimented with piping the output from mythtranscode into mythffmpeg directly and setting ffmpeg to produce a DVD compatible output complete with NAV packets that in theory can be passed directly to dvdauthor.

If that works we can get rid of projectX, mythreplex and mplex altogether.

Paul H.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 05/03/14 07:28, Paul Harrison wrote:

>
> Has anybody experimented with piping the output from mythtranscode into mythffmpeg directly and setting ffmpeg to produce a DVD compatible output complete with NAV packets that in theory can be passed directly to dvdauthor.
>
> If that works we can get rid of projectX, mythreplex and mplex altogether.
>
> Paul H.
>

I think Paul Gardiner has been an advocate of piping with mythtranscode.
And I believe the current MythArchive pal-dvd/ntsc-dvd profiles
(without copyts?) have nav packets.

While I understand the wish to remove dependencies I wouldn't want to
abandon ProjectX. Over the several years that I have been using it (and
my workflow uses it twice, doing the cuts in mythDVBcut) the failure
rate has been very, very small. Before that I would get occasional
failures, probably from recordings with minor faults, and there seemed
to be no way to recover. Building up the same level of confidence in a
new scheme would be a long process that I have no real wish to start.
As far as I'm concerned, it ain't broke...

Cheers,

John P


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 3/4/14, 5:59 AM, John Pilkington wrote:
> On 04/03/14 02:27, Will Dormann wrote:
> <snip>
>>
>> Projectx couldn't find the recording details via:
>> rec = DB.searchRecorded(chanid=chanid, starttime=starttime).next()
>> The chanid and starttime variables seemed OK and matched what's in my
>> recorded database. But the above line fails with a StopIteration error.
>> So if finding the recording via the channel and time fails, I'm falling
>> back to finding it via the recording filename, which was already stored
>> in streaminfo_orig.xml.
>
> I suspect a utc-related cause for this - but you've found a workaround.
> Fine.



Hey John,

In setting up my new Mythbuntu 14.04 system, I figured I'd try using the
stock mythburn.py script with ProjectX 0.91 (installed from zip...
Ubuntu 14.04 still provides 0.90). It exhibits the same problem that I
originally encountered above. I applied only that part of my patch and
nothing else, and mythburn worked perfectly! Commercial cuts and all.

Do you happen to know what utc-related issue might be causing this?
Sure, I have a workaround. But I'd like to be able to use a stock
mythburn.py so that I won't have to worry about re-patching it when
MythTV is updated. Possibly related is that I noticed that my
scheduled mythfilldatabase run time (via mythtv backend setup) is off by
my time zone's offset from UTC. But show recording times and
everything else I can see seems to be fine.



Thanks
-WD

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 08/05/14 01:10, Will Dormann wrote:
> On 3/4/14, 5:59 AM, John Pilkington wrote:
>> On 04/03/14 02:27, Will Dormann wrote:
>> <snip>
>>>
>>> Projectx couldn't find the recording details via:
>>> rec = DB.searchRecorded(chanid=chanid, starttime=starttime).next()
>>> The chanid and starttime variables seemed OK and matched what's in my
>>> recorded database. But the above line fails with a StopIteration error.
>>> So if finding the recording via the channel and time fails, I'm falling
>>> back to finding it via the recording filename, which was already stored
>>> in streaminfo_orig.xml.
>>
>> I suspect a utc-related cause for this - but you've found a workaround.
>> Fine.
>
>
>
> Hey John,
>
> In setting up my new Mythbuntu 14.04 system, I figured I'd try using the
> stock mythburn.py script with ProjectX 0.91 (installed from zip...
> Ubuntu 14.04 still provides 0.90). It exhibits the same problem that I
> originally encountered above. I applied only that part of my patch and
> nothing else, and mythburn worked perfectly! Commercial cuts and all.
>
> Do you happen to know what utc-related issue might be causing this?
> Sure, I have a workaround. But I'd like to be able to use a stock
> mythburn.py so that I won't have to worry about re-patching it when
> MythTV is updated. Possibly related is that I noticed that my
> scheduled mythfilldatabase run time (via mythtv backend setup) is off by
> my time zone's offset from UTC. But show recording times and
> everything else I can see seems to be fine.
>
>
>
> Thanks
> -WD

Hi Will,

I looked briefly at the patch you posted at the start of this thread,
but I wasn't sure I could cleanly pick out the bits you just applied. I
thought the UTC problem had been fixed for recent versions - see

https://code.mythtv.org/trac/ticket/11758

which reveals several shots at it; but I don't recollect ever having
trouble with it myself. I think my path through the maze avoids those
parts. I'm still not running the stock mythburn.py either - mainly mp2
and .iso-location stuff and a few extra diagnostics - and I'm not sure
that my last compare-and-resync exercise ever got completed. Your
mythfilldatabase offset does suggest that your current problem may not
be specific to MythArchive, though.

Cheers,

John


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://www.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: MythArchive 0.27 mythburn.py patch for cutlist support (HD-PVR) [ In reply to ]
On 5/8/14, 5:22 AM, John Pilkington wrote:
> I looked briefly at the patch you posted at the start of this thread,
> but I wasn't sure I could cleanly pick out the bits you just applied. I
> thought the UTC problem had been fixed for recent versions - see
>
> https://code.mythtv.org/trac/ticket/11758
>


Hi John,

I realized that the other patch that needs to happen is the one that
recalculates the post-custlist running length, too. That's required if
you make a DVD that uses a menu.

Attached is the patch that I'm currently using with success. When I
first looked at this problem, I recall manually searching the database
for the recording based on the time that mythburn.py was using, and it
*did* find the recording. Plus my mythburn.py has the patch in the
ticket above.

Given that both aspects need to be patched (and everything else seems to
be working fine), I'm slightly less motivated to figure out what the
first issue is. :-/


-WD