Mailing List Archive

Repairing corrupt recordings with dd
A while back (few weeks?) someone posted info about a technique to strip
the first few megabytes from the beginning of a recording using dd.  It
involved a discussion about the new video rendering system crashing the
frontend (or casing decoder errors) on recordings where the format
changes soon after the recording starts.  I swear I saved that message
somewhere because I have a lot of those (thanks to Comcast and my
firewire recordings) but I'll be dipped if I can find it.  I've searched
the archives, too, with no luck.  I could probably figure it our from
the dd man page, but thought I'd ask for a re-post of that info since
it's a tried and tested method.  Thanks,

Dave D.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
DaveD <mythtv@guiplot.com> wrote:
>
> A while back (few weeks?) someone posted info about a technique to strip the first few megabytes from the beginning of a recording using dd. It involved a discussion about the new video rendering system crashing the frontend (or casing decoder errors) on recordings where the format changes soon after the recording starts. I swear I saved that message somewhere because I have a lot of those (thanks to Comcast and my firewire recordings) but I'll be dipped if I can find it. I've searched the archives, too, with no luck. I could probably figure it our from the dd man page, but thought I'd ask for a re-post of that info since it's a tried and tested method.

Can't help with specifics, but the general syntax for dd would be :

dd if=<input file name> of=<output file name> bs=<block size> skip=<blocks to skip on input> count=<blocks to copy>

Input & output file names are self explanatory. If omitted, defaults to stdin and stdout respectively.

Block size would be some convenient value such as 1M (or 1m, or 1024k - it seems different implementations differ in what they'll accept)

Skip is the key thing if you are wanting to trim the beginning of the file. You'd need to guesstimate how many blocks to skip, and possibly test it if it's a large file by only copying a small amount to see if you've reached the right point. As a rough starting guestimate, consider the length of your pre-roll time as a fraction of the entire recording, and skip a similar fraction of the recorded file size.

And count allows you to limit how much data you copy. Normally you'd want to the end of the file (just omit count altogether), but if finding the right cut point by trial and error, you might only want to copy a short bit of your video initially to save time waiting for disk I/O.


One word of warning ... dd is a powerful tool and comes with no safeties installed. It'll do what you ask it to do regardless of what you might have intended to type, and as long as system permissions allow then it'll happen - even if that means nuking an entire disk in double-short time. Very short is that interval between hitting enter and uttering "oh ****" as you realise that you'd got something wrong.

As an aside, if you ever have a failing disk, then ddrescue does a similar job to dd - but with added smarts specifically for dealing with error laced inputs.

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
On Tue, 23 Jun 2020 16:30:14 +0100, you wrote:

>One word of warning ... dd is a powerful tool and comes with no safeties installed. It'll do what you ask it to do regardless of what you might have intended to type, and as long as system permissions allow then it'll happen - even if that means nuking an entire disk in double-short time. Very short is that interval between hitting enter and uttering "oh ****" as you realise that you'd got something wrong.

Yes, the best policy with dd is to write a script to do the command,
and pass parameters to the script. When you have written the script,
sit back and read it carefully to make sure it is safe. Preferably a
couple of hours after you wrote it, so you will be much more likely to
spot any problems. Doing dd commands directly from the command line
allows a small typo to wipe out hard drives.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
On 23/06/2020 15:21, DaveD wrote:
> A while back (few weeks?) someone posted info about a technique to strip
> the first few megabytes from the beginning of a recording using dd.  It
> involved a discussion about the new video rendering system crashing the
> frontend (or casing decoder errors) on recordings where the format
> changes soon after the recording starts.  I swear I saved that message
> somewhere because I have a lot of those (thanks to Comcast and my
> firewire recordings) but I'll be dipped if I can find it.  I've searched
> the archives, too, with no luck.  I could probably figure it our from
> the dd man page, but thought I'd ask for a re-post of that info since
> it's a tried and tested method.  Thanks,
>
> Dave D.
>
I think you might be remembering this:

https://code.mythtv.org/trac/ticket/13557#comment:13

which looked for an appropriate skip value. applied it, and then rebuilt
the seektable. The output file doesn't start at a keyframe, but that
doesn't seem to matter.

John P
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
On 6/23/20 3:09 PM, John Pilkington wrote:
> On 23/06/2020 15:21, DaveD wrote:
>> A while back (few weeks?) someone posted info about a technique to
>> strip the first few megabytes from the beginning of a recording using
>> dd.  It involved a discussion about the new video rendering system
>> crashing the frontend (or casing decoder errors) on recordings where
>> the format changes soon after the recording starts.  I swear I saved
>> that message somewhere because I have a lot of those (thanks to
>> Comcast and my firewire recordings) but I'll be dipped if I can find
>> it. I've searched the archives, too, with no luck.  I could probably
>> figure it our from the dd man page, but thought I'd ask for a re-post
>> of that info since it's a tried and tested method. Thanks,
>>
>> Dave D.
>>
> I think you might be remembering this:
>
> https://code.mythtv.org/trac/ticket/13557#comment:13
>
> which looked for an appropriate skip value. applied it, and then
> rebuilt the seektable.  The output file doesn't start at a keyframe,
> but that doesn't seem to matter.
>
> John P

That's it!  No wonder I couldn't find it in my old emails.  It was in my
bookmarks.  Thanks you!

Dave D.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
DaveD <mythtv@guiplot.com> writes:

> On 6/23/20 3:09 PM, John Pilkington wrote:
>> On 23/06/2020 15:21, DaveD wrote:
>>> A while back (few weeks?) someone posted info about a technique to
>>> strip the first few megabytes from the beginning of a recording
>>> using dd.  It involved a discussion about the new video rendering
>>> system crashing the frontend (or casing decoder errors) on
>>> recordings where the format changes soon after the recording
>>> starts.  I swear I saved that message somewhere because I have a
>>> lot of those (thanks to Comcast and my firewire recordings) but
>>> I'll be dipped if I can find it. I've searched the archives, too,
>>> with no luck.  I could probably figure it our from the dd man page,
>>> but thought I'd ask for a re-post of that info since it's a tried
>>> and tested method. Thanks,
>>>
>>> Dave D.
>>>
>> I think you might be remembering this:
>>
>> https://code.mythtv.org/trac/ticket/13557#comment:13
>>
>> which looked for an appropriate skip value. applied it, and then
>> rebuilt the seektable.  The output file doesn't start at a keyframe,
>> but that doesn't seem to matter.
>>
>> John P
>
> That's it!  No wonder I couldn't find it in my old emails.  It was in
> my bookmarks.  Thanks you!

FWIW, I would recommend using mythffmpeg or ffmpeg rather than dd. The
former tools present no significant risk to your system.

ffmpeg -ss 00:00:30 -i input.ts -codec copy -map 0 output.ts

will seek (-ss) to the 30 second point in input.ts and copy each stream
in the input to the output. The copy codec is very fast and the safety
is worth the few extra seconds.

I will guess that files with a mangled stream structure will cause
ffmpeg to error out, but I don't have a sample to check that guess on.
In either event, you can increase the seek time to try to find the
beginning of a stable stream structure.

Leo
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
On 6/25/20 7:29 PM, Leo Butler wrote:
> DaveD <mythtv@guiplot.com> writes:
>
>> On 6/23/20 3:09 PM, John Pilkington wrote:
>>> On 23/06/2020 15:21, DaveD wrote:
>>>> A while back (few weeks?) someone posted info about a technique to
>>>> strip the first few megabytes from the beginning of a recording
>>>> using dd.  It involved a discussion about the new video rendering
>>>> system crashing the frontend (or casing decoder errors) on
>>>> recordings where the format changes soon after the recording
>>>> starts.  I swear I saved that message somewhere because I have a
>>>> lot of those (thanks to Comcast and my firewire recordings) but
>>>> I'll be dipped if I can find it. I've searched the archives, too,
>>>> with no luck.  I could probably figure it our from the dd man page,
>>>> but thought I'd ask for a re-post of that info since it's a tried
>>>> and tested method. Thanks,
>>>>
>>>> Dave D.
>>>>
>>> I think you might be remembering this:
>>>
>>> https://code.mythtv.org/trac/ticket/13557#comment:13
>>>
>>> which looked for an appropriate skip value. applied it, and then
>>> rebuilt the seektable.  The output file doesn't start at a keyframe,
>>> but that doesn't seem to matter.
>>>
>>> John P
>> That's it!  No wonder I couldn't find it in my old emails.  It was in
>> my bookmarks.  Thanks you!
> FWIW, I would recommend using mythffmpeg or ffmpeg rather than dd. The
> former tools present no significant risk to your system.
>
> ffmpeg -ss 00:00:30 -i input.ts -codec copy -map 0 output.ts
>
> will seek (-ss) to the 30 second point in input.ts and copy each stream
> in the input to the output. The copy codec is very fast and the safety
> is worth the few extra seconds.
>
> I will guess that files with a mangled stream structure will cause
> ffmpeg to error out, but I don't have a sample to check that guess on.
> In either event, you can increase the seek time to try to find the
> beginning of a stable stream structure.

Unfortunately I ran your command without looking up what -map 0 does and
it left me with only the audio.  Next time I'll try it without the -map
option (and leave a backup! duh!).  When it ran, it gave the same raft
of errors at first, just like ffprobe shows (not surprising), then it
created a nice, audio-only file.  Good to know if I want to extract the
audio from a recording some time.  :)

Dave D.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
On 26/06/2020 16:20, DaveD wrote:
> On 6/25/20 7:29 PM, Leo Butler wrote:
>> DaveD <mythtv@guiplot.com> writes:
>>
>>> On 6/23/20 3:09 PM, John Pilkington wrote:
>>>> On 23/06/2020 15:21, DaveD wrote:
>>>>> A while back (few weeks?) someone posted info about a technique to
>>>>> strip the first few megabytes from the beginning of a recording
>>>>> using dd.  It involved a discussion about the new video rendering
>>>>> system crashing the frontend (or casing decoder errors) on
>>>>> recordings where the format changes soon after the recording
>>>>> starts.  I swear I saved that message somewhere because I have a
>>>>> lot of those (thanks to Comcast and my firewire recordings) but
>>>>> I'll be dipped if I can find it. I've searched the archives, too,
>>>>> with no luck.  I could probably figure it our from the dd man page,
>>>>> but thought I'd ask for a re-post of that info since it's a tried
>>>>> and tested method. Thanks,
>>>>>
>>>>> Dave D.
>>>>>
>>>> I think you might be remembering this:
>>>>
>>>> https://code.mythtv.org/trac/ticket/13557#comment:13
>>>>
>>>> which looked for an appropriate skip value. applied it, and then
>>>> rebuilt the seektable.  The output file doesn't start at a keyframe,
>>>> but that doesn't seem to matter.
>>>>
>>>> John P
>>> That's it!  No wonder I couldn't find it in my old emails.  It was in
>>> my bookmarks.  Thanks you!
>> FWIW, I would recommend using mythffmpeg or ffmpeg rather than dd. The
>> former tools present no significant risk to your system.
>>
>> ffmpeg -ss 00:00:30 -i input.ts -codec copy -map 0 output.ts
>>
>> will seek (-ss) to the 30 second point in input.ts and copy each stream
>> in the input to the output. The copy codec is very fast and the safety
>> is worth the few extra seconds.
>>
>> I will guess that files with a mangled stream structure will cause
>> ffmpeg to error out, but I don't have a sample to check that guess on.
>> In either event, you can increase the seek time to try to find the
>> beginning of a stable stream structure.
>
> Unfortunately I ran your command without looking up what -map 0 does and
> it left me with only the audio.  Next time I'll try it without the -map
> option (and leave a backup! duh!).  When it ran, it gave the same raft
> of errors at first, just like ffprobe shows (not surprising), then it
> created a nice, audio-only file.  Good to know if I want to extract the
> audio from a recording some time.  :)
>
> Dave D.

Some time ago I spent longer than was reasonable trying to get ffmpeg to
cut h264 recordings; eventually it worked fairly well but only with a
pre-run to reject all but 1 video and 1 audio stream - and playback
needs to be working to establish the cutpoints.

I also have a small python script that does byte-accurate cuts of 'any'
file. I've just updated the posted version of that to python3. There
are probably more elegant ways of coding it, but it works. Usually I
use Project-X for cutting mpeg2 video content, but this is a calling
script created as routine during a recent run:

{{{
#!/bin/bash
mv '/mnt/dat2/RSG3/10005_20200625195800.ts'
'/mnt/dat2/RSG3/10005_20200625195800.ts.old'
ionice -c3 ~/pycut.py '/mnt/dat2/RSG3/10005_20200625195800.ts.old'
'/mnt/dat2/RSG3/10005_20200625195800.ts' 75221244 76677116 80142144
218640804 282500644 541396008 612756860 797066044 862314640 1054189132
1121304756 1364012756 1431629588 1568680460
}}}

All here: https://www.mythtv.org/wiki/MythDVBcut

Mostly pycut.py gets used now to cut DVB radio recordings before
extracting the audio.

John P

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
DaveD <mythtv@guiplot.com> writes:

> On 6/25/20 7:29 PM, Leo Butler wrote:
>> DaveD <mythtv@guiplot.com> writes:
>>
>>> On 6/23/20 3:09 PM, John Pilkington wrote:
>>>> On 23/06/2020 15:21, DaveD wrote:
>>>>> A while back (few weeks?) someone posted info about a technique to
>>>>> strip the first few megabytes from the beginning of a recording
>>>>> using dd.  It involved a discussion about the new video rendering
>>>>> system crashing the frontend (or casing decoder errors) on
>>>>> recordings where the format changes soon after the recording
>>>>> starts.  I swear I saved that message somewhere because I have a
>>>>> lot of those (thanks to Comcast and my firewire recordings) but
>>>>> I'll be dipped if I can find it. I've searched the archives, too,
>>>>> with no luck.  I could probably figure it our from the dd man page,
>>>>> but thought I'd ask for a re-post of that info since it's a tried
>>>>> and tested method. Thanks,
>>>>>
>>>>> Dave D.
>>>>>
>>>> I think you might be remembering this:
>>>>
>>>> https://code.mythtv.org/trac/ticket/13557#comment:13
>>>>
>>>> which looked for an appropriate skip value. applied it, and then
>>>> rebuilt the seektable.  The output file doesn't start at a keyframe,
>>>> but that doesn't seem to matter.
>>>>
>>>> John P
>>> That's it!  No wonder I couldn't find it in my old emails.  It was in
>>> my bookmarks.  Thanks you!
>> FWIW, I would recommend using mythffmpeg or ffmpeg rather than dd. The
>> former tools present no significant risk to your system.
>>
>> ffmpeg -ss 00:00:30 -i input.ts -codec copy -map 0 output.ts
>>
>> will seek (-ss) to the 30 second point in input.ts and copy each stream
>> in the input to the output. The copy codec is very fast and the safety
>> is worth the few extra seconds.
>>
>> I will guess that files with a mangled stream structure will cause
>> ffmpeg to error out, but I don't have a sample to check that guess on.
>> In either event, you can increase the seek time to try to find the
>> beginning of a stable stream structure.
>
> Unfortunately I ran your command without looking up what -map 0 does
> and it left me with only the audio.  Next time I'll try it without the
> -map option (and leave a backup! duh!).  When it ran, it gave the same
> raft of errors at first, just like ffprobe shows (not surprising),
> then it created a nice, audio-only file.  Good to know if I want to
> extract the audio from a recording some time.  :)

Re: -map 0

see: https://ffmpeg.org/ffmpeg-all.html#toc-Manual-stream-selection

That selects all streams in input 0 to be mapped to the output,
regardless of ffmpeg's defaults. You will be deceived if you try it to
extract audio only (you would use -vn -map 0:a for that).

Your result indicates ffmpeg couldn't find a stable stream other than
audio. My suggestion was to increase the seek time to find a stable
video stream. Then the command above will work fine.

You can also use substitute ffplay in place of ffmpeg, viz.

ffplay -ss 00:00:30 -i input.ts

This will output info similar to ffmpeg and try to play the file, too.

Could you put a sample of such a file somewhere? It should be long enough
to include all the instability at the beginning and then a 1-2 stretch
with a stable stream structure.

Thanks,
Leo
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
Leo Butler <leo.butler81@googlemail.com> writes:

> DaveD <mythtv@guiplot.com> writes:
>
>> On 6/25/20 7:29 PM, Leo Butler wrote:
>>> DaveD <mythtv@guiplot.com> writes:
>>>
>>>> On 6/23/20 3:09 PM, John Pilkington wrote:
>>>>> On 23/06/2020 15:21, DaveD wrote:
>>>>>> A while back (few weeks?) someone posted info about a technique to
>>>>>> strip the first few megabytes from the beginning of a recording
>>>>>> using dd.  It involved a discussion about the new video rendering
>>>>>> system crashing the frontend (or casing decoder errors) on
>>>>>> recordings where the format changes soon after the recording
>>>>>> starts.  I swear I saved that message somewhere because I have a
>>>>>> lot of those (thanks to Comcast and my firewire recordings) but
>>>>>> I'll be dipped if I can find it. I've searched the archives, too,
>>>>>> with no luck.  I could probably figure it our from the dd man page,
>>>>>> but thought I'd ask for a re-post of that info since it's a tried
>>>>>> and tested method. Thanks,
>>>>>>
>>>>>> Dave D.
>>>>>>
>>>>> I think you might be remembering this:
>>>>>
>>>>> https://code.mythtv.org/trac/ticket/13557#comment:13
>>>>>
>>>>> which looked for an appropriate skip value. applied it, and then
>>>>> rebuilt the seektable.  The output file doesn't start at a keyframe,
>>>>> but that doesn't seem to matter.
>>>>>
>>>>> John P
>>>> That's it!  No wonder I couldn't find it in my old emails.  It was in
>>>> my bookmarks.  Thanks you!
>>> FWIW, I would recommend using mythffmpeg or ffmpeg rather than dd. The
>>> former tools present no significant risk to your system.
>>>
>>> ffmpeg -ss 00:00:30 -i input.ts -codec copy -map 0 output.ts
>>>
>>> will seek (-ss) to the 30 second point in input.ts and copy each stream
>>> in the input to the output. The copy codec is very fast and the safety
>>> is worth the few extra seconds.
>>>
>>> I will guess that files with a mangled stream structure will cause
>>> ffmpeg to error out, but I don't have a sample to check that guess on.
>>> In either event, you can increase the seek time to try to find the
>>> beginning of a stable stream structure.
>>
>> Unfortunately I ran your command without looking up what -map 0 does
>> and it left me with only the audio.  Next time I'll try it without the
>> -map option (and leave a backup! duh!).  When it ran, it gave the same
>> raft of errors at first, just like ffprobe shows (not surprising),
>> then it created a nice, audio-only file.  Good to know if I want to
>> extract the audio from a recording some time.  :)
>
> Re: -map 0
>
> see: https://ffmpeg.org/ffmpeg-all.html#toc-Manual-stream-selection
>
> That selects all streams in input 0 to be mapped to the output,
> regardless of ffmpeg's defaults. You will be deceived if you try it to
> extract audio only (you would use -vn -map 0:a for that).
>
> Your result indicates ffmpeg couldn't find a stable stream other than
> audio. My suggestion was to increase the seek time to find a stable
> video stream. Then the command above will work fine.

A second thought: have you actually run ffplay/ffprobe on the file you
created? Are you sure that it contains only an audio stream?

Console output would be helpful.

Here is another way to probe into the original file:

ffprobe -i input.ts -read_intervals '60%+#1' -show_frames

That seeks to 60 seconds and shows the frame info for 1 packet.

https://ffmpeg.org/ffprobe-all.html#toc-Main-options

Leo
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: Repairing corrupt recordings with dd [ In reply to ]
John Pilkington <johnpilk222@gmail.com> writes:

>
> Some time ago I spent longer than was reasonable trying to get ffmpeg
> to cut h264 recordings; eventually it worked fairly well but only with
> a pre-run to reject all but 1 video and 1 audio stream - and playback
> needs to be working to establish the cutpoints.

Strange. I use the ffmpeg concat filter to trim and transcode my
recordings to use an mkv container. The exact same script does
{.mpg,.ts,.mkv} -> .mkv without any fiddling of the options I pass to
ffmpeg and without having to drop any audio streams.

Of course, I use mythfrontend to set the cut points, so this is not
relevant to DaveD's problem.

Leo
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org