Mailing List Archive

Some video frames mangled on PVR-
I have a PVR-150 that's creating weird video. To start with, here's a
sample:

http://www.cc.gatech.edu/~bigpeteb/test.mpg

It seems to depend on the scene: some scenes display fine while others
have the glitch. I'm running through a set-top box so this is all
recorded on channel 3 on the card, regardless of what station I'm
watching.

This card worked fine under linux-2.6.18 kernel with the appropriate
IVTV drivers. Now it's in a new machine (uname -a: Linux dvr
2.6.31-gentoo-r6 #6 SMP Wed Jan 13 15:21:20 EST 2010 x86_64 Intel(R)
Celeron(R) CPU E3200 @ 2.40GHz GenuineIntel GNU/Linux) and it has this
problem.

My first guess is that it's a buffer issue, perhaps that the buffer is
too small. I can't find in the docs how to set the buffer size. IRQ
latency could be to blame, too (but it's the only device in the system
which has a meaningful latency setting, so that seems unlikely).

I'd appreciate any suggestions to try.
--
Peter Budny \
Georgia Tech \
CS PhD student \

_______________________________________________
ivtv-users mailing list
ivtv-users@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-users
Re: Some video frames mangled on PVR- [ In reply to ]
On Sun, 2010-01-24 at 20:46 -0500, Peter Budny wrote:
> I have a PVR-150 that's creating weird video. To start with, here's a
> sample:
>
> http://www.cc.gatech.edu/~bigpeteb/test.mpg
>
> It seems to depend on the scene: some scenes display fine while others
> have the glitch. I'm running through a set-top box so this is all
> recorded on channel 3 on the card, regardless of what station I'm
> watching.
>
> This card worked fine under linux-2.6.18 kernel with the appropriate
> IVTV drivers. Now it's in a new machine

So just to clarify, new hardware configuration as well as a new kernel
and driver version?


> (uname -a: Linux dvr
> 2.6.31-gentoo-r6 #6 SMP Wed Jan 13 15:21:20 EST 2010 x86_64 Intel(R)
> Celeron(R) CPU E3200 @ 2.40GHz GenuineIntel GNU/Linux) and it has this
> problem.
>
> My first guess is that it's a buffer issue, perhaps that the buffer is
> too small. I can't find in the docs how to set the buffer size.

I concur that the problem is a buffer issue: likely missed or dropped.

Do you get messages from these lines in your logs:

IVTV_WARN("All %s stream buffers are full. Dropping data.\n", s->name);
IVTV_WARN("Cause: the application is not reading fast enough.\n");

? Or maybe some other warnings from the ivtv driver?

Buffer size would not be my first guess. But 'modinfo ivtv' will show
you the options that have the driver set aside more (or less) memory for
buffers for various stream types.


> IRQ
> latency could be to blame, too (but it's the only device in the system
> which has a meaningful latency setting, so that seems unlikely).

IRQ latency and the PCI latency timer are two different things.

The PCI latency timer is the maximum number of PCI bus cycles, of
duration 1/(33 MHz), that a PCI bus master, like the CX23416, can have
the bus in a burst transfer before it has to yield back to the PCI bus
arbiter. It prevents PCI devices from hogging or hanging the PCI bus
which is a shared resource.

Tuning the PCI latency timers, on all the PCI bus devices in a system,
to provide optimal data throughput and latency for all devices in your
common use case, is actually a system level tuning issue. To do it
right would require modeling and simulation. (I always thought it kind
of odd of the ivtv driver to set the CX23416's latency timer to 64 by
default, as that can't be the right answer for every possible system,
but it must work well for most people.)


IRQ latency is the time from the IRQ line being raised by the CX23416
hardware to the time the ivtv IRQ handler services the interrupt and
acknowledges it. System level interactions affect that latency. Poor
quality linux drivers or buggy hardware can severely impact that
latency.


> I'd appreciate any suggestions to try.

When you modprobe the ivtv driver, set some of the debugging options, so
you can get a feel for what might be going wrong:

debug:Debug level (bitmask). Default: 0
1/0x0001: warning
2/0x0002: info
4/0x0004: mailbox
8/0x0008: ioctl
16/0x0010: file
32/0x0020: dma
64/0x0040: irq
128/0x0080: decoder
256/0x0100: yuv
512/0x0200: i2c
1024/0x0400: high volume

I would set the warning, info, mailbox, dma, irq, and high volume masks
so

# modprobe -r ivtv
# modprobe ivtv debug=0x467

High volume will make for many dmesg and log messages. You'll want to
make sure that /etc/rsyslog.conf has 'kern.*' messages logged to some
sensible file.



My first guess would be an IRQ service problem:

1. A device sharing the IRQ line with the CX23416 has a device driver
that sometimes errantly claims the CX23416's interrupt as it's own
(IRQ_HANDLED). Or maybe both devices generate a lot of interrupts and
hence some end up coincident in time.

2. A device sharing the IRQ line with the CX23416 has an irq handler
that sometimes takes a long time to execute and doesn't complete quickly
in that situation. (e.g. The ahci disk controller driver used to have
an error path in its irq handler that could take a really long time.)

'cat /proc/interrupts' will show you if any device is sharing the IRQ
line with the ivtv driver.

3. If this is a single processor machine, then any driver that keeps
IRQs disabled for too long due to a slow irq handler could cause poor
interrupt latency.




My next guess would be PCI bus loading:

Watching LiveTV with MythTV has at least three devices working that
present a non-trivial load on the I/O buses: a disk controller, a
graphics card, and the tv capture card.

'/sbin/lspci -tv' will show you the bus segment hierarchy. See if
there is an obvious bottleneck, such as a PCI bus segment shared with
the CX23416 and a disk controller behind a PCI-PCI bridge. Check the
latency timer settings of the PCI-PCI bridge.

You also might just want to record TV, without displaying at the same
time:

cat /dev/video0 > test.mpg
^C
mplayer test.mpg

and see if you still have the glitches. But then again, that may be how
you captured the one test clip to begin with.


My next guess would be some sort of DMA issue:

I'm not sure what to do here. The DMA process for the CX23416 is very
different from the CX23418 - I still need to come up to speed on it. I
know that the kernel's software IO TLB can affect performance since it
will use bounce buffers under certain conditions (but it shouldn't drop
buffers). Your system will say something about SWIOTLB or SW IOMMU in
the dmesg messages that happen at reboot, if it does.


My final guess would be a CX25843 digitizer VSYNC detection and locking
problem.

If this card has worked with this STB before, then this is highly
unlikely.


Regards,
Andy


_______________________________________________
ivtv-users mailing list
ivtv-users@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-users
Re: Some video frames mangled on PVR- [ In reply to ]
Andy Walls <awalls@radix.net> writes:

> On Sun, 2010-01-24 at 20:46 -0500, Peter Budny wrote:
>> I have a PVR-150 that's creating weird video. To start with, here's a
>> sample:
>>
>> http://www.cc.gatech.edu/~bigpeteb/test.mpg
>>
>> It seems to depend on the scene: some scenes display fine while others
>> have the glitch. I'm running through a set-top box so this is all
>> recorded on channel 3 on the card, regardless of what station I'm
>> watching.
>>
>> This card worked fine under linux-2.6.18 kernel with the appropriate
>> IVTV drivers. Now it's in a new machine
>
> So just to clarify, new hardware configuration as well as a new kernel
> and driver version?

Correct, new hardware configuration and new kernel+driver. But, on the
old machine I recall having similar glitches if I used anything newer
than 2.6.17 (I said .18 above, meant .17). At the time I just put the
old kernel and drivers back on and left it. I'd forgotten about it
until I put the card in a new machine.

>> (uname -a: Linux dvr
>> 2.6.31-gentoo-r6 #6 SMP Wed Jan 13 15:21:20 EST 2010 x86_64 Intel(R)
>> Celeron(R) CPU E3200 @ 2.40GHz GenuineIntel GNU/Linux) and it has this
>> problem.
>>
>> My first guess is that it's a buffer issue, perhaps that the buffer is
>> too small. I can't find in the docs how to set the buffer size.
>
> I concur that the problem is a buffer issue: likely missed or dropped.
>
> Do you get messages from these lines in your logs:
>
> IVTV_WARN("All %s stream buffers are full. Dropping data.\n", s->name);
> IVTV_WARN("Cause: the application is not reading fast enough.\n");
>
> ? Or maybe some other warnings from the ivtv driver?

I tried turning on some debugging, as you suggested, and it generated
only normal-seeming messages (a lot of DMA completed and IRQ completed,
nothing indicating errors).

After playing around with some parameters on the module, I see no
real effects. I tried doubling the size of all the encoding buffers,
but I still had the same broken video.

The only thing that seemed to be out of place was an occasional message
"ivtv0: warn: encoder VBI: Couldn't find start of buffer within the
first 256 bytes"

>> I'd appreciate any suggestions to try.
>
> My first guess would be an IRQ service problem:
>
> 1. A device sharing the IRQ line with the CX23416 has a device driver
> that sometimes errantly claims the CX23416's interrupt as it's own
> (IRQ_HANDLED). Or maybe both devices generate a lot of interrupts and
> hence some end up coincident in time.
>
> 2. A device sharing the IRQ line with the CX23416 has an irq handler
> that sometimes takes a long time to execute and doesn't complete quickly
> in that situation. (e.g. The ahci disk controller driver used to have
> an error path in its irq handler that could take a really long time.)
>
> 'cat /proc/interrupts' will show you if any device is sharing the IRQ
> line with the ivtv driver.

'cat /proc/interrupts' indicates, among other things

14: 1596863 1597175 IO-APIC-edge ata_piix
15: 0 0 IO-APIC-edge ata_piix
16: 6691638 6689226 IO-APIC-fasteoi uhci_hcd:usb5, HDA Intel, nvidia
19: 6894042 6894148 IO-APIC-fasteoi ata_piix, uhci_hcd:usb3, ivtv0

so the disk may be on the same IRQ as the IVTV card. (Probably so,
since it's a SATA disk.)

> My next guess would be PCI bus loading:
>
> Watching LiveTV with MythTV has at least three devices working that
> present a non-trivial load on the I/O buses: a disk controller, a
> graphics card, and the tv capture card.
>
> '/sbin/lspci -tv' will show you the bus segment hierarchy. See if
> there is an obvious bottleneck, such as a PCI bus segment shared with
> the CX23416 and a disk controller behind a PCI-PCI bridge. Check the
> latency timer settings of the PCI-PCI bridge.
>
> You also might just want to record TV, without displaying at the same
> time:
>
> cat /dev/video0 > test.mpg
> ^C
> mplayer test.mpg
>
> and see if you still have the glitches. But then again, that may be how
> you captured the one test clip to begin with.

This was indeed how I captured the test clip. Seems to make no
difference whether I use MythTV, or cat, or mplayer.

> My next guess would be some sort of DMA issue:
>
> I'm not sure what to do here. The DMA process for the CX23416 is very
> different from the CX23418 - I still need to come up to speed on it. I
> know that the kernel's software IO TLB can affect performance since it
> will use bounce buffers under certain conditions (but it shouldn't drop
> buffers). Your system will say something about SWIOTLB or SW IOMMU in
> the dmesg messages that happen at reboot, if it does.
>
>
> My final guess would be a CX25843 digitizer VSYNC detection and locking
> problem.
>
> If this card has worked with this STB before, then this is highly
> unlikely.

Well, I'm not sure how to go about determining which of these is the
problem. I have some kernel development experience from classes I've
taken, but nothing at this level. If you can walk me through some
things to try I'll see if they help, and if there's a bug hiding in the
driver maybe I can help track it down.

Thanks in advance for any help you can provide,
--
Peter Budny \
Georgia Tech \
CS PhD student \

_______________________________________________
ivtv-users mailing list
ivtv-users@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-users
Re: Some video frames mangled on PVR- [ In reply to ]
On Mon, 2010-01-25 at 01:39 -0500, Peter Budny wrote:
> Andy Walls <awalls@radix.net> writes:
>
> > On Sun, 2010-01-24 at 20:46 -0500, Peter Budny wrote:
> >> I have a PVR-150 that's creating weird video. To start with, here's a
> >> sample:
> >>
> >> http://www.cc.gatech.edu/~bigpeteb/test.mpg
> >>

>
> Or maybe some other warnings from the ivtv driver?
>
> I tried turning on some debugging, as you suggested, and it generated
> only normal-seeming messages (a lot of DMA completed and IRQ completed,
> nothing indicating errors).
>
> After playing around with some parameters on the module, I see no
> real effects. I tried doubling the size of all the encoding buffers,
> but I still had the same broken video.
>
> The only thing that seemed to be out of place was an occasional message
> "ivtv0: warn: encoder VBI: Couldn't find start of buffer within the
> first 256 bytes"

Hmmm. I'm not too concerned about the VBI stream, but it may be
indiciative of some IRQ or DMA problem.

This message is coming from
linux/driver/media/video/ivtv/ivtv-irq.c:dma_post().

Can you provide the verbose log to me? (an off list email is fine if it
is very large)

The ivtv buffer transfer code is somewhat complex. I'll have to take
some time this week and look at it closely to learn it myself.

> >> I'd appreciate any suggestions to try.
> >
> > My first guess would be an IRQ service problem:
> >
> > 1. A device sharing the IRQ line with the CX23416 has a device driver
> > that sometimes errantly claims the CX23416's interrupt as it's own
> > (IRQ_HANDLED). Or maybe both devices generate a lot of interrupts and
> > hence some end up coincident in time.
> >
> > 2. A device sharing the IRQ line with the CX23416 has an irq handler
> > that sometimes takes a long time to execute and doesn't complete quickly
> > in that situation. (e.g. The ahci disk controller driver used to have
> > an error path in its irq handler that could take a really long time.)
> >
> > 'cat /proc/interrupts' will show you if any device is sharing the IRQ
> > line with the ivtv driver.
>
> 'cat /proc/interrupts' indicates, among other things
>
> 14: 1596863 1597175 IO-APIC-edge ata_piix
> 15: 0 0 IO-APIC-edge ata_piix
> 16: 6691638 6689226 IO-APIC-fasteoi uhci_hcd:usb5, HDA Intel, nvidia
> 19: 6894042 6894148 IO-APIC-fasteoi ata_piix, uhci_hcd:usb3, ivtv0
>
> so the disk may be on the same IRQ as the IVTV card. (Probably so,
> since it's a SATA disk.)

So try to perform a capture and playback without writing to a disk and
also avoiding activity with the devices on usb3 (I think the bus listed
by /sbin/lsusb as Bus 003 corresponds to usb3):

$ ivtv-tune -d /dev/video0 -c 3
$ mplayer /dev/video0

I'm not hopeful for any improvement, but it's easy enough to test.



> Well, I'm not sure how to go about determining which of these is the
> problem. I have some kernel development experience from classes I've
> taken, but nothing at this level. If you can walk me through some
> things to try I'll see if they help, and if there's a bug hiding in the
> driver maybe I can help track it down.

I have no good advice yet. I know the CX23418 DMA process and driver
structure well, but I have to come up to speed on the CX23416 DMA
process before I can make any reasonable suggestions on what to do.

You are certainly welcome to look at the driver for bugs. If trying to
grok what the driver is doing WRT to data transfers from hardware, I
would suggest looking at these:

linux/Documentation/video4linux/cx2341x/fw-calling.txt
linux/Documentation/video4linux/cx2341x/fw-dma.txt
linux/Documentation/video4linux/cx2341x/fw-encoder-api.txt
linux/Documentation/video4linux/cx2341x/fw-memory.txt

linux/drivers/media/video/ivtv/ivtv-fileops.c:ivtv_start_capture()
linux/drivers/media/video/ivtv/ivtv-streams.c:ivtv_start_v4l2_encode_stream()
linux/drivers/media/video/cx2341x.c:cx2341x_update()

linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_irq_handler()
linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_irq_enc_start_cap()
linux/drivers/media/video/ivtv/ivtv-irq.c:stream_enc_dma_append()
linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_dma_enc_start()
linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_dma_enc_start_xfer()
linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_irq_enc_dma_complete()

Regards,
Andy


_______________________________________________
ivtv-users mailing list
ivtv-users@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-users
Re: Some video frames mangled on PVR- [ In reply to ]
On Mon, 2010-01-25 at 21:51 -0500, Andy Walls wrote:

> You are certainly welcome to look at the driver for bugs. If trying to
> grok what the driver is doing WRT to data transfers from hardware, I
> would suggest looking at these:


> linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_irq_enc_dma_complete()

Hmm. If the kernel's software IOMMU implementation is active on your
machine, ivtv-irq.c:dma_post() can end up performing a buffer memcpy()
in an interrupt context for buffers when calling
ivtv_buf_sync_for_cpu().

I need to fix that. The ivtv driver should try to avoid buffer
memcpy()'s in an interrupt context for system performance reasons.


Can you check your dmesg or messages log for this:

"PCI-DMA: Using software bounce buffering for IO (SWIOTLB)"

?

Regards,
Andy


_______________________________________________
ivtv-users mailing list
ivtv-users@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-users
Re: Some video frames mangled on PVR- [ In reply to ]
On Tue, 2010-01-26 at 21:46 -0500, Peter Budny wrote:
> Andy Walls <awalls@radix.net> writes:
>
> > On Mon, 2010-01-25 at 21:51 -0500, Andy Walls wrote:
> >
> >> You are certainly welcome to look at the driver for bugs. If trying to
> >> grok what the driver is doing WRT to data transfers from hardware, I
> >> would suggest looking at these:
> >
> >
> >> linux/drivers/media/video/ivtv/ivtv-irq.c:ivtv_irq_enc_dma_complete()
> >
> > Hmm. If the kernel's software IOMMU implementation is active on your
> > machine, ivtv-irq.c:dma_post() can end up performing a buffer memcpy()
> > in an interrupt context for buffers when calling
> > ivtv_buf_sync_for_cpu().
> >
> > I need to fix that. The ivtv driver should try to avoid buffer
> > memcpy()'s in an interrupt context for system performance reasons.
> >
> >
> > Can you check your dmesg or messages log for this:
> >
> > "PCI-DMA: Using software bounce buffering for IO (SWIOTLB)"
>
> I don't see this in the log. Would I have to enable anything special to
> get it, such as IOMMU debugging?

No. It's logged at the kern.info level which means it should show up in
dmesg if it were enabled.

So back to more thought experiments and staring at code for the moment.

I may try looking at what ftrace can tell me about ivtv driver
latencies:

http://lwn.net/Articles/322666/
http://lwn.net/Articles/322731/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/365835/


But I'll need to find free time to play with FTrace before I can suggest
specific commands.

Regards,
Andy


_______________________________________________
ivtv-users mailing list
ivtv-users@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-users