Mailing List Archive

On Nov 19, 2008, at 12:10 AM, Andy Walls wrote:

> cx18 driver users:
>
> I have implemented quite a few cx18 driver changes in my current
> experimental repo at
>
> http://linuxtv.org/hg/~awalls/cx18-bugfix
>
> The goal behind these changes is to fix problems with simultaneous
> analog and digital capture causing the driver to quit after a while.
> And to fix related problems with buffers getting lost and falling
> out of
> the driver<->firmware transfer rotation.
>
> To achieve that result, I had to do extensive rework of how interrupts
> were handled and some buffer handling tweaks. I'm looking for (brave)
> testers to give this stuff a try, or an inspection, before I ask Mauro
> to pull it. I also plan to do testing on my other two machines in a
> day
> or two.
>
> Please test the debug parameter with at least info and warn set:
>
> # modprobe cx18 debug=3
>
>
> I am especially interested in
>
>
> 1. How often you get messages like the following on your system?
>
> cx18-0 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. nnnn)

Andy,

dmesg output attached. Seems to happen frequently.

Brandon

On Sun, 2008-11-23 at 09:14 -0500, Brandon Jenkins wrote:
> On Nov 19, 2008, at 12:10 AM, Andy Walls wrote:
>
> > cx18 driver users:
> >
> > I have implemented quite a few cx18 driver changes in my current
> > experimental repo at
> >
> > http://linuxtv.org/hg/~awalls/cx18-bugfix
> >
> > The goal behind these changes is to fix problems with simultaneous
> > analog and digital capture causing the driver to quit after a while.
> > And to fix related problems with buffers getting lost and falling
> > out of
> > the driver<->firmware transfer rotation.
> >
> > To achieve that result, I had to do extensive rework of how interrupts
> > were handled and some buffer handling tweaks. I'm looking for (brave)
> > testers to give this stuff a try, or an inspection, before I ask Mauro
> > to pull it. I also plan to do testing on my other two machines in a
> > day
> > or two.
> >
> > Please test the debug parameter with at least info and warn set:
> >
> > # modprobe cx18 debug=3
> >
> >
> > I am especially interested in
> >
> >
> > 1. How often you get messages like the following on your system?
> >
> > cx18-0 warning: Possibly falling behind: CPU self-ack'ed our
> > incoming CPU to EPU mailbox (sequence no. nnnn)
>
> Andy,
>
> dmesg output attached. Seems to happen frequently.

Brandon,

[culling out and reorganizing interesting stuff:]

cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 7427) while processing
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 9372) while processing
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 10914) while processing
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 12855)
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 12978) while processing
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 13012)
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 14966)
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 17100)
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 17405) while processing
cx18-2 warning: Possibly falling behind: CPU self-ack'ed our incoming CPU to EPU mailbox (sequence no. 17422) while processing

This is all OK. They are all pretty far apart. The ones with "while
processing" on the end are really no big deal, as we're sure we got a
good copy of the mailbox data on those. And there are no messages about
detecting buffers to have fallen out of rotation and being being put
back, so you're actually not loosing buffers (that's why the message
says "Possibly"). You're in good shape.


I would note that only "cx18-2" is experiencing trouble in meeting it's
IRQ handling timeline imposed by it's firmware. There's either:

a. some piece of hardware sharing an interrupt with this cx18-2 board
and it's IRq handling routine is causing delays in invoking the cx18 ISR
routine for this board.

b. the PCI bus MMIO to cx18-2 is slow because latency timer settings on
other devices are set really, really long.

c. You've got 3 boards in an only dual core machine, and the other cx18
board IRQ handling routines have interrupts disabled on both the other
cores when the IRQ for cx18-2 comes in.

d. anything else you can dream up for why the time form hardware
interrupt line being asserted to cx18_irq_handler() is longer than the
other boards. :)



cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-2: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-2: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-1: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement
[snip]
cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for RPU acknowledgement

These are no big deal. We get impatient with the firmware and move on
when it doesn't respond to our command - we can't spend forever sleeping
for something that has little corrective action when it fails. Since
you got no "stuck mailbox" messages, the outgoing command likely did
complete. Even if the SET_MDL command actually fails on occasion, the
"buffer fallen out of rotation" detection logic will pick it up later
and send it again.


It's somewhat annoying that the firmware wants response times on the
order of 100's of usecs (I think) for it's data to be ack'ed, but it
makes us wait over 10 msecs for it to ack us at times. The 10 msec was
an empirical number on a single board machine. I may have to up the
timeout length or just quiet the message to a debug level.




All in all you're in good shape. Your system appears to have low enough
interrupt service latency to meet the demands of the firmware. (At
least one ivtv-users list user has a system that is having real trouble
meeting the interrupt service latency timeline of the firmware. I may
have to add a polled mailbox IO mode to the driver for these systems to
use.)


Again a lot of this was going on previously, it's just that the cx18
driver never bothered to look for it on incoming DMA done notifications,
report the precise condition in the logs, or correct for it very well.

Thanks for the testing and providing data!

Regards,
Andy

> Brandon



_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

On Nov 23, 2008, at 10:31 AM, Andy Walls wrote:
>
> Brandon,
>
> [culling out and reorganizing interesting stuff:]
>
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 7427) while processing
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 9372) while processing
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 10914) while processing
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 12855)
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 12978) while processing
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 13012)
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 14966)
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 17100)
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 17405) while processing
> cx18-2 warning: Possibly falling behind: CPU self-ack'ed our
> incoming CPU to EPU mailbox (sequence no. 17422) while processing
>
> This is all OK. They are all pretty far apart. The ones with "while
> processing" on the end are really no big deal, as we're sure we got a
> good copy of the mailbox data on those. And there are no messages
> about
> detecting buffers to have fallen out of rotation and being being put
> back, so you're actually not loosing buffers (that's why the message
> says "Possibly"). You're in good shape.
>
>
> I would note that only "cx18-2" is experiencing trouble in meeting
> it's
> IRQ handling timeline imposed by it's firmware. There's either:
>
> a. some piece of hardware sharing an interrupt with this cx18-2 board
> and it's IRq handling routine is causing delays in invoking the cx18
> ISR
> routine for this board.
>
> b. the PCI bus MMIO to cx18-2 is slow because latency timer settings
> on
> other devices are set really, really long.
>
> c. You've got 3 boards in an only dual core machine, and the other
> cx18
> board IRQ handling routines have interrupts disabled on both the other
> cores when the IRQ for cx18-2 comes in.
>
> d. anything else you can dream up for why the time form hardware
> interrupt line being asserted to cx18_irq_handler() is longer than the
> other boards. :)
>
>
>
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-2: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-2: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-1: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
> [snip]
> cx18-0: sending CX18_CPU_DE_SET_MDL timed out waiting 10 msecs for
> RPU acknowledgement
>
> These are no big deal. We get impatient with the firmware and move on
> when it doesn't respond to our command - we can't spend forever
> sleeping
> for something that has little corrective action when it fails. Since
> you got no "stuck mailbox" messages, the outgoing command likely did
> complete. Even if the SET_MDL command actually fails on occasion, the
> "buffer fallen out of rotation" detection logic will pick it up later
> and send it again.
>
>
> It's somewhat annoying that the firmware wants response times on the
> order of 100's of usecs (I think) for it's data to be ack'ed, but it
> makes us wait over 10 msecs for it to ack us at times. The 10 msec
> was
> an empirical number on a single board machine. I may have to up the
> timeout length or just quiet the message to a debug level.
>
>
>
>
> All in all you're in good shape. Your system appears to have low
> enough
> interrupt service latency to meet the demands of the firmware. (At
> least one ivtv-users list user has a system that is having real
> trouble
> meeting the interrupt service latency timeline of the firmware. I may
> have to add a polled mailbox IO mode to the driver for these systems
> to
> use.)
>
>
> Again a lot of this was going on previously, it's just that the cx18
> driver never bothered to look for it on incoming DMA done
> notifications,
> report the precise condition in the logs, or correct for it very well.
>
> Thanks for the testing and providing data!
>
> Regards,
> Andy
>
>> Brandon
>
>
Andy,

A couple of points to note:
1) cx18-2 was the only board making use of analog and hd recordings.
The other devices were only performing HD OTA captures.
2) cx18-2 shares irqs with: ls /proc/irq/18/ cx18-2 ehci_hcd:usb4
smp_affinity spurious uhci_hcd:usb3 uhci_hcd:usb7. Can I use irq17?
nothing seems to be on this.
3) This is actually a quad core cpu.

Thanks,

Brandon


_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

On Sun, 2008-11-23 at 10:49 -0500, Brandon Jenkins wrote:
> On Nov 23, 2008, at 10:31 AM, Andy Walls wrote:

> > All in all you're in good shape. Your system appears to have low
> > enough
> > interrupt service latency to meet the demands of the firmware. (At
> > least one ivtv-users list user has a system that is having real
> > trouble
> > meeting the interrupt service latency timeline of the firmware. I may
> > have to add a polled mailbox IO mode to the driver for these systems
> > to
> > use.)
> >
> >
> > Again a lot of this was going on previously, it's just that the cx18
> > driver never bothered to look for it on incoming DMA done
> > notifications,
> > report the precise condition in the logs, or correct for it very well.
> >
> > Thanks for the testing and providing data!
> >
> > Regards,
> > Andy
> >
> >> Brandon
> >
> >
> Andy,
>
> A couple of points to note:
> 1) cx18-2 was the only board making use of analog and hd recordings.
> The other devices were only performing HD OTA captures.

Brandon,

With simlutaneous streams and that *few* warnings about the firmware
self-acking mailboxes it sends, you really do have a responsive system.

I should emphasize that the driver reading the firmware's self-acked
mailbox can lead to dropped buffers because that the firmware may be in
the process of modifying the mailbox. In practice, it's rare that the
firmware has actually modified the mailbox by the time the IRQ handler
gets to it - at least in multi-core or lightly loaded machines. That's
why the cx18 driver has been working acceptably for most users since
it's initial release.

Simultaneous captures makes sense as to why the firmware would be found
more frequently self-acking the mailboxes it sends. I do not know the
firmware's real, minimum latency requirement for the host driver sending
an ack response to a mailbox. I do know that having two streams going
at once is going to have the firmware require the minimum latency of the
host driver more frequently.

I got the cx18 top half IRQ handler (cx18_irq_handler() and it's
supporting routines in cx18-mailbox.c) down to around 46-50 usec
execution time from start to finish on my dual core AMD64 machine.
There's not much room for improvement, as most of that 46-50 usec is PCI
MMIO accesses that can't be avoided or sped up.

The kernel will necessarily serialize servicing of the interrupts from a
particular CX23418. It is conceivable, that the two interrupts from the
CX23418 may be coming in very close together, but that the needed
serialization is putting some delay (e.g. ~ 46-50 usec under some
scenarios) in the timeline of servicing the second one.


> 2) cx18-2 shares irqs with: ls /proc/irq/18/ cx18-2 ehci_hcd:usb4
> smp_affinity spurious uhci_hcd:usb3 uhci_hcd:usb7. Can I use irq17?
> nothing seems to be on this.

If those particular USB host adapters aren't busy, then there is not
much point in trying to avoid the sharing AFAIK. A USB IR dongle would
not be a big deal; a USB disk or USB video capture device that is active
would be a concern.

The only *easy* ways I know to get a PCI card to use a different IRQ
are

a) move it to a different slot or
b) try and configure it explicitly in the BIOS.

I've only had success with moving to a physically different slot.


> 3) This is actually a quad core cpu.

Which is probably why you don't need to worry.

The way to get determinism out of any shared resource, like capacity for
processing interrupts, is to use

a) over-provisioning (having more processors to process interrupts from
various devices)

b) QoS (some interrupts have priority over others with a deterministic
priority scheme)

Regards,
Andy


> Thanks,
>
> Brandon



_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

Hi Andy,

I just tried this new bugfix release from a snapshot from yesterday. I have
to say, this one is a tremendous improvement. Very smooth playback, even
when my system load spikes. No more rtc interrupt errors either. Keep up
the great work...it's appreciated.


On Sunday 23 November 2008 14:02:44 Andy Walls wrote:
> On Sun, 2008-11-23 at 10:49 -0500, Brandon Jenkins wrote:
> > On Nov 23, 2008, at 10:31 AM, Andy Walls wrote:
> > > All in all you're in good shape. Your system appears to have low
> > > enough
> > > interrupt service latency to meet the demands of the firmware. (At
> > > least one ivtv-users list user has a system that is having real
> > > trouble
> > > meeting the interrupt service latency timeline of the firmware. I may
> > > have to add a polled mailbox IO mode to the driver for these systems
> > > to
> > > use.)
> > >
> > >
> > > Again a lot of this was going on previously, it's just that the cx18
> > > driver never bothered to look for it on incoming DMA done
> > > notifications,
> > > report the precise condition in the logs, or correct for it very well.
> > >
> > > Thanks for the testing and providing data!
> > >
> > > Regards,
> > > Andy
> > >
> > >> Brandon
> >
> > Andy,
> >
> > A couple of points to note:
> > 1) cx18-2 was the only board making use of analog and hd recordings.
> > The other devices were only performing HD OTA captures.
>
> Brandon,
>
> With simlutaneous streams and that *few* warnings about the firmware
> self-acking mailboxes it sends, you really do have a responsive system.
>
> I should emphasize that the driver reading the firmware's self-acked
> mailbox can lead to dropped buffers because that the firmware may be in
> the process of modifying the mailbox. In practice, it's rare that the
> firmware has actually modified the mailbox by the time the IRQ handler
> gets to it - at least in multi-core or lightly loaded machines. That's
> why the cx18 driver has been working acceptably for most users since
> it's initial release.
>
> Simultaneous captures makes sense as to why the firmware would be found
> more frequently self-acking the mailboxes it sends. I do not know the
> firmware's real, minimum latency requirement for the host driver sending
> an ack response to a mailbox. I do know that having two streams going
> at once is going to have the firmware require the minimum latency of the
> host driver more frequently.
>
> I got the cx18 top half IRQ handler (cx18_irq_handler() and it's
> supporting routines in cx18-mailbox.c) down to around 46-50 usec
> execution time from start to finish on my dual core AMD64 machine.
> There's not much room for improvement, as most of that 46-50 usec is PCI
> MMIO accesses that can't be avoided or sped up.
>
> The kernel will necessarily serialize servicing of the interrupts from a
> particular CX23418. It is conceivable, that the two interrupts from the
> CX23418 may be coming in very close together, but that the needed
> serialization is putting some delay (e.g. ~ 46-50 usec under some
> scenarios) in the timeline of servicing the second one.
>
> > 2) cx18-2 shares irqs with: ls /proc/irq/18/ cx18-2 ehci_hcd:usb4
> > smp_affinity spurious uhci_hcd:usb3 uhci_hcd:usb7. Can I use irq17?
> > nothing seems to be on this.
>
> If those particular USB host adapters aren't busy, then there is not
> much point in trying to avoid the sharing AFAIK. A USB IR dongle would
> not be a big deal; a USB disk or USB video capture device that is active
> would be a concern.
>
> The only *easy* ways I know to get a PCI card to use a different IRQ
> are
>
> a) move it to a different slot or
> b) try and configure it explicitly in the BIOS.
>
> I've only had success with moving to a physically different slot.
>
> > 3) This is actually a quad core cpu.
>
> Which is probably why you don't need to worry.
>
> The way to get determinism out of any shared resource, like capacity for
> processing interrupts, is to use
>
> a) over-provisioning (having more processors to process interrupts from
> various devices)
>
> b) QoS (some interrupts have priority over others with a deterministic
> priority scheme)
>
> Regards,
> Andy
>
> > Thanks,
> >
> > Brandon
>
> _______________________________________________
> ivtv-devel mailing list
> ivtv-devel@ivtvdriver.org
> http://ivtvdriver.org/mailman/listinfo/ivtv-devel



--
http://www.youtube.com/watch?v=aB3uK5nssck

_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

On Mon, 2008-11-24 at 19:10 -0500, Richard Cox wrote:
> Hi Andy,
>
> I just tried this new bugfix release from a snapshot from yesterday. I have
> to say, this one is a tremendous improvement. Very smooth playback, even
> when my system load spikes. No more rtc interrupt errors either. Keep up
> the great work...it's appreciated.

Thanks! :)

I went through all the work in the hope of getting VBI working - it was
mysteriously stalling. And now, ... well VBI still mysteriously
stalls. :(

BTW, I just got the emails notifying me that the fixes have been merged
to the main v4l-dvb repo.

Thanks for all the testing and feedback.


Also, I still think I'm going to add a polled mailbox IO mode for
machines that just can't reliably keep up with the firmware's timeline.
Al's single core machine would benefit from this mode I suspect.

It'll take me a week or two though to do that polled mode, with the
upcoming US holiday and me having to get the fallen leaves out of the
yard. :P

Regards,
Andy


> On Sunday 23 November 2008 14:02:44 Andy Walls wrote:
> > On Sun, 2008-11-23 at 10:49 -0500, Brandon Jenkins wrote:
> > > On Nov 23, 2008, at 10:31 AM, Andy Walls wrote:

> > I got the cx18 top half IRQ handler (cx18_irq_handler() and it's
> > supporting routines in cx18-mailbox.c) down to around 46-50 usec
> > execution time from start to finish on my dual core AMD64 machine.
> > There's not much room for improvement, as most of that 46-50 usec is PCI
> > MMIO accesses that can't be avoided or sped up.



_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel