Mailing List Archive: cx18: Dual card, init issues

cx18: Dual card, init issues

Mar 13, 2009, 6:37 PM

Post #1 of 5 (3108 views)

Hi Andy,

My apologies for being quiet, I have been tied up on other projects. I'm
back with some fresh feedback and will be more active again now.

Last night I updated my driver to the latest mainline from the v4l-dvb repo.

We have been seeing some consistent problems with the driver being
inoperable after various reboot conditions.

As a reminder I am running two cx18's on a PCI riser card.

I did some repeated testing last night and found some interesting results.

I did identical tests - booting the system, letting it come on line and
stream from both encoders and then shutting down with the reboot command.
In 3/10 of those cases, I had issues with one or both video streams on
reboot. The other 7/10 times both streams came back fine. I adjusted no
other settings. When I had issues, on one card I had video and no audio,
sometimes I had nothing but a black, grey or red screen. Once I also had
video only on the second card.

I then did some experimenting and discovered that if I stopped accessing the
device and did an rmmod and then a new modprobe, I would restore full video
and audio on both devices.

I then did 10 more identical tests. This time I powered on the box, let it
boot to operational and then powered it off (hard power off, no shut down is
done on the OS side - don't worry I'm running from ram, no disk sync
issues). This time 8/10 of the boots failed and only 2/10 worked properly.
Again, if I log in, stop accessing the device, unload the cx18 module and
reload it, everything returns to normal.

After observing these issues, I modified my boot script to do a modprobe
during early init, immediately do an rmmod and then do another modprobe.
After I made that change I repeated my tests doing both soft reboot and hard
power off resets of the device. All 10/10 of those tests were successful
with both video and audio working fine for both streams.

I am unable to offer any suggestion as to what exactly is causing it, but it
does appear that there are some issues with the initialization process that
seem to be cured in the process of the removal of the module such that a
subsequent insert works properly. I hope that is enough testing feedback to
help you locate whatever may be causing the issue.

If you need me to check something else, please let me know.

-Jeff

Re: cx18: Dual card, init issues [ In reply to ]

awalls at radix

Mar 13, 2009, 7:58 PM

Post #2 of 5 (2943 views)

Permalink

On Fri, 2009-03-13 at 21:37 -0400, Jeff Campbell wrote:
> Hi Andy,
>
> My apologies for being quiet, I have been tied up on other projects.
> I'm back with some fresh feedback and will be more active again now.
>
> Last night I updated my driver to the latest mainline from the v4l-dvb
> repo.
>
> We have been seeing some consistent problems with the driver being
> inoperable after various reboot conditions.
>
> As a reminder I am running two cx18's on a PCI riser card.

Right I remember.

> I did some repeated testing last night and found some interesting
> results.
>
> I did identical tests - booting the system, letting it come on line
> and stream from both encoders and then shutting down with the reboot
> command. In 3/10 of those cases, I had issues with one or both video
> streams on reboot. The other 7/10 times both streams came back fine.
> I adjusted no other settings. When I had issues, on one card I had
> video and no audio, sometimes I had nothing but a black, grey or red
> screen. Once I also had video only on the second card.
>
> I then did some experimenting and discovered that if I stopped
> accessing the device and did an rmmod and then a new modprobe, I would
> restore full video and audio on both devices.
>
> I then did 10 more identical tests. This time I powered on the box,
> let it boot to operational and then powered it off (hard power off, no
> shut down is done on the OS side - don't worry I'm running from ram,
> no disk sync issues). This time 8/10 of the boots failed and only
> 2/10 worked properly. Again, if I log in, stop accessing the device,
> unload the cx18 module and reload it, everything returns to normal.
>
> After observing these issues, I modified my boot script to do a
> modprobe during early init, immediately do an rmmod and then do
> another modprobe. After I made that change I repeated my tests doing
> both soft reboot and hard power off resets of the device. All 10/10
> of those tests were successful with both video and audio working fine
> for both streams.

That is very good feedback. The important part is the repeatability of
the problem and the repeatability of the resolution.

That means with some well placed debug statements, differential analysis
can show what is going on. The "well placed" will be the hard part.

> I am unable to offer any suggestion as to what exactly is causing it,
> but it does appear that there are some issues with the initialization
> process that seem to be cured in the process of the removal of the
> module such that a subsequent insert works properly. I hope that is
> enough testing feedback to help you locate whatever may be causing the
> issue.

I'll make a repo in the next week with some extra debug and some extra
verification of firmware load.

My thought is that the routines in cx18-io.[ch] are reaching their retry
limit due to the PCI bus being very busy and a register or memory
location not getting written properly. Doing it again gets all the
registers and memory locations written properly.

A quick "fix" would be to increase the retry limit. Also some debug
statements about when the retry limit is reached may provide some
positive confirmation of my hypothesis. Then if that doesn't work I'll
press for information on the CX18_AUDIO_ENABLE register which just plain
bothers me in it's behavior.

Regards,
Andy

> If you need me to check something else, please let me know.
>
> -Jeff
> _______________________________________________
> ivtv-devel mailing list
> ivtv-devel@ivtvdriver.org
> http://ivtvdriver.org/mailman/listinfo/ivtv-devel

_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

Re: cx18: Dual card, init issues [ In reply to ]

awalls at radix

Mar 19, 2009, 8:57 PM

Post #3 of 5 (2872 views)

Permalink

On Fri, 2009-03-13 at 22:58 -0400, Andy Walls wrote:
> On Fri, 2009-03-13 at 21:37 -0400, Jeff Campbell wrote:
> > Hi Andy,
> >
> > My apologies for being quiet, I have been tied up on other projects.
> > I'm back with some fresh feedback and will be more active again now.
> >
> > Last night I updated my driver to the latest mainline from the v4l-dvb
> > repo.
> >
> > We have been seeing some consistent problems with the driver being
> > inoperable after various reboot conditions.
> >
> > As a reminder I am running two cx18's on a PCI riser card.
>
> Right I remember.
>
>
> > I did some repeated testing last night and found some interesting
> > results.
> >
> > I did identical tests - booting the system, letting it come on line
> > and stream from both encoders and then shutting down with the reboot
> > command. In 3/10 of those cases, I had issues with one or both video
> > streams on reboot. The other 7/10 times both streams came back fine.
> > I adjusted no other settings. When I had issues, on one card I had
> > video and no audio, sometimes I had nothing but a black, grey or red
> > screen. Once I also had video only on the second card.
> >
> > I then did some experimenting and discovered that if I stopped
> > accessing the device and did an rmmod and then a new modprobe, I would
> > restore full video and audio on both devices.
> >
> > I then did 10 more identical tests. This time I powered on the box,
> > let it boot to operational and then powered it off (hard power off, no
> > shut down is done on the OS side - don't worry I'm running from ram,
> > no disk sync issues). This time 8/10 of the boots failed and only
> > 2/10 worked properly. Again, if I log in, stop accessing the device,
> > unload the cx18 module and reload it, everything returns to normal.
> >
> > After observing these issues, I modified my boot script to do a
> > modprobe during early init, immediately do an rmmod and then do
> > another modprobe. After I made that change I repeated my tests doing
> > both soft reboot and hard power off resets of the device. All 10/10
> > of those tests were successful with both video and audio working fine
> > for both streams.
>
> That is very good feedback. The important part is the repeatability of
> the problem and the repeatability of the resolution.
>
> That means with some well placed debug statements, differential analysis
> can show what is going on. The "well placed" will be the hard part.
>
>
>
> > I am unable to offer any suggestion as to what exactly is causing it,
> > but it does appear that there are some issues with the initialization
> > process that seem to be cured in the process of the removal of the
> > module such that a subsequent insert works properly. I hope that is
> > enough testing feedback to help you locate whatever may be causing the
> > issue.
>
> I'll make a repo in the next week with some extra debug and some extra
> verification of firmware load.
>
> My thought is that the routines in cx18-io.[ch] are reaching their retry
> limit due to the PCI bus being very busy and a register or memory
> location not getting written properly. Doing it again gets all the
> registers and memory locations written properly.
>
> A quick "fix" would be to increase the retry limit. Also some debug
> statements about when the retry limit is reached may provide some
> positive confirmation of my hypothesis. Then if that doesn't work I'll
> press for information on the CX18_AUDIO_ENABLE register which just plain
> bothers me in it's behavior.
>
> Regards,
> Andy
>
>
> > If you need me to check something else, please let me know.

Jeff,

Please test the changes at

http://linuxtv.org/hg/~awalls/cx18-init-debug

The changes do a few things:

1. Verify the A/V digitizer core audio standard autodetection firmware
image was loaded properly, and logs a message.

2. Log a message when we write to a register and fail to read back what
we wrote, or fail to read back what we expected to read back, after
several retries.

3. Whenever we write to the CX18_AUDIO_ENABLE register, force bits that
we can read back to toggle, so we know we have actually written to the
register. I got this right on input change: Tuner, SVideo 1, CVBS 1,
Svideo 2, CVBS2. I may have goofed this slightly in the initialization,
function though. (I'm too tired right now.)

If you could repeat a few experiments and let me know if you see any
messages regarding failure in the log. I don't think you need to set
any debug switches.

Regards,
Andy

> > -Jeff

_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel

Re: cx18: Dual card, init issues [ In reply to ]

jac1dlists at gmail

Mar 20, 2009, 2:52 PM

Post #4 of 5 (2885 views)

Permalink

Hi Andy,

THanks for this, I'll try it tonight if this flu doesn't overcome me too
soon. I think it might.

I've been testing this all day under various conditions. We think we have
isolated it to a tuner init issue. The serial audio inits properly all the
time as far as we can tell, but we are seeing issues with the tuner input.
If I rmmod and modprobe the driver then the tuner starts working again.

I always think of the tuner portion of things as quite stable, but I guess
with all the recent v4l structure changes perhaps something got adjusted in
there that caused the issue?

I'll try your changes as well.

Thanks,

-Jeff

On Thu, Mar 19, 2009 at 11:57 PM, Andy Walls <awalls@radix.net> wrote:

> On Fri, 2009-03-13 at 22:58 -0400, Andy Walls wrote:
> > On Fri, 2009-03-13 at 21:37 -0400, Jeff Campbell wrote:
> > > Hi Andy,
> > >
> > > My apologies for being quiet, I have been tied up on other projects.
> > > I'm back with some fresh feedback and will be more active again now.
> > >
> > > Last night I updated my driver to the latest mainline from the v4l-dvb
> > > repo.
> > >
> > > We have been seeing some consistent problems with the driver being
> > > inoperable after various reboot conditions.
> > >
> > > As a reminder I am running two cx18's on a PCI riser card.
> >
> > Right I remember.
> >
> >
> > > I did some repeated testing last night and found some interesting
> > > results.
> > >
> > > I did identical tests - booting the system, letting it come on line
> > > and stream from both encoders and then shutting down with the reboot
> > > command. In 3/10 of those cases, I had issues with one or both video
> > > streams on reboot. The other 7/10 times both streams came back fine.
> > > I adjusted no other settings. When I had issues, on one card I had
> > > video and no audio, sometimes I had nothing but a black, grey or red
> > > screen. Once I also had video only on the second card.
> > >
> > > I then did some experimenting and discovered that if I stopped
> > > accessing the device and did an rmmod and then a new modprobe, I would
> > > restore full video and audio on both devices.
> > >
> > > I then did 10 more identical tests. This time I powered on the box,
> > > let it boot to operational and then powered it off (hard power off, no
> > > shut down is done on the OS side - don't worry I'm running from ram,
> > > no disk sync issues). This time 8/10 of the boots failed and only
> > > 2/10 worked properly. Again, if I log in, stop accessing the device,
> > > unload the cx18 module and reload it, everything returns to normal.
> > >
> > > After observing these issues, I modified my boot script to do a
> > > modprobe during early init, immediately do an rmmod and then do
> > > another modprobe. After I made that change I repeated my tests doing
> > > both soft reboot and hard power off resets of the device. All 10/10
> > > of those tests were successful with both video and audio working fine
> > > for both streams.
> >
> > That is very good feedback. The important part is the repeatability of
> > the problem and the repeatability of the resolution.
> >
> > That means with some well placed debug statements, differential analysis
> > can show what is going on. The "well placed" will be the hard part.
> >
> >
> >
> > > I am unable to offer any suggestion as to what exactly is causing it,
> > > but it does appear that there are some issues with the initialization
> > > process that seem to be cured in the process of the removal of the
> > > module such that a subsequent insert works properly. I hope that is
> > > enough testing feedback to help you locate whatever may be causing the
> > > issue.
> >
> > I'll make a repo in the next week with some extra debug and some extra
> > verification of firmware load.
> >
> > My thought is that the routines in cx18-io.[ch] are reaching their retry
> > limit due to the PCI bus being very busy and a register or memory
> > location not getting written properly. Doing it again gets all the
> > registers and memory locations written properly.
> >
> > A quick "fix" would be to increase the retry limit. Also some debug
> > statements about when the retry limit is reached may provide some
> > positive confirmation of my hypothesis. Then if that doesn't work I'll
> > press for information on the CX18_AUDIO_ENABLE register which just plain
> > bothers me in it's behavior.
> >
> > Regards,
> > Andy
> >
> >
> > > If you need me to check something else, please let me know.
>
> Jeff,
>
> Please test the changes at
>
> http://linuxtv.org/hg/~awalls/cx18-init-debug<http://linuxtv.org/hg/%7Eawalls/cx18-init-debug>
>
> The changes do a few things:
>
> 1. Verify the A/V digitizer core audio standard autodetection firmware
> image was loaded properly, and logs a message.
>
> 2. Log a message when we write to a register and fail to read back what
> we wrote, or fail to read back what we expected to read back, after
> several retries.
>
> 3. Whenever we write to the CX18_AUDIO_ENABLE register, force bits that
> we can read back to toggle, so we know we have actually written to the
> register. I got this right on input change: Tuner, SVideo 1, CVBS 1,
> Svideo 2, CVBS2. I may have goofed this slightly in the initialization,
> function though. (I'm too tired right now.)
>
>
> If you could repeat a few experiments and let me know if you see any
> messages regarding failure in the log. I don't think you need to set
> any debug switches.
>
> Regards,
> Andy
>
> > > -Jeff
>
>
>
> _______________________________________________
> ivtv-devel mailing list
> ivtv-devel@ivtvdriver.org
> http://ivtvdriver.org/mailman/listinfo/ivtv-devel
>

Re: cx18: Dual card, init issues [ In reply to ]

awalls at radix

Mar 20, 2009, 3:36 PM

Post #5 of 5 (2877 views)

Permalink

On Fri, 2009-03-20 at 17:52 -0400, Jeff Campbell wrote:
> Hi Andy,
>
> THanks for this, I'll try it tonight if this flu doesn't overcome me
> too soon. I think it might.
>
> I've been testing this all day under various conditions. We think we
> have isolated it to a tuner init issue. The serial audio inits
> properly all the time as far as we can tell, but we are seeing issues
> with the tuner input. If I rmmod and modprobe the driver then the
> tuner starts working again.

When the tuner doesn't work, what is

$ v4l2-ctl --log-status

saying. I'm especially interested in the lines that start with "tuner",
"tda" and "cx18-0 843:":

cx18-0 843: Video signal: present
cx18-0 843: Detected format: NTSC-M
cx18-0 843: Specified standard: NTSC-M
cx18-0 843: Specified video input: Composite 7
cx18-0 843: Specified audioclock freq: 48000 Hz
cx18-0 843: Detected audio mode: stereo with SAP
cx18-0 843: Detected audio standard: BTSC
cx18-0 843: Audio muted: no
cx18-0 843: Audio microcontroller: running
cx18-0 843: Configured audio standard: automatic detection
cx18-0 843: Configured audio system: BTSC
cx18-0 843: Specified audio input: Tuner (In8)
cx18-0 843: Preferred audio mode: stereo
cx18-0 gpio-reset-ctrl: GPIO: direction 0x00003001, value 0x00003001
tda9887 2-0043: Data bytes: b=0x14 c=0x30 e=0x44
tuner 2-0061: Tuner mode: analog TV
tuner 2-0061: Frequency: 77.25 MHz
tuner 2-0061: Standard: 0x0000b000

And do they look any different if you switch to serial audio in and back
again?

Does tuner audio get better if you change channels to a different band
(from low VHF to UHF) and back again?

> I always think of the tuner portion of things as quite stable, but I
> guess with all the recent v4l structure changes perhaps something got
> adjusted in there that caused the issue?

Hmmm. I doubt it - at least in cx18. Maybe something went on with the
tuner modules. You can also always have the tuner modules that are used
by your particular board start to log things. You can add something
like this in /etc/modprobe.conf:

options tuner-simple debug=7
options tuner debug=7
options tda9887 debug=7

> I'll try your changes as well.

Sure. The ideas is to try and catch something that's different between
the non-working conditions working conditions.

Thanks,
Andy

> Thanks,
>
> -Jeff
>
>
> On Thu, Mar 19, 2009 at 11:57 PM, Andy Walls <awalls@radix.net> wrote:
>
> On Fri, 2009-03-13 at 22:58 -0400, Andy Walls wrote:
> > On Fri, 2009-03-13 at 21:37 -0400, Jeff Campbell wrote:
> > > Hi Andy,
> > >
> > > My apologies for being quiet, I have been tied up on other
> projects.
> > > I'm back with some fresh feedback and will be more active
> again now.
> > >
> > > Last night I updated my driver to the latest mainline from
> the v4l-dvb
> > > repo.
> > >
> > > We have been seeing some consistent problems with the
> driver being
> > > inoperable after various reboot conditions.
> > >
> > > As a reminder I am running two cx18's on a PCI riser card.
> >
> > Right I remember.
> >
> >
> > > I did some repeated testing last night and found some
> interesting
> > > results.
> > >
> > > I did identical tests - booting the system, letting it
> come on line
> > > and stream from both encoders and then shutting down with
> the reboot
> > > command. In 3/10 of those cases, I had issues with one or
> both video
> > > streams on reboot. The other 7/10 times both streams came
> back fine.
> > > I adjusted no other settings. When I had issues, on one
> card I had
> > > video and no audio, sometimes I had nothing but a black,
> grey or red
> > > screen. Once I also had video only on the second card.
> > >
> > > I then did some experimenting and discovered that if I
> stopped
> > > accessing the device and did an rmmod and then a new
> modprobe, I would
> > > restore full video and audio on both devices.
> > >
> > > I then did 10 more identical tests. This time I powered
> on the box,
> > > let it boot to operational and then powered it off (hard
> power off, no
> > > shut down is done on the OS side - don't worry I'm running
> from ram,
> > > no disk sync issues). This time 8/10 of the boots failed
> and only
> > > 2/10 worked properly. Again, if I log in, stop accessing
> the device,
> > > unload the cx18 module and reload it, everything returns
> to normal.
> > >
> > > After observing these issues, I modified my boot script to
> do a
> > > modprobe during early init, immediately do an rmmod and
> then do
> > > another modprobe. After I made that change I repeated my
> tests doing
> > > both soft reboot and hard power off resets of the device.
> All 10/10
> > > of those tests were successful with both video and audio
> working fine
> > > for both streams.
> >
> > That is very good feedback. The important part is the
> repeatability of
> > the problem and the repeatability of the resolution.
> >
> > That means with some well placed debug statements,
> differential analysis
> > can show what is going on. The "well placed" will be the
> hard part.
> >
> >
> >
> > > I am unable to offer any suggestion as to what exactly is
> causing it,
> > > but it does appear that there are some issues with the
> initialization
> > > process that seem to be cured in the process of the
> removal of the
> > > module such that a subsequent insert works properly. I
> hope that is
> > > enough testing feedback to help you locate whatever may be
> causing the
> > > issue.
> >
> > I'll make a repo in the next week with some extra debug and
> some extra
> > verification of firmware load.
> >
> > My thought is that the routines in cx18-io.[ch] are reaching
> their retry
> > limit due to the PCI bus being very busy and a register or
> memory
> > location not getting written properly. Doing it again gets
> all the
> > registers and memory locations written properly.
> >
> > A quick "fix" would be to increase the retry limit. Also
> some debug
> > statements about when the retry limit is reached may provide
> some
> > positive confirmation of my hypothesis. Then if that
> doesn't work I'll
> > press for information on the CX18_AUDIO_ENABLE register
> which just plain
> > bothers me in it's behavior.
> >
> > Regards,
> > Andy
> >
> >
> > > If you need me to check something else, please let me
> know.
>
>
> Jeff,
>
> Please test the changes at
>
> http://linuxtv.org/hg/~awalls/cx18-init-debug
>
> The changes do a few things:
>
> 1. Verify the A/V digitizer core audio standard autodetection
> firmware
> image was loaded properly, and logs a message.
>
> 2. Log a message when we write to a register and fail to read
> back what
> we wrote, or fail to read back what we expected to read back,
> after
> several retries.
>
> 3. Whenever we write to the CX18_AUDIO_ENABLE register, force
> bits that
> we can read back to toggle, so we know we have actually
> written to the
> register. I got this right on input change: Tuner, SVideo 1,
> CVBS 1,
> Svideo 2, CVBS2. I may have goofed this slightly in the
> initialization,
> function though. (I'm too tired right now.)
>
>
> If you could repeat a few experiments and let me know if you
> see any
> messages regarding failure in the log. I don't think you need
> to set
> any debug switches.
>
> Regards,
> Andy
>
>
> > > -Jeff
>
>
>
> _______________________________________________
> ivtv-devel mailing list
> ivtv-devel@ivtvdriver.org
> http://ivtvdriver.org/mailman/listinfo/ivtv-devel
>
>
> _______________________________________________
> ivtv-devel mailing list
> ivtv-devel@ivtvdriver.org
> http://ivtvdriver.org/mailman/listinfo/ivtv-devel

_______________________________________________
ivtv-devel mailing list
ivtv-devel@ivtvdriver.org
http://ivtvdriver.org/mailman/listinfo/ivtv-devel