Mailing List Archive

Hi Josiah
there is a fix for this on github (new packages will be availabel soon)

Alfredo

> On 15 Jun 2016, at 03:46, Josiah White <josiah@protrafsolutions.com> wrote:
>
> Hello,
>
> I recently switched from the DNA drivers to the latest drivers on the git. The software seems to be working fine with one exception that after 12-24 hours it stops sending out traffic. When this happens dmesg is filled with errors saying "ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds".
> I am using it on two interfaces and both interfaces are saying this.
>
> Thanks
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Hello,

I just updated the drivers so we will see what happens in the next day or
so.
These are updated logs from dmesg from earlier today before I updated the
driver. The problem seems to go away after a reboot of the box.

[83918.291077] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[83998.415598] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84078.284118] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84158.408635] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84238.277158] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84318.401676] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84398.270197] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84478.394715] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84558.263238] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84638.387756] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84718.256281] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
of 80 seconds
[84798.380811] ixgbe 0000:01:00.0 eth1: Reset adapter
[84798.462267] [PF_RING] Unloading ZC driver while the device is in use
from userspace!!
[84798.462270] [PF_RING] Unloading ZC driver while the device is in use
from userspace!!
[84798.652772] ixgbe 0000:01:00.0 eth1: detected SFP+: 5
[84798.888783] ixgbe 0000:01:00.0 eth1: NIC Link is Up 10 Gbps, Flow
Control: RX

At this in point if I stop the app, it will still not send traffic outbound.
dmesg also will print this:

[88298.496623] [PF_RING] pfring_release_zc_dev:6493 something got wrong
removing socket bound to eth1@0
[88298.600622] [PF_RING] pfring_release_zc_dev:6493 something got wrong
removing socket bound to eth1@0
[88298.808612] [PF_RING] pfring_release_zc_dev:6493 something got wrong
removing socket bound to eth1@1
[88298.912613] [PF_RING] pfring_release_zc_dev:6493 something got wrong
removing socket bound to eth1@1

I will check back in 24 hours to see if the problem reoccurs.

Thanks



On Wed, Jun 15, 2016 at 9:34 AM, Alfredo Cardigliano <cardigliano@ntop.org>
wrote:

> Hi Josiah
> there is a fix for this on github (new packages will be availabel soon)
>
> Alfredo
>
> > On 15 Jun 2016, at 03:46, Josiah White <josiah@protrafsolutions.com>
> wrote:
> >
> > Hello,
> >
> > I recently switched from the DNA drivers to the latest drivers on the
> git. The software seems to be working fine with one exception that after
> 12-24 hours it stops sending out traffic. When this happens dmesg is filled
> with errors saying "ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
> timeout of 80 seconds".
> > I am using it on two interfaces and both interfaces are saying this.
> >
> > Thanks
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>

The issue seemed to happen again. This time the only errors printed in
dmesg are the following:

[14079.778092] [PF_RING] Unable to activate two or more ZC sockets on the
same interface eth1/link direction
[14079.790079] [PF_RING] Unable to activate two or more ZC sockets on the
same interface eth1/link direction
[14079.814090] [PF_RING] Unable to activate two or more ZC sockets on the
same interface eth1/link direction
[14079.826089] [PF_RING] Unable to activate two or more ZC sockets on the
same interface eth1/link direction
[14079.850091] [PF_RING] Unable to activate two or more ZC sockets on the
same interface eth2/link direction
[14079.862091] [PF_RING] Unable to activate two or more ZC sockets on the
same interface eth2/link direction

There are no errors about TX hang and the software never tried to reopen
the sockets so this confuses me.

Thanks again

On Wed, Jun 15, 2016 at 10:45 PM, Josiah White <josiah@protrafsolutions.com>
wrote:

> Hello,
>
> I just updated the drivers so we will see what happens in the next day or
> so.
> These are updated logs from dmesg from earlier today before I updated the
> driver. The problem seems to go away after a reboot of the box.
>
> [83918.291077] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [83998.415598] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84078.284118] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84158.408635] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84238.277158] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84318.401676] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84398.270197] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84478.394715] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84558.263238] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84638.387756] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84718.256281] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout
> of 80 seconds
> [84798.380811] ixgbe 0000:01:00.0 eth1: Reset adapter
> [84798.462267] [PF_RING] Unloading ZC driver while the device is in use
> from userspace!!
> [84798.462270] [PF_RING] Unloading ZC driver while the device is in use
> from userspace!!
> [84798.652772] ixgbe 0000:01:00.0 eth1: detected SFP+: 5
> [84798.888783] ixgbe 0000:01:00.0 eth1: NIC Link is Up 10 Gbps, Flow
> Control: RX
>
> At this in point if I stop the app, it will still not send traffic
> outbound.
> dmesg also will print this:
>
> [88298.496623] [PF_RING] pfring_release_zc_dev:6493 something got wrong
> removing socket bound to eth1@0
> [88298.600622] [PF_RING] pfring_release_zc_dev:6493 something got wrong
> removing socket bound to eth1@0
> [88298.808612] [PF_RING] pfring_release_zc_dev:6493 something got wrong
> removing socket bound to eth1@1
> [88298.912613] [PF_RING] pfring_release_zc_dev:6493 something got wrong
> removing socket bound to eth1@1
>
> I will check back in 24 hours to see if the problem reoccurs.
>
> Thanks
>
>
>
> On Wed, Jun 15, 2016 at 9:34 AM, Alfredo Cardigliano <cardigliano@ntop.org
> > wrote:
>
>> Hi Josiah
>> there is a fix for this on github (new packages will be availabel soon)
>>
>> Alfredo
>>
>> > On 15 Jun 2016, at 03:46, Josiah White <josiah@protrafsolutions.com>
>> wrote:
>> >
>> > Hello,
>> >
>> > I recently switched from the DNA drivers to the latest drivers on the
>> git. The software seems to be working fine with one exception that after
>> 12-24 hours it stops sending out traffic. When this happens dmesg is filled
>> with errors saying "ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds".
>> > I am using it on two interfaces and both interfaces are saying this.
>> >
>> > Thanks
>> > _______________________________________________
>> > Ntop-misc mailing list
>> > Ntop-misc@listgateway.unipi.it
>> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>
>

I'd also like to note that restarting the software fixed the issue this
time and the whole server did not need to be rebooted.

On Thu, Jun 16, 2016 at 11:34 PM, Josiah White <josiah@protrafsolutions.com>
wrote:

> The issue seemed to happen again. This time the only errors printed in
> dmesg are the following:
>
> [14079.778092] [PF_RING] Unable to activate two or more ZC sockets on the
> same interface eth1/link direction
> [14079.790079] [PF_RING] Unable to activate two or more ZC sockets on the
> same interface eth1/link direction
> [14079.814090] [PF_RING] Unable to activate two or more ZC sockets on the
> same interface eth1/link direction
> [14079.826089] [PF_RING] Unable to activate two or more ZC sockets on the
> same interface eth1/link direction
> [14079.850091] [PF_RING] Unable to activate two or more ZC sockets on the
> same interface eth2/link direction
> [14079.862091] [PF_RING] Unable to activate two or more ZC sockets on the
> same interface eth2/link direction
>
> There are no errors about TX hang and the software never tried to reopen
> the sockets so this confuses me.
>
> Thanks again
>
> On Wed, Jun 15, 2016 at 10:45 PM, Josiah White <
> josiah@protrafsolutions.com> wrote:
>
>> Hello,
>>
>> I just updated the drivers so we will see what happens in the next day or
>> so.
>> These are updated logs from dmesg from earlier today before I updated the
>> driver. The problem seems to go away after a reboot of the box.
>>
>> [83918.291077] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [83998.415598] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84078.284118] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84158.408635] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84238.277158] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84318.401676] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84398.270197] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84478.394715] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84558.263238] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84638.387756] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84718.256281] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>> timeout of 80 seconds
>> [84798.380811] ixgbe 0000:01:00.0 eth1: Reset adapter
>> [84798.462267] [PF_RING] Unloading ZC driver while the device is in use
>> from userspace!!
>> [84798.462270] [PF_RING] Unloading ZC driver while the device is in use
>> from userspace!!
>> [84798.652772] ixgbe 0000:01:00.0 eth1: detected SFP+: 5
>> [84798.888783] ixgbe 0000:01:00.0 eth1: NIC Link is Up 10 Gbps, Flow
>> Control: RX
>>
>> At this in point if I stop the app, it will still not send traffic
>> outbound.
>> dmesg also will print this:
>>
>> [88298.496623] [PF_RING] pfring_release_zc_dev:6493 something got wrong
>> removing socket bound to eth1@0
>> [88298.600622] [PF_RING] pfring_release_zc_dev:6493 something got wrong
>> removing socket bound to eth1@0
>> [88298.808612] [PF_RING] pfring_release_zc_dev:6493 something got wrong
>> removing socket bound to eth1@1
>> [88298.912613] [PF_RING] pfring_release_zc_dev:6493 something got wrong
>> removing socket bound to eth1@1
>>
>> I will check back in 24 hours to see if the problem reoccurs.
>>
>> Thanks
>>
>>
>>
>> On Wed, Jun 15, 2016 at 9:34 AM, Alfredo Cardigliano <
>> cardigliano@ntop.org> wrote:
>>
>>> Hi Josiah
>>> there is a fix for this on github (new packages will be availabel soon)
>>>
>>> Alfredo
>>>
>>> > On 15 Jun 2016, at 03:46, Josiah White <josiah@protrafsolutions.com>
>>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > I recently switched from the DNA drivers to the latest drivers on the
>>> git. The software seems to be working fine with one exception that after
>>> 12-24 hours it stops sending out traffic. When this happens dmesg is filled
>>> with errors saying "ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with
>>> timeout of 80 seconds".
>>> > I am using it on two interfaces and both interfaces are saying this.
>>> >
>>> > Thanks
>>> > _______________________________________________
>>> > Ntop-misc mailing list
>>> > Ntop-misc@listgateway.unipi.it
>>> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>>
>>>
>>> _______________________________________________
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>>
>>
>>
>

Hi Josiah
this seems to be unrelated to the previous issue actually,
pf_ring is unable to activate more than one socket on the same interface when in ZC mode,
thus it prints the messages at the moment you start your application, don’t know exactly what
tests you are doing..

Alfredo

> On 17 Jun 2016, at 05:36, Josiah White <josiah@protrafsolutions.com> wrote:
>
> I'd also like to note that restarting the software fixed the issue this time and the whole server did not need to be rebooted.
>
> On Thu, Jun 16, 2016 at 11:34 PM, Josiah White <josiah@protrafsolutions.com <mailto:josiah@protrafsolutions.com>> wrote:
> The issue seemed to happen again. This time the only errors printed in dmesg are the following:
>
> [14079.778092] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth1/link direction
> [14079.790079] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth1/link direction
> [14079.814090] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth1/link direction
> [14079.826089] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth1/link direction
> [14079.850091] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth2/link direction
> [14079.862091] [PF_RING] Unable to activate two or more ZC sockets on the same interface eth2/link direction
>
> There are no errors about TX hang and the software never tried to reopen the sockets so this confuses me.
>
> Thanks again
>
> On Wed, Jun 15, 2016 at 10:45 PM, Josiah White <josiah@protrafsolutions.com <mailto:josiah@protrafsolutions.com>> wrote:
> Hello,
>
> I just updated the drivers so we will see what happens in the next day or so.
> These are updated logs from dmesg from earlier today before I updated the driver. The problem seems to go away after a reboot of the box.
>
> [83918.291077] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [83998.415598] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84078.284118] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84158.408635] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84238.277158] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84318.401676] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84398.270197] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84478.394715] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84558.263238] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84638.387756] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84718.256281] ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds
> [84798.380811] ixgbe 0000:01:00.0 eth1: Reset adapter
> [84798.462267] [PF_RING] Unloading ZC driver while the device is in use from userspace!!
> [84798.462270] [PF_RING] Unloading ZC driver while the device is in use from userspace!!
> [84798.652772] ixgbe 0000:01:00.0 eth1: detected SFP+: 5
> [84798.888783] ixgbe 0000:01:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: RX
>
> At this in point if I stop the app, it will still not send traffic outbound.
> dmesg also will print this:
>
> [88298.496623] [PF_RING] pfring_release_zc_dev:6493 something got wrong removing socket bound to eth1@0
> [88298.600622] [PF_RING] pfring_release_zc_dev:6493 something got wrong removing socket bound to eth1@0
> [88298.808612] [PF_RING] pfring_release_zc_dev:6493 something got wrong removing socket bound to eth1@1
> [88298.912613] [PF_RING] pfring_release_zc_dev:6493 something got wrong removing socket bound to eth1@1
>
> I will check back in 24 hours to see if the problem reoccurs.
>
> Thanks
>
>
>
> On Wed, Jun 15, 2016 at 9:34 AM, Alfredo Cardigliano <cardigliano@ntop.org <mailto:cardigliano@ntop.org>> wrote:
> Hi Josiah
> there is a fix for this on github (new packages will be availabel soon)
>
> Alfredo
>
> > On 15 Jun 2016, at 03:46, Josiah White <josiah@protrafsolutions.com <mailto:josiah@protrafsolutions.com>> wrote:
> >
> > Hello,
> >
> > I recently switched from the DNA drivers to the latest drivers on the git. The software seems to be working fine with one exception that after 12-24 hours it stops sending out traffic. When this happens dmesg is filled with errors saying "ixgbe 0000:01:00.0 eth1: Fake Tx hang detected with timeout of 80 seconds".
> > I am using it on two interfaces and both interfaces are saying this.
> >
> > Thanks
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc