Mailing List Archive: nprobe dying with Accolade

nprobe dying with Accolade

Sep 13, 2018, 12:26 AM

Post #1 of 3 (1083 views)

Hello,

We're using nprobe with an Accolade ANIC-20ku card. We run zbalance_ipc to
merge the 2 anic interfaces using GTP hash to 16 queues. We run 16 nprobe
instances reading this traffic.
The problem is the nprobe instances are dying after some minutes working.
They dump to console the backtrace I'm pasting below.

Do you have any idea of why this is happening or what should I look into?
Thanks a lot in advance.

I just bought the nprobe-accolade license today, sorry if if this is not
the place to raise this issue).

-- System:

nProbe: nprobe-8.5.180814-6251.x86_64
System RAM: 128GB
System CPU: 56 cores (2 sockets * 14 cores* 2 threads)
System OS: CentOS Linux release 7.5.1804 (Core)
Linux Kernel: 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

-- Backtrace:

*** Error in `/usr/local/bin/nprobe': corrupted size vs. prev_size:
0x00007f4af2b9c720 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5e4)[0x7f4b9a86b5e4]
/lib64/libc.so.6(+0x816db)[0x7f4b9a86d6db]
/usr/local/lib/libnprobe-8.5.180814.so(purgeBucket+0x6d5)[0x7f4b9d719342]
/usr/local/lib/libnprobe-8.5.180814.so
(dequeueBucketToExport+0x307)[0x7f4b9d718b5a]
/lib64/libpthread.so.0(+0x7e25)[0x7f4b9c4a2e25]
/lib64/libc.so.6(clone+0x6d)[0x7f4b9a8eabad]
======= Memory map: ========
00400000-0043f000 r-xp 00000000 08:02 3564759
/usr/local/bin/nprobe
0063e000-0063f000 r--p 0003e000 08:02 3564759
/usr/local/bin/nprobe
0063f000-00640000 rw-p 0003f000 08:02 3564759
/usr/local/bin/nprobe
00640000-00689000 rw-p 00000000 00:00 0
02296000-02617000 rw-p 00000000 00:00 0
[heap]
2aaaaac00000-2aab2ac00000 rw-s 00000000 00:24 176861
/dev/hugepages/pfring_zc_1
7f4aec000000-7f4af3fa2000 rw-p 00000000 00:00 0

[...lines deleted, mail getting too big...]

7f4b04021000-7f4b08000000 ---p 00000000 00:00 0
7f4b080d1000-7f4b0c000000 r--s 00000000 08:02 534485
/usr/share/ntopng/httpdocs/geoip/GeoLite2-City.mmdb
7f4b0c000000-7f4b0c021000 rw-p 00000000 00:00 0
7f4b0c021000-7f4b10000000 ---p 00000000 00:00 0

[...lines deleted, mail getting too big...]

7f4b912c1000-7f4b91ac1000 rw-p 00000000 00:00 0
7f4b91ac1000-7f4b91ac5000 r-xp 00000000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91ac5000-7f4b91cc4000 ---p 00004000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc4000-7f4b91cc5000 r--p 00003000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc5000-7f4b91cc6000 rw-p 00004000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc6000-7f4b91cca000 r-xp 00000000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91cca000-7f4b91ec9000 ---p 00004000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91ec9000-7f4b91eca000 r--p 00003000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91eca000-7f4b91ecb000 rw-p 00004000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91ecb000-7f4b92080000 r-xp 00000000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92080000-7f4b92280000 ---p 001b5000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92280000-7f4b92287000 r--p 001b5000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92287000-7f4b9228a000 rw-p 001bc000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b9228a000-7f4b92290000 r-xp 00000000 08:02 3540943
/usr/lib64/sasl2/libsasldb.so.3.0.0
7f4b92290000-7f4b9248f000 ---p 00006000 08:02 3540943
/usr/lib64/sasl2/libsasldb.so.3.0.0

[...lines deleted, mail getting too big...]

7f4b92695000-7f4b92696000 rw-p 00004000 08:02 3557681
/usr/lib64/sasl2/libplain.so.3.0.0
7f4b92696000-7f4b9270a000 r-xp 00000000 08:02 534538
/usr/local/lib/nprobe/plugins/libssdpPlugin-8.5.180814.so
<http://libssdpplugin-8.5.180814.so/>

[...lines deleted, mail getting too big...]

7f4b9d6c1000-7f4b9d8cd000 r-xp 00000000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9d8cd000-7f4b9dacd000 ---p 0020c000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9dacd000-7f4b9dad1000 r--p 0020c000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9dad1000-7f4b9daf1000 rw-p 00210000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9daf1000-7f4b9dbb1000 rw-p 00000000 00:00 0
7f4b9dbb1000-7f4b9dbd3000 r-xp 00000000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9dd6e000-7f4b9ddbd000 rw-p 00000000 00:00 0
7f4b9ddce000-7f4b9ddd2000 rw-p 00000000 00:00 0
7f4b9ddd2000-7f4b9ddd3000 r--p 00021000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9ddd3000-7f4b9ddd4000 rw-p 00022000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9ddd4000-7f4b9ddd5000 rw-p 00000000 00:00 0
7fff7a7ae000-7fff7a7cf000 rw-p 00000000 00:00 0
[stack]
7fff7a7f4000-7fff7a7f6000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]

--
Saludos,
David Notivol
dnotivol@gmail.com

Re: nprobe dying with Accolade [ In reply to ]

cardigliano at ntop

Sep 13, 2018, 1:48 AM

Post #2 of 3 (1082 views)

Permalink

Hi David
I will send you instructions for generating a core dump in order to debug this issue,
btw please use https://github.com/ntop/nProbe/issues <https://github.com/ntop/nProbe/issues> for support requests/tickets.

Thank you
Alfredo

> On 13 Sep 2018, at 09:26, David Notivol <dnotivol@gmail.com> wrote:
>
> Hello,
>
> We're using nprobe with an Accolade ANIC-20ku card. We run zbalance_ipc to merge the 2 anic interfaces using GTP hash to 16 queues. We run 16 nprobe instances reading this traffic.
> The problem is the nprobe instances are dying after some minutes working. They dump to console the backtrace I'm pasting below.
>
> Do you have any idea of why this is happening or what should I look into?
> Thanks a lot in advance.
>
> I just bought the nprobe-accolade license today, sorry if if this is not the place to raise this issue).
>
> -- System:
>
> nProbe: nprobe-8.5.180814-6251.x86_64
> System RAM: 128GB
> System CPU: 56 cores (2 sockets * 14 cores* 2 threads)
> System OS: CentOS Linux release 7.5.1804 (Core)
> Linux Kernel: 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>
>
> -- Backtrace:
>
> *** Error in `/usr/local/bin/nprobe': corrupted size vs. prev_size: 0x00007f4af2b9c720 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x7f5e4)[0x7f4b9a86b5e4]
> /lib64/libc.so.6(+0x816db)[0x7f4b9a86d6db]
> /usr/local/lib/libnprobe-8.5.180814.so <http://libnprobe-8.5.180814.so/>(purgeBucket+0x6d5)[0x7f4b9d719342]
> /usr/local/lib/libnprobe-8.5.180814.so <http://libnprobe-8.5.180814.so/>(dequeueBucketToExport+0x307)[0x7f4b9d718b5a]
> /lib64/libpthread.so.0(+0x7e25)[0x7f4b9c4a2e25]
> /lib64/libc.so.6(clone+0x6d)[0x7f4b9a8eabad]
> ======= Memory map: ========
> 00400000-0043f000 r-xp 00000000 08:02 3564759 /usr/local/bin/nprobe
> 0063e000-0063f000 r--p 0003e000 08:02 3564759 /usr/local/bin/nprobe
> 0063f000-00640000 rw-p 0003f000 08:02 3564759 /usr/local/bin/nprobe
> 00640000-00689000 rw-p 00000000 00:00 0
> 02296000-02617000 rw-p 00000000 00:00 0 [heap]
> 2aaaaac00000-2aab2ac00000 rw-s 00000000 00:24 176861 /dev/hugepages/pfring_zc_1
> 7f4aec000000-7f4af3fa2000 rw-p 00000000 00:00 0
>
> [...lines deleted, mail getting too big...]
>
> 7f4b04021000-7f4b08000000 ---p 00000000 00:00 0
> 7f4b080d1000-7f4b0c000000 r--s 00000000 08:02 534485 /usr/share/ntopng/httpdocs/geoip/GeoLite2-City.mmdb
> 7f4b0c000000-7f4b0c021000 rw-p 00000000 00:00 0
> 7f4b0c021000-7f4b10000000 ---p 00000000 00:00 0
>
> [...lines deleted, mail getting too big...]
>
> 7f4b912c1000-7f4b91ac1000 rw-p 00000000 00:00 0
> 7f4b91ac1000-7f4b91ac5000 r-xp 00000000 08:02 3540940 /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91ac5000-7f4b91cc4000 ---p 00004000 08:02 3540940 /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91cc4000-7f4b91cc5000 r--p 00003000 08:02 3540940 /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91cc5000-7f4b91cc6000 rw-p 00004000 08:02 3540940 /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91cc6000-7f4b91cca000 r-xp 00000000 08:02 3557678 /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91cca000-7f4b91ec9000 ---p 00004000 08:02 3557678 /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91ec9000-7f4b91eca000 r--p 00003000 08:02 3557678 /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91eca000-7f4b91ecb000 rw-p 00004000 08:02 3557678 /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91ecb000-7f4b92080000 r-xp 00000000 08:02 3540137 /usr/lib64/libdb-5.3.so <http://libdb-5.3.so/>
> 7f4b92080000-7f4b92280000 ---p 001b5000 08:02 3540137 /usr/lib64/libdb-5.3.so <http://libdb-5.3.so/>
> 7f4b92280000-7f4b92287000 r--p 001b5000 08:02 3540137 /usr/lib64/libdb-5.3.so <http://libdb-5.3.so/>
> 7f4b92287000-7f4b9228a000 rw-p 001bc000 08:02 3540137 /usr/lib64/libdb-5.3.so <http://libdb-5.3.so/>
> 7f4b9228a000-7f4b92290000 r-xp 00000000 08:02 3540943 /usr/lib64/sasl2/libsasldb.so.3.0.0
> 7f4b92290000-7f4b9248f000 ---p 00006000 08:02 3540943 /usr/lib64/sasl2/libsasldb.so.3.0.0
>
> [...lines deleted, mail getting too big...]
>
> 7f4b92695000-7f4b92696000 rw-p 00004000 08:02 3557681 /usr/lib64/sasl2/libplain.so.3.0.0
> 7f4b92696000-7f4b9270a000 r-xp 00000000 08:02 534538 /usr/local/lib/nprobe/plugins/libssdpPlugin-8.5.180814.so <http://libssdpplugin-8.5.180814.so/>
>
> [...lines deleted, mail getting too big...]
>
> 7f4b9d6c1000-7f4b9d8cd000 r-xp 00000000 08:02 3564760 /usr/local/lib/libnprobe-8.5.180814.so <http://libnprobe-8.5.180814.so/>
> 7f4b9d8cd000-7f4b9dacd000 ---p 0020c000 08:02 3564760 /usr/local/lib/libnprobe-8.5.180814.so <http://libnprobe-8.5.180814.so/>
> 7f4b9dacd000-7f4b9dad1000 r--p 0020c000 08:02 3564760 /usr/local/lib/libnprobe-8.5.180814.so <http://libnprobe-8.5.180814.so/>
> 7f4b9dad1000-7f4b9daf1000 rw-p 00210000 08:02 3564760 /usr/local/lib/libnprobe-8.5.180814.so <http://libnprobe-8.5.180814.so/>
> 7f4b9daf1000-7f4b9dbb1000 rw-p 00000000 00:00 0
> 7f4b9dbb1000-7f4b9dbd3000 r-xp 00000000 08:02 3539550 /usr/lib64/ld-2.17.so <http://ld-2.17.so/>
> 7f4b9dd6e000-7f4b9ddbd000 rw-p 00000000 00:00 0
> 7f4b9ddce000-7f4b9ddd2000 rw-p 00000000 00:00 0
> 7f4b9ddd2000-7f4b9ddd3000 r--p 00021000 08:02 3539550 /usr/lib64/ld-2.17.so <http://ld-2.17.so/>
> 7f4b9ddd3000-7f4b9ddd4000 rw-p 00022000 08:02 3539550 /usr/lib64/ld-2.17.so <http://ld-2.17.so/>
> 7f4b9ddd4000-7f4b9ddd5000 rw-p 00000000 00:00 0
> 7fff7a7ae000-7fff7a7cf000 rw-p 00000000 00:00 0 [stack]
> 7fff7a7f4000-7fff7a7f6000 r-xp 00000000 00:00 0 [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
>
> --
> Saludos,
> David Notivol
> dnotivol@gmail.com <mailto:dnotivol@gmail.com>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: nprobe dying with Accolade [ In reply to ]

dnotivol at gmail

Sep 13, 2018, 3:21 AM

Post #3 of 3 (1082 views)

Permalink

Thanks Alfredo. I just opened the issue in github.

Regards,
David.

El jue., 13 sept. 2018 a las 10:48, Alfredo Cardigliano (<
cardigliano@ntop.org>) escribió:

> Hi David
> I will send you instructions for generating a core dump in order to debug
> this issue,
> btw please use https://github.com/ntop/nProbe/issues for support
> requests/tickets.
>
> Thank you
> Alfredo
>
> On 13 Sep 2018, at 09:26, David Notivol <dnotivol@gmail.com> wrote:
>
> Hello,
>
> We're using nprobe with an Accolade ANIC-20ku card. We run zbalance_ipc to
> merge the 2 anic interfaces using GTP hash to 16 queues. We run 16 nprobe
> instances reading this traffic.
> The problem is the nprobe instances are dying after some minutes working.
> They dump to console the backtrace I'm pasting below.
>
> Do you have any idea of why this is happening or what should I look into?
> Thanks a lot in advance.
>
> I just bought the nprobe-accolade license today, sorry if if this is not
> the place to raise this issue).
>
> -- System:
>
> nProbe: nprobe-8.5.180814-6251.x86_64
> System RAM: 128GB
> System CPU: 56 cores (2 sockets * 14 cores* 2 threads)
> System OS: CentOS Linux release 7.5.1804 (Core)
> Linux Kernel: 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> -- Backtrace:
>
> *** Error in `/usr/local/bin/nprobe': corrupted size vs. prev_size:
> 0x00007f4af2b9c720 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x7f5e4)[0x7f4b9a86b5e4]
> /lib64/libc.so.6(+0x816db)[0x7f4b9a86d6db]
> /usr/local/lib/libnprobe-8.5.180814.so(purgeBucket+0x6d5)[0x7f4b9d719342]
> /usr/local/lib/libnprobe-8.5.180814.so
> (dequeueBucketToExport+0x307)[0x7f4b9d718b5a]
> /lib64/libpthread.so.0(+0x7e25)[0x7f4b9c4a2e25]
> /lib64/libc.so.6(clone+0x6d)[0x7f4b9a8eabad]
> ======= Memory map: ========
> 00400000-0043f000 r-xp 00000000 08:02 3564759
> /usr/local/bin/nprobe
> 0063e000-0063f000 r--p 0003e000 08:02 3564759
> /usr/local/bin/nprobe
> 0063f000-00640000 rw-p 0003f000 08:02 3564759
> /usr/local/bin/nprobe
> 00640000-00689000 rw-p 00000000 00:00 0
> 02296000-02617000 rw-p 00000000 00:00 0
> [heap]
> 2aaaaac00000-2aab2ac00000 rw-s 00000000 00:24 176861
> /dev/hugepages/pfring_zc_1
> 7f4aec000000-7f4af3fa2000 rw-p 00000000 00:00 0
>
> [...lines deleted, mail getting too big...]
>
> 7f4b04021000-7f4b08000000 ---p 00000000 00:00 0
> 7f4b080d1000-7f4b0c000000 r--s 00000000 08:02 534485
> /usr/share/ntopng/httpdocs/geoip/GeoLite2-City.mmdb
> 7f4b0c000000-7f4b0c021000 rw-p 00000000 00:00 0
> 7f4b0c021000-7f4b10000000 ---p 00000000 00:00 0
>
> [...lines deleted, mail getting too big...]
>
> 7f4b912c1000-7f4b91ac1000 rw-p 00000000 00:00 0
> 7f4b91ac1000-7f4b91ac5000 r-xp 00000000 08:02 3540940
> /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91ac5000-7f4b91cc4000 ---p 00004000 08:02 3540940
> /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91cc4000-7f4b91cc5000 r--p 00003000 08:02 3540940
> /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91cc5000-7f4b91cc6000 rw-p 00004000 08:02 3540940
> /usr/lib64/sasl2/libanonymous.so.3.0.0
> 7f4b91cc6000-7f4b91cca000 r-xp 00000000 08:02 3557678
> /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91cca000-7f4b91ec9000 ---p 00004000 08:02 3557678
> /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91ec9000-7f4b91eca000 r--p 00003000 08:02 3557678
> /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91eca000-7f4b91ecb000 rw-p 00004000 08:02 3557678
> /usr/lib64/sasl2/liblogin.so.3.0.0
> 7f4b91ecb000-7f4b92080000 r-xp 00000000 08:02 3540137
> /usr/lib64/libdb-5.3.so
> 7f4b92080000-7f4b92280000 ---p 001b5000 08:02 3540137
> /usr/lib64/libdb-5.3.so
> 7f4b92280000-7f4b92287000 r--p 001b5000 08:02 3540137
> /usr/lib64/libdb-5.3.so
> 7f4b92287000-7f4b9228a000 rw-p 001bc000 08:02 3540137
> /usr/lib64/libdb-5.3.so
> 7f4b9228a000-7f4b92290000 r-xp 00000000 08:02 3540943
> /usr/lib64/sasl2/libsasldb.so.3.0.0
> 7f4b92290000-7f4b9248f000 ---p 00006000 08:02 3540943
> /usr/lib64/sasl2/libsasldb.so.3.0.0
>
> [...lines deleted, mail getting too big...]
>
> 7f4b92695000-7f4b92696000 rw-p 00004000 08:02 3557681
> /usr/lib64/sasl2/libplain.so.3.0.0
> 7f4b92696000-7f4b9270a000 r-xp 00000000 08:02 534538
> /usr/local/lib/nprobe/plugins/libssdpPlugin-8.5.180814.so
> <http://libssdpplugin-8.5.180814.so/>
>
> [...lines deleted, mail getting too big...]
>
> 7f4b9d6c1000-7f4b9d8cd000 r-xp 00000000 08:02 3564760
> /usr/local/lib/libnprobe-8.5.180814.so
> 7f4b9d8cd000-7f4b9dacd000 ---p 0020c000 08:02 3564760
> /usr/local/lib/libnprobe-8.5.180814.so
> 7f4b9dacd000-7f4b9dad1000 r--p 0020c000 08:02 3564760
> /usr/local/lib/libnprobe-8.5.180814.so
> 7f4b9dad1000-7f4b9daf1000 rw-p 00210000 08:02 3564760
> /usr/local/lib/libnprobe-8.5.180814.so
> 7f4b9daf1000-7f4b9dbb1000 rw-p 00000000 00:00 0
> 7f4b9dbb1000-7f4b9dbd3000 r-xp 00000000 08:02 3539550
> /usr/lib64/ld-2.17.so
> 7f4b9dd6e000-7f4b9ddbd000 rw-p 00000000 00:00 0
> 7f4b9ddce000-7f4b9ddd2000 rw-p 00000000 00:00 0
> 7f4b9ddd2000-7f4b9ddd3000 r--p 00021000 08:02 3539550
> /usr/lib64/ld-2.17.so
> 7f4b9ddd3000-7f4b9ddd4000 rw-p 00022000 08:02 3539550
> /usr/lib64/ld-2.17.so
> 7f4b9ddd4000-7f4b9ddd5000 rw-p 00000000 00:00 0
> 7fff7a7ae000-7fff7a7cf000 rw-p 00000000 00:00 0
> [stack]
> 7fff7a7f4000-7fff7a7f6000 r-xp 00000000 00:00 0
> [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
>
> --
> Saludos,
> David Notivol
> dnotivol@gmail.com
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

--
Saludos,
David Notivol
dnotivol@gmail.com

Mailing List Archive

Mailing List Archive

Attached Files: