Hello,
We're using nprobe with an Accolade ANIC-20ku card. We run zbalance_ipc to
merge the 2 anic interfaces using GTP hash to 16 queues. We run 16 nprobe
instances reading this traffic.
The problem is the nprobe instances are dying after some minutes working.
They dump to console the backtrace I'm pasting below.
Do you have any idea of why this is happening or what should I look into?
Thanks a lot in advance.
I just bought the nprobe-accolade license today, sorry if if this is not
the place to raise this issue).
-- System:
nProbe: nprobe-8.5.180814-6251.x86_64
System RAM: 128GB
System CPU: 56 cores (2 sockets * 14 cores* 2 threads)
System OS: CentOS Linux release 7.5.1804 (Core)
Linux Kernel: 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
-- Backtrace:
*** Error in `/usr/local/bin/nprobe': corrupted size vs. prev_size:
0x00007f4af2b9c720 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5e4)[0x7f4b9a86b5e4]
/lib64/libc.so.6(+0x816db)[0x7f4b9a86d6db]
/usr/local/lib/libnprobe-8.5.180814.so(purgeBucket+0x6d5)[0x7f4b9d719342]
/usr/local/lib/libnprobe-8.5.180814.so
(dequeueBucketToExport+0x307)[0x7f4b9d718b5a]
/lib64/libpthread.so.0(+0x7e25)[0x7f4b9c4a2e25]
/lib64/libc.so.6(clone+0x6d)[0x7f4b9a8eabad]
======= Memory map: ========
00400000-0043f000 r-xp 00000000 08:02 3564759
/usr/local/bin/nprobe
0063e000-0063f000 r--p 0003e000 08:02 3564759
/usr/local/bin/nprobe
0063f000-00640000 rw-p 0003f000 08:02 3564759
/usr/local/bin/nprobe
00640000-00689000 rw-p 00000000 00:00 0
02296000-02617000 rw-p 00000000 00:00 0
[heap]
2aaaaac00000-2aab2ac00000 rw-s 00000000 00:24 176861
/dev/hugepages/pfring_zc_1
7f4aec000000-7f4af3fa2000 rw-p 00000000 00:00 0
[...lines deleted, mail getting too big...]
7f4b04021000-7f4b08000000 ---p 00000000 00:00 0
7f4b080d1000-7f4b0c000000 r--s 00000000 08:02 534485
/usr/share/ntopng/httpdocs/geoip/GeoLite2-City.mmdb
7f4b0c000000-7f4b0c021000 rw-p 00000000 00:00 0
7f4b0c021000-7f4b10000000 ---p 00000000 00:00 0
[...lines deleted, mail getting too big...]
7f4b912c1000-7f4b91ac1000 rw-p 00000000 00:00 0
7f4b91ac1000-7f4b91ac5000 r-xp 00000000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91ac5000-7f4b91cc4000 ---p 00004000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc4000-7f4b91cc5000 r--p 00003000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc5000-7f4b91cc6000 rw-p 00004000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc6000-7f4b91cca000 r-xp 00000000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91cca000-7f4b91ec9000 ---p 00004000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91ec9000-7f4b91eca000 r--p 00003000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91eca000-7f4b91ecb000 rw-p 00004000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91ecb000-7f4b92080000 r-xp 00000000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92080000-7f4b92280000 ---p 001b5000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92280000-7f4b92287000 r--p 001b5000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92287000-7f4b9228a000 rw-p 001bc000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b9228a000-7f4b92290000 r-xp 00000000 08:02 3540943
/usr/lib64/sasl2/libsasldb.so.3.0.0
7f4b92290000-7f4b9248f000 ---p 00006000 08:02 3540943
/usr/lib64/sasl2/libsasldb.so.3.0.0
[...lines deleted, mail getting too big...]
7f4b92695000-7f4b92696000 rw-p 00004000 08:02 3557681
/usr/lib64/sasl2/libplain.so.3.0.0
7f4b92696000-7f4b9270a000 r-xp 00000000 08:02 534538
/usr/local/lib/nprobe/plugins/libssdpPlugin-8.5.180814.so
<http://libssdpplugin-8.5.180814.so/>
[...lines deleted, mail getting too big...]
7f4b9d6c1000-7f4b9d8cd000 r-xp 00000000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9d8cd000-7f4b9dacd000 ---p 0020c000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9dacd000-7f4b9dad1000 r--p 0020c000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9dad1000-7f4b9daf1000 rw-p 00210000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9daf1000-7f4b9dbb1000 rw-p 00000000 00:00 0
7f4b9dbb1000-7f4b9dbd3000 r-xp 00000000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9dd6e000-7f4b9ddbd000 rw-p 00000000 00:00 0
7f4b9ddce000-7f4b9ddd2000 rw-p 00000000 00:00 0
7f4b9ddd2000-7f4b9ddd3000 r--p 00021000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9ddd3000-7f4b9ddd4000 rw-p 00022000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9ddd4000-7f4b9ddd5000 rw-p 00000000 00:00 0
7fff7a7ae000-7fff7a7cf000 rw-p 00000000 00:00 0
[stack]
7fff7a7f4000-7fff7a7f6000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
--
Saludos,
David Notivol
dnotivol@gmail.com
We're using nprobe with an Accolade ANIC-20ku card. We run zbalance_ipc to
merge the 2 anic interfaces using GTP hash to 16 queues. We run 16 nprobe
instances reading this traffic.
The problem is the nprobe instances are dying after some minutes working.
They dump to console the backtrace I'm pasting below.
Do you have any idea of why this is happening or what should I look into?
Thanks a lot in advance.
I just bought the nprobe-accolade license today, sorry if if this is not
the place to raise this issue).
-- System:
nProbe: nprobe-8.5.180814-6251.x86_64
System RAM: 128GB
System CPU: 56 cores (2 sockets * 14 cores* 2 threads)
System OS: CentOS Linux release 7.5.1804 (Core)
Linux Kernel: 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
-- Backtrace:
*** Error in `/usr/local/bin/nprobe': corrupted size vs. prev_size:
0x00007f4af2b9c720 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5e4)[0x7f4b9a86b5e4]
/lib64/libc.so.6(+0x816db)[0x7f4b9a86d6db]
/usr/local/lib/libnprobe-8.5.180814.so(purgeBucket+0x6d5)[0x7f4b9d719342]
/usr/local/lib/libnprobe-8.5.180814.so
(dequeueBucketToExport+0x307)[0x7f4b9d718b5a]
/lib64/libpthread.so.0(+0x7e25)[0x7f4b9c4a2e25]
/lib64/libc.so.6(clone+0x6d)[0x7f4b9a8eabad]
======= Memory map: ========
00400000-0043f000 r-xp 00000000 08:02 3564759
/usr/local/bin/nprobe
0063e000-0063f000 r--p 0003e000 08:02 3564759
/usr/local/bin/nprobe
0063f000-00640000 rw-p 0003f000 08:02 3564759
/usr/local/bin/nprobe
00640000-00689000 rw-p 00000000 00:00 0
02296000-02617000 rw-p 00000000 00:00 0
[heap]
2aaaaac00000-2aab2ac00000 rw-s 00000000 00:24 176861
/dev/hugepages/pfring_zc_1
7f4aec000000-7f4af3fa2000 rw-p 00000000 00:00 0
[...lines deleted, mail getting too big...]
7f4b04021000-7f4b08000000 ---p 00000000 00:00 0
7f4b080d1000-7f4b0c000000 r--s 00000000 08:02 534485
/usr/share/ntopng/httpdocs/geoip/GeoLite2-City.mmdb
7f4b0c000000-7f4b0c021000 rw-p 00000000 00:00 0
7f4b0c021000-7f4b10000000 ---p 00000000 00:00 0
[...lines deleted, mail getting too big...]
7f4b912c1000-7f4b91ac1000 rw-p 00000000 00:00 0
7f4b91ac1000-7f4b91ac5000 r-xp 00000000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91ac5000-7f4b91cc4000 ---p 00004000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc4000-7f4b91cc5000 r--p 00003000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc5000-7f4b91cc6000 rw-p 00004000 08:02 3540940
/usr/lib64/sasl2/libanonymous.so.3.0.0
7f4b91cc6000-7f4b91cca000 r-xp 00000000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91cca000-7f4b91ec9000 ---p 00004000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91ec9000-7f4b91eca000 r--p 00003000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91eca000-7f4b91ecb000 rw-p 00004000 08:02 3557678
/usr/lib64/sasl2/liblogin.so.3.0.0
7f4b91ecb000-7f4b92080000 r-xp 00000000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92080000-7f4b92280000 ---p 001b5000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92280000-7f4b92287000 r--p 001b5000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b92287000-7f4b9228a000 rw-p 001bc000 08:02 3540137
/usr/lib64/libdb-5.3.so
7f4b9228a000-7f4b92290000 r-xp 00000000 08:02 3540943
/usr/lib64/sasl2/libsasldb.so.3.0.0
7f4b92290000-7f4b9248f000 ---p 00006000 08:02 3540943
/usr/lib64/sasl2/libsasldb.so.3.0.0
[...lines deleted, mail getting too big...]
7f4b92695000-7f4b92696000 rw-p 00004000 08:02 3557681
/usr/lib64/sasl2/libplain.so.3.0.0
7f4b92696000-7f4b9270a000 r-xp 00000000 08:02 534538
/usr/local/lib/nprobe/plugins/libssdpPlugin-8.5.180814.so
<http://libssdpplugin-8.5.180814.so/>
[...lines deleted, mail getting too big...]
7f4b9d6c1000-7f4b9d8cd000 r-xp 00000000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9d8cd000-7f4b9dacd000 ---p 0020c000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9dacd000-7f4b9dad1000 r--p 0020c000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9dad1000-7f4b9daf1000 rw-p 00210000 08:02 3564760
/usr/local/lib/libnprobe-8.5.180814.so
7f4b9daf1000-7f4b9dbb1000 rw-p 00000000 00:00 0
7f4b9dbb1000-7f4b9dbd3000 r-xp 00000000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9dd6e000-7f4b9ddbd000 rw-p 00000000 00:00 0
7f4b9ddce000-7f4b9ddd2000 rw-p 00000000 00:00 0
7f4b9ddd2000-7f4b9ddd3000 r--p 00021000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9ddd3000-7f4b9ddd4000 rw-p 00022000 08:02 3539550
/usr/lib64/ld-2.17.so
7f4b9ddd4000-7f4b9ddd5000 rw-p 00000000 00:00 0
7fff7a7ae000-7fff7a7cf000 rw-p 00000000 00:00 0
[stack]
7fff7a7f4000-7fff7a7f6000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
--
Saludos,
David Notivol
dnotivol@gmail.com