Thanks Alfredo, all is working as expected now!
________________________________
From: ntop-misc-bounces@listgateway.unipi.it <ntop-misc-bounces@listgateway.unipi.it> on behalf of Alfredo Cardigliano <cardigliano@ntop.org>
Sent: Thursday, November 17, 2016 5:49 AM
To: ntop-misc@listgateway.unipi.it
Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64
Exactly, you need a configuration line per CPU node
node=0 hugepagenumber=1024
node=1 hugepagenumber=1024
Alfredo
On 16 Nov 2016, at 20:30, Spransy, Derek <dsprans@emory.edu<mailto:dsprans@emory.edu>> wrote:
Hi Alfredo,
Here's the output:
Node 0 HugePages_Total: 1024
Node 0 HugePages_Free: 1024
Node 0 HugePages_Surp: 0
Node 1 HugePages_Total: 0
Node 1 HugePages_Free: 0
Node 1 HugePages_Surp: 0
Our hugepages.conf consists of:
node=0 hugepagenumber=1024
Do you think this needs to be added for node 1 as well?
Thanks,
Derek
________________________________
From: ntop-misc-bounces@listgateway.unipi.it<mailto:ntop-misc-bounces@listgateway.unipi.it> <ntop-misc-bounces@listgateway.unipi.it<mailto:ntop-misc-bounces@listgateway.unipi.it>> on behalf of Alfredo Cardigliano <cardigliano@ntop.org<mailto:cardigliano@ntop.org>>
Sent: Tuesday, November 15, 2016 9:01 AM
To: ntop-misc@listgateway.unipi.it<mailto:ntop-misc@listgateway.unipi.it>
Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64
Hi Derek
this seems to be related to hugepages, could you provide more info about memory in your system including:
cat /sys/devices/system/node/node*/meminfo | grep Huge
Alfredo
On 14 Nov 2016, at 15:13, Spransy, Derek <dsprans@emory.edu<mailto:dsprans@emory.edu>> wrote:
Hi Alfredo,
Yes the behavior is the same as before:
$ sudo /usr/local/bin/n2disk10g /etc/n2disk/n2disk-eth5.conf
14/Nov/2016 09:04:21 [n2disk.c:4819] Welcome to n2disk10g v.2.6.161111 (r4672) [SandyBridge]
14/Nov/2016 09:04:21 [n2disk.c:4846] Running on 2 node(s) system with 24 core(s). NUMA affinity set to node 1.
14/Nov/2016 09:04:21 [n2disk.c:4875] Using PF_RING for packet capture
14/Nov/2016 09:04:21 [n2disk.c:4901] WARNING: If you are using standard drivers (packet capture via kernel) please disable time-pulse thread
14/Nov/2016 09:04:21 [n2disk.c:4904] Multithread support enabled
14/Nov/2016 09:04:21 [n2disk.c:5018] Dump files max size is set to 1024 MB
14/Nov/2016 09:04:21 [n2disk.c:5035] Buffer memory is set to 3 GB (x 3 pcap files)
14/Nov/2016 09:04:21 [n2disk.c:5070] Using directory /data1/captures for dump files
14/Nov/2016 09:04:21 [n2disk.c:5075] No sub-directories will be created
14/Nov/2016 09:04:21 [n2disk.c:5080] Up to 43000 files will be written before overwriting
14/Nov/2016 09:04:21 [n2disk.c:5090] Dump files max duration is set to 600 sec
14/Nov/2016 09:04:21 [n2disk.c:5106] Dumping data in 0.1 MB chunks
14/Nov/2016 09:04:21 [n2disk.c:5149] Index processing memory is set to 847 MB (x 3 index files)
14/Nov/2016 09:04:24 [n2disk.c:5339] Memory allocated successfully
14/Nov/2016 09:04:24 [n2disk.c:3597] Using time pulse timestamps
14/Nov/2016 09:04:24 [n2disk.c:3630] Started PF_RING packet reader thread for device zc:eth5
$
Running zbalance works fine:
$ sudo /usr/local/bin/zbalance_ipc -i zc:eth5 -c 99 -n 1 -m 1
14/Nov/2016 09:01:24 [zbalance_ipc.c:686] Starting balancer with 1 consumer queues..
14/Nov/2016 09:01:24 [zbalance_ipc.c:696] Run your application instances as follows:
14/Nov/2016 09:01:24 [zbalance_ipc.c:701] pfcount -i zc:99@0
14/Nov/2016 09:01:25 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:25 [zbalance_ipc.c:152] Absolute Stats: Recv 354'797 pkts (0 drops) - Forwarded 8'190 pkts (346'607 drops)
14/Nov/2016 09:01:25 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:26 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:26 [zbalance_ipc.c:152] Absolute Stats: Recv 714'213 pkts (0 drops) - Forwarded 8'190 pkts (706'023 drops)
14/Nov/2016 09:01:26 [zbalance_ipc.c:224] Actual Stats: Recv 359'348.08 pps (0.00 drops) - Forwarded 0.00 pps (359'348.08 drops)
14/Nov/2016 09:01:26 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:27 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:27 [zbalance_ipc.c:152] Absolute Stats: Recv 1'053'124 pkts (0 drops) - Forwarded 8'190 pkts (1'044'926 drops)
14/Nov/2016 09:01:27 [zbalance_ipc.c:224] Actual Stats: Recv 338'839.84 pps (0.00 drops) - Forwarded 0.00 pps (338'831.84 drops)
14/Nov/2016 09:01:27 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:28 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:28 [zbalance_ipc.c:152] Absolute Stats: Recv 1'410'053 pkts (0 drops) - Forwarded 8'190 pkts (1'401'863 drops)
14/Nov/2016 09:01:28 [zbalance_ipc.c:224] Actual Stats: Recv 356'860.48 pps (0.00 drops) - Forwarded 0.00 pps (356'868.48 drops)
14/Nov/2016 09:01:28 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:29 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:29 [zbalance_ipc.c:152] Absolute Stats: Recv 1'782'541 pkts (0 drops) - Forwarded 8'190 pkts (1'774'351 drops)
14/Nov/2016 09:01:29 [zbalance_ipc.c:224] Actual Stats: Recv 372'419.84 pps (0.00 drops) - Forwarded 0.00 pps (372'419.84 drops)
14/Nov/2016 09:01:29 [zbalance_ipc.c:234] =========================
Here's strace output of the crash, if that helps any:
write(1, "14/Nov/2016 09:06:12 [n2disk.c:3"..., 9314/Nov/2016 09:06:12 [n2disk.c:3630] Started PF_RING packet reader thread for device zc:eth5
) = 93
stat("/sys/devices/system/node/node0/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node1/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node2/cpumap", 0x7ffc10d5fa60) = -1 ENOENT (No such file or directory)
stat("/sys/devices/system/node/node0/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node1/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node2/cpumap", 0x7ffc10d5fa40) = -1 ENOENT (No such file or directory)
set_mempolicy(MPOL_BIND, 0x2d3b3b0, 4) = 0
socket(0x1b /* PF_??? */, SOCK_RAW, 768) = 5
setsockopt(5, SOL_IP, 0x87 /* IP_??? */, "\0\0\0\0\0\0\0\0", 8) = 0
setsockopt(5, SOL_IP, 0x6e /* IP_??? */, "pfring-zc-cluster-0", 19) = 0
open("/proc/meminfo", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faed3cf3000
read(6, "MemTotal: 16291360 kB\nMemF"..., 1024) = 1024
read(6, "e: 1024\nHugePages_Rsvd: "..., 1024) = 174
close(6) = 0
munmap(0x7faed3cf3000, 4096) = 0
open("/proc/mounts", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faed3cf3000
read(6, "rootfs / rootfs rw 0 0\nproc /pro"..., 1024) = 1024
read(6, "ugepages hugetlbfs rw,relatime 0"..., 1024) = 35
open("/mnt/hugepages/pfring_zc_0", O_RDWR|O_CREAT, 0755) = 7
flock(7, LOCK_SH) = 0
mmap(NULL, 134217728, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_LOCKED, 7, 0) = 0x7fb033200000
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7fb033200018} ---
+++ killed by SIGBUS (core dumped) +++
Thanks,
Derek
________________________________
This e-mail message (including any attachments) is for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution
or copying of this message (including any attachments) is strictly
prohibited.
If you have received this message in error, please contact
the sender by reply e-mail message and destroy all copies of the
original message (including attachments).
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it<mailto:Ntop-misc@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
________________________________
From: ntop-misc-bounces@listgateway.unipi.it <ntop-misc-bounces@listgateway.unipi.it> on behalf of Alfredo Cardigliano <cardigliano@ntop.org>
Sent: Thursday, November 17, 2016 5:49 AM
To: ntop-misc@listgateway.unipi.it
Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64
Exactly, you need a configuration line per CPU node
node=0 hugepagenumber=1024
node=1 hugepagenumber=1024
Alfredo
On 16 Nov 2016, at 20:30, Spransy, Derek <dsprans@emory.edu<mailto:dsprans@emory.edu>> wrote:
Hi Alfredo,
Here's the output:
Node 0 HugePages_Total: 1024
Node 0 HugePages_Free: 1024
Node 0 HugePages_Surp: 0
Node 1 HugePages_Total: 0
Node 1 HugePages_Free: 0
Node 1 HugePages_Surp: 0
Our hugepages.conf consists of:
node=0 hugepagenumber=1024
Do you think this needs to be added for node 1 as well?
Thanks,
Derek
________________________________
From: ntop-misc-bounces@listgateway.unipi.it<mailto:ntop-misc-bounces@listgateway.unipi.it> <ntop-misc-bounces@listgateway.unipi.it<mailto:ntop-misc-bounces@listgateway.unipi.it>> on behalf of Alfredo Cardigliano <cardigliano@ntop.org<mailto:cardigliano@ntop.org>>
Sent: Tuesday, November 15, 2016 9:01 AM
To: ntop-misc@listgateway.unipi.it<mailto:ntop-misc@listgateway.unipi.it>
Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64
Hi Derek
this seems to be related to hugepages, could you provide more info about memory in your system including:
cat /sys/devices/system/node/node*/meminfo | grep Huge
Alfredo
On 14 Nov 2016, at 15:13, Spransy, Derek <dsprans@emory.edu<mailto:dsprans@emory.edu>> wrote:
Hi Alfredo,
Yes the behavior is the same as before:
$ sudo /usr/local/bin/n2disk10g /etc/n2disk/n2disk-eth5.conf
14/Nov/2016 09:04:21 [n2disk.c:4819] Welcome to n2disk10g v.2.6.161111 (r4672) [SandyBridge]
14/Nov/2016 09:04:21 [n2disk.c:4846] Running on 2 node(s) system with 24 core(s). NUMA affinity set to node 1.
14/Nov/2016 09:04:21 [n2disk.c:4875] Using PF_RING for packet capture
14/Nov/2016 09:04:21 [n2disk.c:4901] WARNING: If you are using standard drivers (packet capture via kernel) please disable time-pulse thread
14/Nov/2016 09:04:21 [n2disk.c:4904] Multithread support enabled
14/Nov/2016 09:04:21 [n2disk.c:5018] Dump files max size is set to 1024 MB
14/Nov/2016 09:04:21 [n2disk.c:5035] Buffer memory is set to 3 GB (x 3 pcap files)
14/Nov/2016 09:04:21 [n2disk.c:5070] Using directory /data1/captures for dump files
14/Nov/2016 09:04:21 [n2disk.c:5075] No sub-directories will be created
14/Nov/2016 09:04:21 [n2disk.c:5080] Up to 43000 files will be written before overwriting
14/Nov/2016 09:04:21 [n2disk.c:5090] Dump files max duration is set to 600 sec
14/Nov/2016 09:04:21 [n2disk.c:5106] Dumping data in 0.1 MB chunks
14/Nov/2016 09:04:21 [n2disk.c:5149] Index processing memory is set to 847 MB (x 3 index files)
14/Nov/2016 09:04:24 [n2disk.c:5339] Memory allocated successfully
14/Nov/2016 09:04:24 [n2disk.c:3597] Using time pulse timestamps
14/Nov/2016 09:04:24 [n2disk.c:3630] Started PF_RING packet reader thread for device zc:eth5
$
Running zbalance works fine:
$ sudo /usr/local/bin/zbalance_ipc -i zc:eth5 -c 99 -n 1 -m 1
14/Nov/2016 09:01:24 [zbalance_ipc.c:686] Starting balancer with 1 consumer queues..
14/Nov/2016 09:01:24 [zbalance_ipc.c:696] Run your application instances as follows:
14/Nov/2016 09:01:24 [zbalance_ipc.c:701] pfcount -i zc:99@0
14/Nov/2016 09:01:25 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:25 [zbalance_ipc.c:152] Absolute Stats: Recv 354'797 pkts (0 drops) - Forwarded 8'190 pkts (346'607 drops)
14/Nov/2016 09:01:25 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:26 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:26 [zbalance_ipc.c:152] Absolute Stats: Recv 714'213 pkts (0 drops) - Forwarded 8'190 pkts (706'023 drops)
14/Nov/2016 09:01:26 [zbalance_ipc.c:224] Actual Stats: Recv 359'348.08 pps (0.00 drops) - Forwarded 0.00 pps (359'348.08 drops)
14/Nov/2016 09:01:26 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:27 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:27 [zbalance_ipc.c:152] Absolute Stats: Recv 1'053'124 pkts (0 drops) - Forwarded 8'190 pkts (1'044'926 drops)
14/Nov/2016 09:01:27 [zbalance_ipc.c:224] Actual Stats: Recv 338'839.84 pps (0.00 drops) - Forwarded 0.00 pps (338'831.84 drops)
14/Nov/2016 09:01:27 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:28 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:28 [zbalance_ipc.c:152] Absolute Stats: Recv 1'410'053 pkts (0 drops) - Forwarded 8'190 pkts (1'401'863 drops)
14/Nov/2016 09:01:28 [zbalance_ipc.c:224] Actual Stats: Recv 356'860.48 pps (0.00 drops) - Forwarded 0.00 pps (356'868.48 drops)
14/Nov/2016 09:01:28 [zbalance_ipc.c:234] =========================
14/Nov/2016 09:01:29 [zbalance_ipc.c:151] =========================
14/Nov/2016 09:01:29 [zbalance_ipc.c:152] Absolute Stats: Recv 1'782'541 pkts (0 drops) - Forwarded 8'190 pkts (1'774'351 drops)
14/Nov/2016 09:01:29 [zbalance_ipc.c:224] Actual Stats: Recv 372'419.84 pps (0.00 drops) - Forwarded 0.00 pps (372'419.84 drops)
14/Nov/2016 09:01:29 [zbalance_ipc.c:234] =========================
Here's strace output of the crash, if that helps any:
write(1, "14/Nov/2016 09:06:12 [n2disk.c:3"..., 9314/Nov/2016 09:06:12 [n2disk.c:3630] Started PF_RING packet reader thread for device zc:eth5
) = 93
stat("/sys/devices/system/node/node0/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node1/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node2/cpumap", 0x7ffc10d5fa60) = -1 ENOENT (No such file or directory)
stat("/sys/devices/system/node/node0/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node1/cpumap", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
stat("/sys/devices/system/node/node2/cpumap", 0x7ffc10d5fa40) = -1 ENOENT (No such file or directory)
set_mempolicy(MPOL_BIND, 0x2d3b3b0, 4) = 0
socket(0x1b /* PF_??? */, SOCK_RAW, 768) = 5
setsockopt(5, SOL_IP, 0x87 /* IP_??? */, "\0\0\0\0\0\0\0\0", 8) = 0
setsockopt(5, SOL_IP, 0x6e /* IP_??? */, "pfring-zc-cluster-0", 19) = 0
open("/proc/meminfo", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faed3cf3000
read(6, "MemTotal: 16291360 kB\nMemF"..., 1024) = 1024
read(6, "e: 1024\nHugePages_Rsvd: "..., 1024) = 174
close(6) = 0
munmap(0x7faed3cf3000, 4096) = 0
open("/proc/mounts", O_RDONLY) = 6
fstat(6, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faed3cf3000
read(6, "rootfs / rootfs rw 0 0\nproc /pro"..., 1024) = 1024
read(6, "ugepages hugetlbfs rw,relatime 0"..., 1024) = 35
open("/mnt/hugepages/pfring_zc_0", O_RDWR|O_CREAT, 0755) = 7
flock(7, LOCK_SH) = 0
mmap(NULL, 134217728, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_LOCKED, 7, 0) = 0x7fb033200000
--- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7fb033200018} ---
+++ killed by SIGBUS (core dumped) +++
Thanks,
Derek
________________________________
This e-mail message (including any attachments) is for the sole use of
the intended recipient(s) and may contain confidential and privileged
information. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution
or copying of this message (including any attachments) is strictly
prohibited.
If you have received this message in error, please contact
the sender by reply e-mail message and destroy all copies of the
original message (including attachments).
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it<mailto:Ntop-misc@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop-misc