Mailing List Archive

Hi Mike
could you try running pfcount -i zc:enp1s0 (configuring the interface with a single RSS queue)?

Thank you
Alfredo

> On 15 Dec 2017, at 16:29, Lang, Michael <mike.lang@uconn.edu> wrote:
>
> Hello,
>
> I’m in the process of configuring new 40G hardware with nProbe Cento to replace older hardware with 2 x 10G and nProbe. It is going well however I’m not yet getting zero packet loss which is what I was expecting. Packet loss is low but not zero, about 0.1% with traffic rate 13.5 Gbps, 1.7 Mpps.
>
> Are there tuning recommendations that I can try to achieve zero packet loss?
>
> Here are some details of my environment:
>
> OS: CentOS Linux release 7.4.1708
> Kernel: Linux cento 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> REPO: ntop CentOS Stable
> CPU: Intel(R) Xeon(R) E3-1275 v6 (4 cores + HT = 8 Logical CPUs)
> MEM: 8 GB
> NIC: Intel XL710-QDA1
> NIC Driver: i40e_zc v2.2.4
> RSS: Auto configured to 8
> PFRING: 7.0.0 ($Revision: 7.0.0-stable:f18cdc778a3c957689125dd7c52c40c2277703fa$)
> Huge pages: 2048
> Cento version: nProbe cento v.1.2.171211 1.2-stable:de3cd10fb80990c50e753af6026787fe62b6d2b3:20171211
> Cento Invocation: cento --interface zc:enp1s0@[0-7] --lifetime-timeout 300 --v5 <IP>:<PORT>
>
> CPU Utilization is generally under 2% for all cores except 7 which is around 10%. Memory Utilization is 5.0 GB / 7.43 GB, no swap usage.
>
> - Mike
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>

Hi Alfredo,

Thank you for the response. Here is the output you requested:

[root@madmax ~]# pfcount -i zc:enp1s0
Using PF_RING v.7.0.0
Capturing from zc:enp1s0 [mac: 3C:FD:FE:A2:B9:58][if_index: 5][speed: 40000Mb/s]
# Device RX channels: 1
# Polling threads: 1
Dumping statistics on /proc/net/pf_ring/stats/7413-enp1s0.682
=========================
Absolute Stats: [1'596'557 pkts total][1'123'809 pkts dropped][70.4% dropped]
[472'748 pkts rcvd][478'241'120 bytes rcvd]
=========================

=========================
Absolute Stats: [3'105'703 pkts total][2'154'407 pkts dropped][69.4% dropped]
[951'296 pkts rcvd][959'971'922 bytes rcvd][951'229.41 pkt/sec][7'679.23 Mbit/sec]
=========================
Actual Stats: [478'548 pkts rcvd][1'000.07 ms][478'514.50 pps][3.85 Gbps]
=========================

=========================
Absolute Stats: [4'753'113 pkts total][3'317'837 pkts dropped][69.8% dropped]
[1'435'276 pkts rcvd][1'449'061'259 bytes rcvd][717'576.28 pkt/sec][5'795.74 Mbit/sec]
=========================
Actual Stats: [483'980 pkts rcvd][1'000.10 ms][483'930.63 pps][3.91 Gbps]
=========================

=========================
Absolute Stats: [6'388'321 pkts total][4'472'657 pkts dropped][70.0% dropped]
[1'915'664 pkts rcvd][1'932'853'839 bytes rcvd][638'461.66 pkt/sec][5'153.52 Mbit/sec]
=========================
Actual Stats: [480'388 pkts rcvd][1'000.26 ms][480'260.73 pps][3.87 Gbps]
=========================

=========================
Absolute Stats: [8'078'586 pkts total][5'674'602 pkts dropped][70.2% dropped]
[2'403'984 pkts rcvd][2'424'236'935 bytes rcvd][600'918.63 pkt/sec][4'847.84 Mbit/sec]
=========================
Actual Stats: [488'320 pkts rcvd][1'000.07 ms][488'281.91 pps][3.93 Gbps]
=========================

=========================
Absolute Stats: [9'862'549 pkts total][6'968'657 pkts dropped][70.7% dropped]
[2'893'892 pkts rcvd][2'917'179'126 bytes rcvd][578'701.89 pkt/sec][4'666.86 Mbit/sec]
=========================
Actual Stats: [489'908 pkts rcvd][1'000.14 ms][489'836.48 pps][3.94 Gbps]
=========================

=========================
Absolute Stats: [11'655'152 pkts total][8'271'880 pkts dropped][71.0% dropped]
[3'383'272 pkts rcvd][3'412'279'814 bytes rcvd][563'801.98 pkt/sec][4'549.08 Mbit/sec]
=========================
Actual Stats: [489'380 pkts rcvd][1'000.15 ms][489'304.15 pps][3.96 Gbps]


- Mike


From: ntop-misc-bounces@listgateway.unipi.it [mailto:ntop-misc-bounces@listgateway.unipi.it] On Behalf Of Alfredo Cardigliano
Sent: Friday, December 15, 2017 10:57 AM
To: ntop-misc@listgateway.unipi.it
Subject: Re: [Ntop-misc] nProbe Cento Performance Tuning

Hi Mike
could you try running pfcount -i zc:enp1s0 (configuring the interface with a single RSS queue)?

Thank you
Alfredo

On 15 Dec 2017, at 16:29, Lang, Michael <mike.lang@uconn.edu<mailto:mike.lang@uconn.edu>> wrote:

Hello,

I'm in the process of configuring new 40G hardware with nProbe Cento to replace older hardware with 2 x 10G and nProbe. It is going well however I'm not yet getting zero packet loss which is what I was expecting. Packet loss is low but not zero, about 0.1% with traffic rate 13.5 Gbps, 1.7 Mpps.

Are there tuning recommendations that I can try to achieve zero packet loss?

Here are some details of my environment:

OS: CentOS Linux release 7.4.1708
Kernel: Linux cento 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
REPO: ntop CentOS Stable
CPU: Intel(R) Xeon(R) E3-1275 v6 (4 cores + HT = 8 Logical CPUs)
MEM: 8 GB
NIC: Intel XL710-QDA1
NIC Driver: i40e_zc v2.2.4
RSS: Auto configured to 8
PFRING: 7.0.0 ($Revision: 7.0.0-stable:f18cdc778a3c957689125dd7c52c40c2277703fa$)
Huge pages: 2048
Cento version: nProbe cento v.1.2.171211 1.2-stable:de3cd10fb80990c50e753af6026787fe62b6d2b3:20171211
Cento Invocation: cento --interface zc:enp1s0@[0-7] --lifetime-timeout 300 --v5 <IP>:<PORT>

CPU Utilization is generally under 2% for all cores except 7 which is around 10%. Memory Utilization is 5.0 GB / 7.43 GB, no swap usage.

- Mike
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it<mailto:Ntop-misc@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

What happens if you add -a to pfcount?

Alfredo

> On 15 Dec 2017, at 20:25, Lang, Michael <mike.lang@uconn.edu> wrote:
>
> Hi Alfredo,
>
> Thank you for the response. Here is the output you requested:
>
> [root@madmax ~]# pfcount -i zc:enp1s0
> Using PF_RING v.7.0.0
> Capturing from zc:enp1s0 [mac: 3C:FD:FE:A2:B9:58][if_index: 5][speed: 40000Mb/s]
> # Device RX channels: 1
> # Polling threads: 1
> Dumping statistics on /proc/net/pf_ring/stats/7413-enp1s0.682
> =========================
> Absolute Stats: [1'596'557 pkts total][1'123'809 pkts dropped][70.4% dropped]
> [472'748 pkts rcvd][478'241'120 bytes rcvd]
> =========================
>
> =========================
> Absolute Stats: [3'105'703 pkts total][2'154'407 pkts dropped][69.4% dropped]
> [951'296 pkts rcvd][959'971'922 bytes rcvd][951'229.41 pkt/sec][7'679.23 Mbit/sec]
> =========================
> Actual Stats: [478'548 pkts rcvd][1'000.07 ms][478'514.50 pps][3.85 Gbps]
> =========================

Alfredo,

Here is what that looks like (much better, I don't understand why):

[root@madmax ~]# pfcount -i zc:enp1s0 -a
Using PF_RING v.7.0.0
Capturing from zc:enp1s0 [mac: 3C:FD:FE:A2:B9:58][if_index: 7][speed: 40000Mb/s]
# Device RX channels: 1
# Polling threads: 1
Dumping statistics on /proc/net/pf_ring/stats/7583-enp1s0.700
=========================
Absolute Stats: [1'136'044 pkts total][0 pkts dropped][0.0% dropped]
[1'136'044 pkts rcvd][1'066'666'346 bytes rcvd]
=========================

=========================
Absolute Stats: [2'319'312 pkts total][0 pkts dropped][0.0% dropped]
[2'319'312 pkts rcvd][2'194'636'588 bytes rcvd][2'319'247.06 pkt/sec][17'556.60 Mbit/sec]
=========================
Actual Stats: [1'183'268 pkts rcvd][1'000.02 ms][1'183'234.86 pps][9.02 Gbps]
=========================

=========================
Absolute Stats: [3'439'460 pkts total][0 pkts dropped][0.0% dropped]
[3'439'460 pkts rcvd][3'260'323'805 bytes rcvd][1'719'672.39 pkt/sec][13'040.85 Mbit/sec]
=========================
Actual Stats: [1'120'148 pkts rcvd][1'000.03 ms][1'120'104.31 pps][8.53 Gbps]
=========================

=========================
Absolute Stats: [4'591'564 pkts total][0 pkts dropped][0.0% dropped]
[4'591'564 pkts rcvd][4'352'135'503 bytes rcvd][1'530'475.41 pkt/sec][11'605.34 Mbit/sec]
=========================
Actual Stats: [1'152'104 pkts rcvd][1'000.02 ms][1'152'077.50 pps][8.73 Gbps]
=========================

=========================
Absolute Stats: [5'761'348 pkts total][0 pkts dropped][0.0% dropped]
[5'761'348 pkts rcvd][5'457'944'041 bytes rcvd][1'440'296.67 pkt/sec][10'915.58 Mbit/sec]
=========================
Actual Stats: [1'169'784 pkts rcvd][1'000.02 ms][1'169'758.26 pps][8.85 Gbps]
=========================

=========================
Absolute Stats: [6'992'356 pkts total][0 pkts dropped][0.0% dropped]
[6'992'356 pkts rcvd][6'634'827'335 bytes rcvd][1'398'433.44 pkt/sec][10'615.43 Mbit/sec]
=========================
Actual Stats: [1'231'008 pkts rcvd][1'000.02 ms][1'230'979.68 pps][9.41 Gbps]
=========================

^CLeaving...
=========================
Absolute Stats: [7'437'880 pkts total][0 pkts dropped][0.0% dropped]
[7'437'880 pkts rcvd][7'054'472'445 bytes rcvd][1'386'151.05 pkt/sec][10'517.58 Mbit/sec]
=========================
Actual Stats: [445'524 pkts rcvd][365.72 ms][1'218'223.97 pps][9.18 Gbps]


- Mike


From: ntop-misc-bounces@listgateway.unipi.it [mailto:ntop-misc-bounces@listgateway.unipi.it] On Behalf Of Alfredo Cardigliano
Sent: Friday, December 15, 2017 3:44 PM
To: ntop-misc@listgateway.unipi.it
Subject: Re: [Ntop-misc] nProbe Cento Performance Tuning

What happens if you add -a to pfcount?

Alfredo

On 15 Dec 2017, at 20:25, Lang, Michael <mike.lang@uconn.edu<mailto:mike.lang@uconn.edu>> wrote:

Hi Alfredo,

Thank you for the response. Here is the output you requested:

[root@madmax ~]# pfcount -i zc:enp1s0
Using PF_RING v.7.0.0
Capturing from zc:enp1s0 [mac: 3C:FD:FE:A2:B9:58][if_index: 5][speed: 40000Mb/s]
# Device RX channels: 1
# Polling threads: 1
Dumping statistics on /proc/net/pf_ring/stats/7413-enp1s0.682
=========================
Absolute Stats: [1'596'557 pkts total][1'123'809 pkts dropped][70.4% dropped]
[472'748 pkts rcvd][478'241'120 bytes rcvd]
=========================

=========================
Absolute Stats: [3'105'703 pkts total][2'154'407 pkts dropped][69.4% dropped]
[951'296 pkts rcvd][959'971'922 bytes rcvd][951'229.41 pkt/sec][7'679.23 Mbit/sec]
=========================
Actual Stats: [478'548 pkts rcvd][1'000.07 ms][478'514.50 pps][3.85 Gbps]
=========================

Hi Mike
it seems that running in passive wait is causing some loss, this is due to the
poor buffering power in i40e cards: when there is no packets, the application
calls poll() to wait for new packets until an interrupt is raised, then it wakes up
and continue processing traffic. What happens is that the buffer fills up during
this period of time, discarding some packet.
In essence it could be that 1. the poll/wake up period takes too much in passive
wait, or 2. the cpu is in power saving mode slowing down processing (please
check that it is set to “performance” in your Bios) when there is no much to do.
This also depends a lot on the traffic rate, we will run some test to check how
poll behaves at that rate.

Note that this does not happen with -a as the application is not calling poll(),
but it’s just spinning on the cpu.

Alfredo

> On 15 Dec 2017, at 22:00, Lang, Michael <mike.lang@uconn.edu> wrote:
>
> Alfredo,
>
> Here is what that looks like (much better, I don’t understand why):
>