Mailing List Archive: sar output/zbalance

sar output/zbalance_ipc

Jun 18, 2019, 3:11 PM

Post #1 of 8 (2516 views)

I've noticed some strange output from sar when running zbalance_ipc.
It seems I only get stats every other second, on odd seconds in this
case:

% sar -n DEV 1 10 | grep ens5f0
05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00

Checking for drops, I see them on the even seconds:

% sar -n EDEV 1 10 | grep ens5f0
05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00

ethtool is reporting a lot of missed packets:

% ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
rx_packets: 1775168454257
rx_errors: 0
tx_errors: 0
rx_dropped: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 1031957653637
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
rx_length_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_csum_offload_errors: 13923182
fcoe_last_errors: 0

zbalance_ipc :

/usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p

48 snorts are running along with zbalance_ipc .

Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
or are there config changes I should make?

Card info :

81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Thanks,
--
Jim Hranicky
Data Security Specialist
UF Information Technology

Re: sar output/zbalance_ipc [ In reply to ]

jfh at ufl

Jun 18, 2019, 4:30 PM

Post #2 of 8 (2515 views)

Permalink

https://lists.gt.net/ntop/misc/48984

After fighting with the fucking rules (copied from the master!
Shit, kept overwriting my files. That's ok, local rules are
rolled up into the downloaded rules file - NOT), I finally
checked the UFIRT rule with snort and suri. 100 hits to
one of the FSU servers, 1/sec. suri with pf-ring caught
all 100, af-packet caught 97. snort on ewansens2 only caught
75.

May be some changes on the way for ewansens2.

Also, I added a bpf rule ('not esp') to block that encapsulated
IPSEC crap. Doesn't seem to be working, though.

Not feeling too swell right now, may not be in tomorrow. If not,
good luck on all this crap. I'll let you know if I'm going to
be fucking with the rules from home if I don't come in.

Jim

On 6/18/19 6:11 PM, Jim Hranicky wrote:
> I've noticed some strange output from sar when running zbalance_ipc.
> It seems I only get stats every other second, on odd seconds in this
> case:
>
> % sar -n DEV 1 10 | grep ens5f0
> 05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
> 05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
> 05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
> 05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
> 05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
> 05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
> 05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00
>
> Checking for drops, I see them on the even seconds:
>
> % sar -n EDEV 1 10 | grep ens5f0
> 05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
> 05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
> 05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
> 05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
> 05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
> 05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
> Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00
>
> ethtool is reporting a lot of missed packets:
>
> % ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
> rx_packets: 1775168454257
> rx_errors: 0
> tx_errors: 0
> rx_dropped: 0
> rx_over_errors: 0
> rx_crc_errors: 0
> rx_frame_errors: 0
> rx_fifo_errors: 0
> rx_missed_errors: 1031957653637
> tx_aborted_errors: 0
> tx_carrier_errors: 0
> tx_fifo_errors: 0
> tx_heartbeat_errors: 0
> rx_length_errors: 0
> rx_long_length_errors: 0
> rx_short_length_errors: 0
> rx_csum_offload_errors: 13923182
> fcoe_last_errors: 0
>
> zbalance_ipc :
>
> /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>
> 48 snorts are running along with zbalance_ipc .
>
> Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
> or are there config changes I should make?
>
> Card info :
>
> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>
> Thanks,
> --
> Jim Hranicky
> Data Security Specialist
> UF Information Technology
>

Re: sar output/zbalance_ipc [ In reply to ]

jfh at ufl

Jun 18, 2019, 4:32 PM

Post #3 of 8 (2515 views)

Permalink

OMG, this wasn't supposed to go to the list. I'm so sorry :(

Jim

On 6/18/19 7:30 PM, Jim Hranicky wrote:
> https://lists.gt.net/ntop/misc/48984

Re: sar output/zbalance_ipc [ In reply to ]

cardigliano at ntop

Jun 20, 2019, 1:26 AM

Post #4 of 8 (2507 views)

Permalink

Hi Jim
please note that you are not using ZC, your adapter is supported by the ixgbe-zc driver,
for best performance please configure it following this guide: http://www.ntop.org/guides/pf_ring/get_started/packages_installation.html <http://www.ntop.org/guides/pf_ring/get_started/packages_installation.html>

Regards
Alfredo

> On 19 Jun 2019, at 00:11, Jim Hranicky <jfh@ufl.edu> wrote:
>
> I've noticed some strange output from sar when running zbalance_ipc.
> It seems I only get stats every other second, on odd seconds in this
> case:
>
> % sar -n DEV 1 10 | grep ens5f0
> 05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
> 05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
> 05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
> 05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
> 05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
> 05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
> 05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00
>
> Checking for drops, I see them on the even seconds:
>
> % sar -n EDEV 1 10 | grep ens5f0
> 05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
> 05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
> 05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
> 05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
> 05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
> 05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
> Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00
>
> ethtool is reporting a lot of missed packets:
>
> % ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
> rx_packets: 1775168454257
> rx_errors: 0
> tx_errors: 0
> rx_dropped: 0
> rx_over_errors: 0
> rx_crc_errors: 0
> rx_frame_errors: 0
> rx_fifo_errors: 0
> rx_missed_errors: 1031957653637
> tx_aborted_errors: 0
> tx_carrier_errors: 0
> tx_fifo_errors: 0
> tx_heartbeat_errors: 0
> rx_length_errors: 0
> rx_long_length_errors: 0
> rx_short_length_errors: 0
> rx_csum_offload_errors: 13923182
> fcoe_last_errors: 0
>
> zbalance_ipc :
>
> /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>
> 48 snorts are running along with zbalance_ipc .
>
> Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
> or are there config changes I should make?
>
> Card info :
>
> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>
> Thanks,
> --
> Jim Hranicky
> Data Security Specialist
> UF Information Technology
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: sar output/zbalance_ipc [ In reply to ]

jfh at ufl

Jun 20, 2019, 12:30 PM

Post #5 of 8 (2501 views)

Permalink

I seem to be :

% lsmod | egrep 'pf_ring|ixgbe'
ixgbe_zc 339253 0
pf_ring 1246667 103 ixgbe_zc
ptp 19231 2 tg3,ixgbe_zc

Still seeing the same behavior with sar:

<root@ewansens2|~> # sar -n DEV 1 4 | grep ens5f0
03:27:05 PM ens5f0 1074552.00 0.00 2394420.68 0.00 0.00 0.00 3.00
03:27:06 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:27:07 PM ens5f0 1093220.00 0.00 2394094.27 0.00 0.00 0.00 8.00
03:27:08 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: ens5f0 541943.00 0.00 1197128.74 0.00 0.00 0.00 2.75

<root@ewansens2|~> # sar -n EDEV 1 4 | grep ens5f0
03:27:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:27:17 PM ens5f0 0.00 0.00 0.00 1646283.00 0.00 0.00 0.00 0.00 0.00
03:27:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:27:19 PM ens5f0 0.00 0.00 0.00 1618365.00 0.00 0.00 0.00 0.00 0.00
Average: ens5f0 0.00 0.00 0.00 816162.00 0.00 0.00 0.00 0.00 0.00

zbalance_ipc is running without throwing any errors:

% ps -ef | grep zbalance
root 7554 99 06:27 ? 12:12:02 /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p

Output at start:

20/Jun/2019 06:27:29 [zbalance_ipc.c:1116] Starting balancer with 48 consumer queues..
20/Jun/2019 06:27:29 [zbalance_ipc.c:1126] Run your application instances as follows:
20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@0
20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@1

Jim

On 6/20/19 4:26 AM, Alfredo Cardigliano wrote:
> Hi Jim
> please note that you are not using ZC, your adapter is supported by the ixgbe-zc driver,
> for best performance please configure it following this guide: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= >
>
> Regards
> Alfredo
>
>> On 19 Jun 2019, at 00:11, Jim Hranicky <jfh@ufl.edu> wrote:
>>
>> I've noticed some strange output from sar when running zbalance_ipc.
>> It seems I only get stats every other second, on odd seconds in this
>> case:
>>
>> % sar -n DEV 1 10 | grep ens5f0
>> 05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
>> 05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
>> 05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
>> 05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
>> 05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
>> 05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
>> 05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00
>>
>> Checking for drops, I see them on the even seconds:
>>
>> % sar -n EDEV 1 10 | grep ens5f0
>> 05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
>> 05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
>> Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00
>>
>> ethtool is reporting a lot of missed packets:
>>
>> % ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
>> rx_packets: 1775168454257
>> rx_errors: 0
>> tx_errors: 0
>> rx_dropped: 0
>> rx_over_errors: 0
>> rx_crc_errors: 0
>> rx_frame_errors: 0
>> rx_fifo_errors: 0
>> rx_missed_errors: 1031957653637
>> tx_aborted_errors: 0
>> tx_carrier_errors: 0
>> tx_fifo_errors: 0
>> tx_heartbeat_errors: 0
>> rx_length_errors: 0
>> rx_long_length_errors: 0
>> rx_short_length_errors: 0
>> rx_csum_offload_errors: 13923182
>> fcoe_last_errors: 0
>>
>> zbalance_ipc :
>>
>> /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>>
>> 48 snorts are running along with zbalance_ipc .
>>
>> Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
>> or are there config changes I should make?
>>
>> Card info :
>>
>> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>
>> Thanks,
>> --
>> Jim Hranicky
>> Data Security Specialist
>> UF Information Technology
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: sar output/zbalance_ipc [ In reply to ]

jfh at ufl

Jun 20, 2019, 3:04 PM

Post #6 of 8 (2501 views)

Permalink

Should have sent this in. PF_RING version v.7.5.0.190404 .

Jim

On 6/20/19 3:30 PM, Jim Hranicky wrote:
> I seem to be :
>
> % lsmod | egrep 'pf_ring|ixgbe'
> ixgbe_zc 339253 0
> pf_ring 1246667 103 ixgbe_zc
> ptp 19231 2 tg3,ixgbe_zc
>
> Still seeing the same behavior with sar:
>
> <root@ewansens2|~> # sar -n DEV 1 4 | grep ens5f0
> 03:27:05 PM ens5f0 1074552.00 0.00 2394420.68 0.00 0.00 0.00 3.00
> 03:27:06 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 03:27:07 PM ens5f0 1093220.00 0.00 2394094.27 0.00 0.00 0.00 8.00
> 03:27:08 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> Average: ens5f0 541943.00 0.00 1197128.74 0.00 0.00 0.00 2.75
>
> <root@ewansens2|~> # sar -n EDEV 1 4 | grep ens5f0
> 03:27:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 03:27:17 PM ens5f0 0.00 0.00 0.00 1646283.00 0.00 0.00 0.00 0.00 0.00
> 03:27:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 03:27:19 PM ens5f0 0.00 0.00 0.00 1618365.00 0.00 0.00 0.00 0.00 0.00
> Average: ens5f0 0.00 0.00 0.00 816162.00 0.00 0.00 0.00 0.00 0.00
>
> zbalance_ipc is running without throwing any errors:
>
> % ps -ef | grep zbalance
> root 7554 99 06:27 ? 12:12:02 /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>
> Output at start:
>
> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1116] Starting balancer with 48 consumer queues..
> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1126] Run your application instances as follows:
> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@0
> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@1
>
> Jim
>
> On 6/20/19 4:26 AM, Alfredo Cardigliano wrote:
>> Hi Jim
>> please note that you are not using ZC, your adapter is supported by the ixgbe-zc driver,
>> for best performance please configure it following this guide: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= >
>>
>> Regards
>> Alfredo
>>
>>> On 19 Jun 2019, at 00:11, Jim Hranicky <jfh@ufl.edu> wrote:
>>>
>>> I've noticed some strange output from sar when running zbalance_ipc.
>>> It seems I only get stats every other second, on odd seconds in this
>>> case:
>>>
>>> % sar -n DEV 1 10 | grep ens5f0
>>> 05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
>>> 05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
>>> 05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
>>> 05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
>>> 05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
>>> 05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
>>> 05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00
>>>
>>> Checking for drops, I see them on the even seconds:
>>>
>>> % sar -n EDEV 1 10 | grep ens5f0
>>> 05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
>>> 05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
>>> Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00
>>>
>>> ethtool is reporting a lot of missed packets:
>>>
>>> % ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
>>> rx_packets: 1775168454257
>>> rx_errors: 0
>>> tx_errors: 0
>>> rx_dropped: 0
>>> rx_over_errors: 0
>>> rx_crc_errors: 0
>>> rx_frame_errors: 0
>>> rx_fifo_errors: 0
>>> rx_missed_errors: 1031957653637
>>> tx_aborted_errors: 0
>>> tx_carrier_errors: 0
>>> tx_fifo_errors: 0
>>> tx_heartbeat_errors: 0
>>> rx_length_errors: 0
>>> rx_long_length_errors: 0
>>> rx_short_length_errors: 0
>>> rx_csum_offload_errors: 13923182
>>> fcoe_last_errors: 0
>>>
>>> zbalance_ipc :
>>>
>>> /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>>>
>>> 48 snorts are running along with zbalance_ipc .
>>>
>>> Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
>>> or are there config changes I should make?
>>>
>>> Card info :
>>>
>>> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>>
>>> Thanks,
>>> --
>>> Jim Hranicky
>>> Data Security Specialist
>>> UF Information Technology
>>>
>>> _______________________________________________
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>>
>>
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=LF-8RGmNT3RBgHTIWnB5S_YucaJYB9eu6KGxofRcOCk&s=MvuxbrqnKoewXCWvX4mpbE3WtGKBwJ9n2vSoGAxPPBI&e=
>
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: sar output/zbalance_ipc [ In reply to ]

cardigliano at ntop

Jun 21, 2019, 12:42 AM

Post #7 of 8 (2501 views)

Permalink

Hi Jim
in order to enable ZC you should add the zc: prefix to the interface name:

zbalance_ipc -i zc:ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p

Alfredo

> On 21 Jun 2019, at 00:04, Jim Hranicky <jfh@ufl.edu> wrote:
>
> Should have sent this in. PF_RING version v.7.5.0.190404 .
>
> Jim
>
> On 6/20/19 3:30 PM, Jim Hranicky wrote:
>> I seem to be :
>>
>> % lsmod | egrep 'pf_ring|ixgbe'
>> ixgbe_zc 339253 0
>> pf_ring 1246667 103 ixgbe_zc
>> ptp 19231 2 tg3,ixgbe_zc
>>
>> Still seeing the same behavior with sar:
>>
>> <root@ewansens2|~> # sar -n DEV 1 4 | grep ens5f0
>> 03:27:05 PM ens5f0 1074552.00 0.00 2394420.68 0.00 0.00 0.00 3.00
>> 03:27:06 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 03:27:07 PM ens5f0 1093220.00 0.00 2394094.27 0.00 0.00 0.00 8.00
>> 03:27:08 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> Average: ens5f0 541943.00 0.00 1197128.74 0.00 0.00 0.00 2.75
>>
>> <root@ewansens2|~> # sar -n EDEV 1 4 | grep ens5f0
>> 03:27:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 03:27:17 PM ens5f0 0.00 0.00 0.00 1646283.00 0.00 0.00 0.00 0.00 0.00
>> 03:27:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 03:27:19 PM ens5f0 0.00 0.00 0.00 1618365.00 0.00 0.00 0.00 0.00 0.00
>> Average: ens5f0 0.00 0.00 0.00 816162.00 0.00 0.00 0.00 0.00 0.00
>>
>> zbalance_ipc is running without throwing any errors:
>>
>> % ps -ef | grep zbalance
>> root 7554 99 06:27 ? 12:12:02 /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>>
>> Output at start:
>>
>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1116] Starting balancer with 48 consumer queues..
>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1126] Run your application instances as follows:
>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@0
>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@1
>>
>> Jim
>>
>> On 6/20/19 4:26 AM, Alfredo Cardigliano wrote:
>>> Hi Jim
>>> please note that you are not using ZC, your adapter is supported by the ixgbe-zc driver,
>>> for best performance please configure it following this guide: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= >
>>>
>>> Regards
>>> Alfredo
>>>
>>>> On 19 Jun 2019, at 00:11, Jim Hranicky <jfh@ufl.edu> wrote:
>>>>
>>>> I've noticed some strange output from sar when running zbalance_ipc.
>>>> It seems I only get stats every other second, on odd seconds in this
>>>> case:
>>>>
>>>> % sar -n DEV 1 10 | grep ens5f0
>>>> 05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
>>>> 05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
>>>> 05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
>>>> 05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
>>>> 05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
>>>> 05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
>>>> 05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00
>>>>
>>>> Checking for drops, I see them on the even seconds:
>>>>
>>>> % sar -n EDEV 1 10 | grep ens5f0
>>>> 05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
>>>> 05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
>>>> Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00
>>>>
>>>> ethtool is reporting a lot of missed packets:
>>>>
>>>> % ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
>>>> rx_packets: 1775168454257
>>>> rx_errors: 0
>>>> tx_errors: 0
>>>> rx_dropped: 0
>>>> rx_over_errors: 0
>>>> rx_crc_errors: 0
>>>> rx_frame_errors: 0
>>>> rx_fifo_errors: 0
>>>> rx_missed_errors: 1031957653637
>>>> tx_aborted_errors: 0
>>>> tx_carrier_errors: 0
>>>> tx_fifo_errors: 0
>>>> tx_heartbeat_errors: 0
>>>> rx_length_errors: 0
>>>> rx_long_length_errors: 0
>>>> rx_short_length_errors: 0
>>>> rx_csum_offload_errors: 13923182
>>>> fcoe_last_errors: 0
>>>>
>>>> zbalance_ipc :
>>>>
>>>> /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>>>>
>>>> 48 snorts are running along with zbalance_ipc .
>>>>
>>>> Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
>>>> or are there config changes I should make?
>>>>
>>>> Card info :
>>>>
>>>> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>>>
>>>> Thanks,
>>>> --
>>>> Jim Hranicky
>>>> Data Security Specialist
>>>> UF Information Technology
>>>>
>>>> _______________________________________________
>>>> Ntop-misc mailing list
>>>> Ntop-misc@listgateway.unipi.it
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>>>
>>>
>>>
>>> _______________________________________________
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=LF-8RGmNT3RBgHTIWnB5S_YucaJYB9eu6KGxofRcOCk&s=MvuxbrqnKoewXCWvX4mpbE3WtGKBwJ9n2vSoGAxPPBI&e=
>>

_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: sar output/zbalance_ipc [ In reply to ]

jfh at ufl

Jun 21, 2019, 9:33 AM

Post #8 of 8 (2501 views)

Permalink

Ok, thanks. I seem to remember at some point getting an error
doing that, so I changed it to the non-zc version thinking
perhaps the syntax had changed. I'll switch it back, thanks
for your help.

Jim

On 6/21/19 3:42 AM, Alfredo Cardigliano wrote:
> Hi Jim
> in order to enable ZC you should add the zc: prefix to the interface name:
>
> zbalance_ipc -i zc:ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>
> Alfredo
>
>> On 21 Jun 2019, at 00:04, Jim Hranicky <jfh@ufl.edu> wrote:
>>
>> Should have sent this in. PF_RING version v.7.5.0.190404 .
>>
>> Jim
>>
>> On 6/20/19 3:30 PM, Jim Hranicky wrote:
>>> I seem to be :
>>>
>>> % lsmod | egrep 'pf_ring|ixgbe'
>>> ixgbe_zc 339253 0
>>> pf_ring 1246667 103 ixgbe_zc
>>> ptp 19231 2 tg3,ixgbe_zc
>>>
>>> Still seeing the same behavior with sar:
>>>
>>> <root@ewansens2|~> # sar -n DEV 1 4 | grep ens5f0
>>> 03:27:05 PM ens5f0 1074552.00 0.00 2394420.68 0.00 0.00 0.00 3.00
>>> 03:27:06 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 03:27:07 PM ens5f0 1093220.00 0.00 2394094.27 0.00 0.00 0.00 8.00
>>> 03:27:08 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> Average: ens5f0 541943.00 0.00 1197128.74 0.00 0.00 0.00 2.75
>>>
>>> <root@ewansens2|~> # sar -n EDEV 1 4 | grep ens5f0
>>> 03:27:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 03:27:17 PM ens5f0 0.00 0.00 0.00 1646283.00 0.00 0.00 0.00 0.00 0.00
>>> 03:27:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> 03:27:19 PM ens5f0 0.00 0.00 0.00 1618365.00 0.00 0.00 0.00 0.00 0.00
>>> Average: ens5f0 0.00 0.00 0.00 816162.00 0.00 0.00 0.00 0.00 0.00
>>>
>>> zbalance_ipc is running without throwing any errors:
>>>
>>> % ps -ef | grep zbalance
>>> root 7554 99 06:27 ? 12:12:02 /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>>>
>>> Output at start:
>>>
>>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1116] Starting balancer with 48 consumer queues..
>>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1126] Run your application instances as follows:
>>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@0
>>> 20/Jun/2019 06:27:29 [zbalance_ipc.c:1131] pfcount -i zc:99@1
>>>
>>> Jim
>>>
>>> On 6/20/19 4:26 AM, Alfredo Cardigliano wrote:
>>>> Hi Jim
>>>> please note that you are not using ZC, your adapter is supported by the ixgbe-zc driver,
>>>> for best performance please configure it following this guide: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ntop.org_guides_pf-5Fring_get-5Fstarted_packages-5Finstallation.html&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=EViEEq_oXhcA_9Idbzm7eCsxLYKENRsp1G6_7HRV4aU&e= >
>>>>
>>>> Regards
>>>> Alfredo
>>>>
>>>>> On 19 Jun 2019, at 00:11, Jim Hranicky <jfh@ufl.edu> wrote:
>>>>>
>>>>> I've noticed some strange output from sar when running zbalance_ipc.
>>>>> It seems I only get stats every other second, on odd seconds in this
>>>>> case:
>>>>>
>>>>> % sar -n DEV 1 10 | grep ens5f0
>>>>> 05:58:09 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
>>>>> 05:58:09 PM ens5f0 1001033.00 0.00 1440901.59 0.00 0.00 0.00 20.00
>>>>> 05:58:10 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:11 PM ens5f0 1024305.00 0.00 1422458.64 0.00 0.00 0.00 24.00
>>>>> 05:58:12 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:13 PM ens5f0 1028284.00 0.00 1567748.74 0.00 0.00 0.00 36.00
>>>>> 05:58:14 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:15 PM ens5f0 1037512.00 0.00 1568361.43 0.00 0.00 0.00 20.00
>>>>> 05:58:16 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:17 PM ens5f0 1009894.00 0.00 1482515.69 0.00 0.00 0.00 10.00
>>>>> 05:58:18 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> Average: ens5f0 510102.80 0.00 748198.61 0.00 0.00 0.00 11.00
>>>>>
>>>>> Checking for drops, I see them on the even seconds:
>>>>>
>>>>> % sar -n EDEV 1 10 | grep ens5f0
>>>>> 05:58:38 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
>>>>> 05:58:39 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:40 PM ens5f0 0.00 0.00 0.00 629536.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:41 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:42 PM ens5f0 0.00 0.00 0.00 517831.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:43 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:44 PM ens5f0 0.00 0.00 0.00 558658.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:45 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:46 PM ens5f0 0.00 0.00 0.00 595583.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:47 PM ens5f0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>>>> 05:58:48 PM ens5f0 0.00 0.00 0.00 588821.00 0.00 0.00 0.00 0.00 0.00
>>>>> Average: ens5f0 0.00 0.00 0.00 289042.90 0.00 0.00 0.00 0.00 0.00
>>>>>
>>>>> ethtool is reporting a lot of missed packets:
>>>>>
>>>>> % ethtool -S ens5f0 | egrep 'rx_dropped|rx_missed|rx_packets|errors'
>>>>> rx_packets: 1775168454257
>>>>> rx_errors: 0
>>>>> tx_errors: 0
>>>>> rx_dropped: 0
>>>>> rx_over_errors: 0
>>>>> rx_crc_errors: 0
>>>>> rx_frame_errors: 0
>>>>> rx_fifo_errors: 0
>>>>> rx_missed_errors: 1031957653637
>>>>> tx_aborted_errors: 0
>>>>> tx_carrier_errors: 0
>>>>> tx_fifo_errors: 0
>>>>> tx_heartbeat_errors: 0
>>>>> rx_length_errors: 0
>>>>> rx_long_length_errors: 0
>>>>> rx_short_length_errors: 0
>>>>> rx_csum_offload_errors: 13923182
>>>>> fcoe_last_errors: 0
>>>>>
>>>>> zbalance_ipc :
>>>>>
>>>>> /usr/local/pf/sbin/zbalance_ipc -i ens5f0 -m 4 -n 48 -c 99 -g 70 -S 71 -p
>>>>>
>>>>> 48 snorts are running along with zbalance_ipc .
>>>>>
>>>>> Can anyone account for this behavior? Is zbalance_ipc unable to keep up,
>>>>> or are there config changes I should make?
>>>>>
>>>>> Card info :
>>>>>
>>>>> 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> Jim Hranicky
>>>>> Data Security Specialist
>>>>> UF Information Technology
>>>>>
>>>>> _______________________________________________
>>>>> Ntop-misc mailing list
>>>>> Ntop-misc@listgateway.unipi.it
>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwIFAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Ntop-misc mailing list
>>>> Ntop-misc@listgateway.unipi.it
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=aQ70XP8PJ2JSfZ7Uq3B3nHVW3Odt3gEkhvAOp5O6PiI&s=ZX__Hj4a-geUCeo2W2b3gpN4o2qSmb8Hiw3aNrGZeD8&e=
>>>>
>>> _______________________________________________
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=LF-8RGmNT3RBgHTIWnB5S_YucaJYB9eu6KGxofRcOCk&s=MvuxbrqnKoewXCWvX4mpbE3WtGKBwJ9n2vSoGAxPPBI&e=
>>>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> https://urldefense.proofpoint.com/v2/url?u=http-3A__listgateway.unipi.it_mailman_listinfo_ntop-2Dmisc&d=DwICAg&c=sJ6xIWYx-zLMB3EPkvcnVg&r=V4XBkE-sKNhwbB4FTyGW0Q&m=bvqSrb7jiisHF6-mirq5DVlfE75-m0x6fYtJvT4EH3w&s=6bCdiuCjmpXGIcMc9qc-XwDUWwt3ROzq6w_SM7Yd4Ls&e=
>
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Mailing List Archive

Attached Files:

Attached Files:

Attached Files: