[ Including the thread from osif-users for context ]
I'm testing out suricata and pfring. Currently, I'm running suri
using 22 pinned threads on the default 10G/82599ES interface (enp4s0),
and it's working fairly well.
Suricata config :
- interface: enp4s0 # Number of receive threads (>1 will enable experimental flow pinned
# runmode)
threads: 22
I tried using 22 ZC queues, configuring suri like this:
- interface: zc:99@0
threads: 1
- interface: zc:99@1
threads: 1
[...]
- interface: zc:99@21
threads: 1
and zbalance like so:
./zbalance_ipc -i zc:enp4s0 -m 4 -n 22,1 -c 99 -g 0 -S 1
and it seemed to work, however, the performance was drastically reduced.
With normal pf_ring (-i enp4s0) :
capture.kernel_packets | Total | 13211759633
capture.kernel_drops | Total | 3867805
With 22 zc queues:
capture.kernel_packets | Total | 92423279
capture.kernel_drops | Total | 619667008
I also tried just using one ZC queue, but found I seem to get better performance
with the multi-threaded non-ZC version.
Is there something I can do to get the benefits of both ZC and multi-threading?
Thanks,
--
Jim Hranicky
Data Security Specialist
UF Information Technology
105 NW 16TH ST Room #104 GAINESVILLE FL 32603-1826
352-273-1341
-------- Forwarded Message --------
Subject: Re: [Oisf-users] suricata with PF_RING Zero Copy/Pinned CPUs
Date: Thu, 11 Aug 2016 10:51:36 -0400
From: Jim Hranicky <jfh@ufl.edu>
To: Chris Wakelin <cwakelin@emergingthreats.net>, oisf-users@lists.openinfosecfoundation.org
Leaving off the -i seems work, thanks for your help.
Unfortunately, performance drops significantly with this setup.
With normal pf_ring (-i enp4s0) :
capture.kernel_packets | Total | 13211759633
capture.kernel_drops | Total | 3867805
With 22 zc queues:
capture.kernel_packets | Total | 92423279
capture.kernel_drops | Total | 619667008
I guess this is a question for the pfring list.
Jim
On 08/10/2016 09:08 PM, Jim Hranicky wrote:
> If you leave off the -i flag, will suri just use all the interfaces
> found in the config?
>
> Thanks for the response,
> Jim
>
> On 08/10/2016 07:40 PM, Chris Wakelin wrote:
>> I had this working when I still worked for the University of Reading
>> (which I left a year ago to join ET/Proofpoint). Alas I don't have
>> access to a machine running PF_RING at the moment.
>>
>> I did have a message thread in November 2014 about ZC + hugepages and
>> Suricata on the PF_RING (ntop-misc) mailing list
>> (http://lists.ntop.org/mailman/listinfo/ntop-misc - you need to
>> subscribe to see archives though).
>>
>> It looks like I had (uisng pfdnacluser_master rather than zbalance_ipc)
>>
>> insmod ixgbe.ko RSS=1,1 mtu=1522 adapters_to_enable=xx:xx:xx:xx:xx:xx
>> num_rx_slots=32768 num_tx_slots=0 numa_cpu_affinity=1,1
>> ifconfig up dna0
>>
>> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>> cat /proc/meminfo | grep Huge
>> mount -t hugetlbfs none /mnt/huge
>>
>> pfdnacluster_master -i dna0 -c 1 -n 15,1 -r 15 -m 4 -u /mnt/huge -d
>>
>> I was running ARGUS (flow-capture) on dnacl:1@15
>>
>> My Suricata config looked like (I know the cluster settings are ignored):-
>>
>> pfring:
>> - interface: dnacl:1@0
>> threads: 1
>> cluster-id: 99
>> cluster-type: cluster_flow
>> - interface: dnacl:1@1
>> threads: 1
>> cluster-id: 99
>> cluster-type: cluster_flow
>>
>> ...
>>
>> - interface: dnacl:1@14
>> threads: 1
>> cluster-id: 99
>> cluster-type: cluster_flow
>>
>>
>> My problem with hugepages turned out to be having Suricata running as a
>> non-root user meant it didn't get the right permissions to access them.
>> I said I was going to investigate a fix in Suricata to drop privileges
>> later, but seems I never got around to it. I ran Suricata as root
>> instead as a workaround (obviously not ideal). I also had CPUs with 16
>> real cores and hyperthreading disabled (that latter I now understand was
>> probably a bad idea!)
>>
>> Hope this gives some useful pointers,
>> Best Wishes,
>> Chris
>>
>> On 10/08/16 18:07, Jim Hranicky wrote:
>>> I'm able to run and get good results with using multiple threads
>>> on a pf-enabled interface when not running in ZC mode. I'm a little
>>> stumped though as to how to configure zbalance_ipc/suricata to use
>>> multiple threads using ZC.
>>>
>>> When run 1 queue for suri
>>>
>>> ./zbalance_ipc -i zc:enp4s0 -m 4 -n 1,1 -c 99 -g 0 -S 1
>>>
>>> then specify the interface like so
>>>
>>> - interface: zc:99@0
>>> threads: 22
>>>
>>> and run this command
>>>
>>> /opt/suricata/bin/suricata -i zc:99@0 -c /opt/suricata/etc/suricata/suricata.yaml --pfring -vv
>>>
>>> I get this:
>>>
>>> 10/8/2016 -- 13:00:01 - <Perf> - (RX#01) Using PF_RING v.6.5.0,
>>> interface zc:99@0, cluster-id 1
>>>
>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] -
>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and pf_ring module is loaded.
>>>
>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] -
>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and pf_ring module is loaded.
>>>
>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] -
>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and pf_ring module is loaded.
>>>
>>> Should I run zbalance_ipc with multiple queues? How do I specify the interfaces on
>>> the CL and the config file? FWIW I seem to get about 40% more events per second
>>> when running with multiple threads over running with 1 ZC queue.
>>>
>>> Thanks,
>>>
>>> --
>>> Jim Hranicky
>>> Data Security Specialist
>>> UF Information Technology
>>> 105 NW 16TH ST Room #104 GAINESVILLE FL 32603-1826
>>> 352-273-1341
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>
_______________________________________________
Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
I'm testing out suricata and pfring. Currently, I'm running suri
using 22 pinned threads on the default 10G/82599ES interface (enp4s0),
and it's working fairly well.
Suricata config :
- interface: enp4s0 # Number of receive threads (>1 will enable experimental flow pinned
# runmode)
threads: 22
I tried using 22 ZC queues, configuring suri like this:
- interface: zc:99@0
threads: 1
- interface: zc:99@1
threads: 1
[...]
- interface: zc:99@21
threads: 1
and zbalance like so:
./zbalance_ipc -i zc:enp4s0 -m 4 -n 22,1 -c 99 -g 0 -S 1
and it seemed to work, however, the performance was drastically reduced.
With normal pf_ring (-i enp4s0) :
capture.kernel_packets | Total | 13211759633
capture.kernel_drops | Total | 3867805
With 22 zc queues:
capture.kernel_packets | Total | 92423279
capture.kernel_drops | Total | 619667008
I also tried just using one ZC queue, but found I seem to get better performance
with the multi-threaded non-ZC version.
Is there something I can do to get the benefits of both ZC and multi-threading?
Thanks,
--
Jim Hranicky
Data Security Specialist
UF Information Technology
105 NW 16TH ST Room #104 GAINESVILLE FL 32603-1826
352-273-1341
-------- Forwarded Message --------
Subject: Re: [Oisf-users] suricata with PF_RING Zero Copy/Pinned CPUs
Date: Thu, 11 Aug 2016 10:51:36 -0400
From: Jim Hranicky <jfh@ufl.edu>
To: Chris Wakelin <cwakelin@emergingthreats.net>, oisf-users@lists.openinfosecfoundation.org
Leaving off the -i seems work, thanks for your help.
Unfortunately, performance drops significantly with this setup.
With normal pf_ring (-i enp4s0) :
capture.kernel_packets | Total | 13211759633
capture.kernel_drops | Total | 3867805
With 22 zc queues:
capture.kernel_packets | Total | 92423279
capture.kernel_drops | Total | 619667008
I guess this is a question for the pfring list.
Jim
On 08/10/2016 09:08 PM, Jim Hranicky wrote:
> If you leave off the -i flag, will suri just use all the interfaces
> found in the config?
>
> Thanks for the response,
> Jim
>
> On 08/10/2016 07:40 PM, Chris Wakelin wrote:
>> I had this working when I still worked for the University of Reading
>> (which I left a year ago to join ET/Proofpoint). Alas I don't have
>> access to a machine running PF_RING at the moment.
>>
>> I did have a message thread in November 2014 about ZC + hugepages and
>> Suricata on the PF_RING (ntop-misc) mailing list
>> (http://lists.ntop.org/mailman/listinfo/ntop-misc - you need to
>> subscribe to see archives though).
>>
>> It looks like I had (uisng pfdnacluser_master rather than zbalance_ipc)
>>
>> insmod ixgbe.ko RSS=1,1 mtu=1522 adapters_to_enable=xx:xx:xx:xx:xx:xx
>> num_rx_slots=32768 num_tx_slots=0 numa_cpu_affinity=1,1
>> ifconfig up dna0
>>
>> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>> cat /proc/meminfo | grep Huge
>> mount -t hugetlbfs none /mnt/huge
>>
>> pfdnacluster_master -i dna0 -c 1 -n 15,1 -r 15 -m 4 -u /mnt/huge -d
>>
>> I was running ARGUS (flow-capture) on dnacl:1@15
>>
>> My Suricata config looked like (I know the cluster settings are ignored):-
>>
>> pfring:
>> - interface: dnacl:1@0
>> threads: 1
>> cluster-id: 99
>> cluster-type: cluster_flow
>> - interface: dnacl:1@1
>> threads: 1
>> cluster-id: 99
>> cluster-type: cluster_flow
>>
>> ...
>>
>> - interface: dnacl:1@14
>> threads: 1
>> cluster-id: 99
>> cluster-type: cluster_flow
>>
>>
>> My problem with hugepages turned out to be having Suricata running as a
>> non-root user meant it didn't get the right permissions to access them.
>> I said I was going to investigate a fix in Suricata to drop privileges
>> later, but seems I never got around to it. I ran Suricata as root
>> instead as a workaround (obviously not ideal). I also had CPUs with 16
>> real cores and hyperthreading disabled (that latter I now understand was
>> probably a bad idea!)
>>
>> Hope this gives some useful pointers,
>> Best Wishes,
>> Chris
>>
>> On 10/08/16 18:07, Jim Hranicky wrote:
>>> I'm able to run and get good results with using multiple threads
>>> on a pf-enabled interface when not running in ZC mode. I'm a little
>>> stumped though as to how to configure zbalance_ipc/suricata to use
>>> multiple threads using ZC.
>>>
>>> When run 1 queue for suri
>>>
>>> ./zbalance_ipc -i zc:enp4s0 -m 4 -n 1,1 -c 99 -g 0 -S 1
>>>
>>> then specify the interface like so
>>>
>>> - interface: zc:99@0
>>> threads: 22
>>>
>>> and run this command
>>>
>>> /opt/suricata/bin/suricata -i zc:99@0 -c /opt/suricata/etc/suricata/suricata.yaml --pfring -vv
>>>
>>> I get this:
>>>
>>> 10/8/2016 -- 13:00:01 - <Perf> - (RX#01) Using PF_RING v.6.5.0,
>>> interface zc:99@0, cluster-id 1
>>>
>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] -
>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and pf_ring module is loaded.
>>>
>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] -
>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and pf_ring module is loaded.
>>>
>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] -
>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and pf_ring module is loaded.
>>>
>>> Should I run zbalance_ipc with multiple queues? How do I specify the interfaces on
>>> the CL and the config file? FWIW I seem to get about 40% more events per second
>>> when running with multiple threads over running with 1 ZC queue.
>>>
>>> Thanks,
>>>
>>> --
>>> Jim Hranicky
>>> Data Security Specialist
>>> UF Information Technology
>>> 105 NW 16TH ST Room #104 GAINESVILLE FL 32603-1826
>>> 352-273-1341
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>
_______________________________________________
Suricata IDS Users mailing list: oisf-users@openinfosecfoundation.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
_______________________________________________
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
