Mailing List Archive: distributed nprobe

distributed nprobe

Sep 7, 2018, 5:55 AM

Post #1 of 3 (1130 views)

Dear ntop people,

I use nprobe to aggregate ip packets to IPFIX flows (and then analyze
them on another machine). Because I also aggregate http fields I had to
use multiple nprobe instances to keep up with high throughput rates.
Until now I used zbalance_ipc -m 1 to distribute packets according to
their IP hash to the single nprobe instances.
The problem is that now I need to do kernel routing on the incoming
device, and thus can not use zero copy (or zbalance_ipc) anymore because
that makes the device invisible to the kernel.
The question is:

-Is there another way to distribute the incoming traffic to multiple
nprobe instances (as with IP hashing)?

-Is there a way that I can filter packets in nprobe, so that they are
distributed more or less equally among multiple nprobe instances (again,
same IP should go to same instance)?

Thanks for any hints!

regards

Felix

Re: distributed nprobe [ In reply to ]

cardigliano at ntop

Sep 7, 2018, 12:20 PM

Post #2 of 3 (1130 views)

Permalink

Hi Felix
you can use the standard pf_ring kernel clustering in nProbe
adding the --cluster-id <id> option (you need to specify the same id
for all nProbe instances in the group in order to distribute the traffic).
You can use a bpf filter (--bpf-filter|-f <filter>) to filter traffic.

Regards
Alfredo

> On 7 Sep 2018, at 14:55, erlacher@campus.uni-paderborn.de wrote:
>
> Signed PGP part
> Dear ntop people,
>
> I use nprobe to aggregate ip packets to IPFIX flows (and then analyze
> them on another machine). Because I also aggregate http fields I had to
> use multiple nprobe instances to keep up with high throughput rates.
> Until now I used zbalance_ipc -m 1 to distribute packets according to
> their IP hash to the single nprobe instances.
> The problem is that now I need to do kernel routing on the incoming
> device, and thus can not use zero copy (or zbalance_ipc) anymore because
> that makes the device invisible to the kernel.
> The question is:
>
> -Is there another way to distribute the incoming traffic to multiple
> nprobe instances (as with IP hashing)?
>
> -Is there a way that I can filter packets in nprobe, so that they are
> distributed more or less equally among multiple nprobe instances (again,
> same IP should go to same instance)?
>
> Thanks for any hints!
>
> regards
>
> Felix
>
>
>
>

distributed nprobe [ In reply to ]

erlacher at campus

Sep 15, 2018, 8:36 AM

Post #3 of 3 (1122 views)

Permalink

Hi Alfredo,

thx for the advice. I just tested it and it seems to work as expected.
I have a follow-up question: How are packets distributed to the single
instances?
The produced flows seem ok, so I guess it is based on the ip/port or
similar. A quick search in the nprobe or pfring manuals didn't produce
any hits , can you point me to a docu for the --cluster-id stuff?

thx again

regards

felix

On 07/09/18 21:20, Alfredo Cardigliano wrote:
> Hi Felix
> you can use the standard pf_ring kernel clustering in nProbe
> adding the --cluster-id <id> option (you need to specify the same id
> for all nProbe instances in the group in order to distribute the traffic).
> You can use a bpf filter (--bpf-filter|-f <filter>) to filter traffic.
>
> Regards
> Alfredo
>
>> On 7 Sep 2018, at 14:55, erlacher@campus.uni-paderborn.de wrote:
>>
>> Signed PGP part
>> Dear ntop people,
>>
>> I use nprobe to aggregate ip packets to IPFIX flows (and then analyze
>> them on another machine). Because I also aggregate http fields I had to
>> use multiple nprobe instances to keep up with high throughput rates.
>> Until now I used zbalance_ipc -m 1 to distribute packets according to
>> their IP hash to the single nprobe instances.
>> The problem is that now I need to do kernel routing on the incoming
>> device, and thus can not use zero copy (or zbalance_ipc) anymore because
>> that makes the device invisible to the kernel.
>> The question is:
>>
>> -Is there another way to distribute the incoming traffic to multiple
>> nprobe instances (as with IP hashing)?
>>
>> -Is there a way that I can filter packets in nprobe, so that they are
>> distributed more or less equally among multiple nprobe instances (again,
>> same IP should go to same instance)?
>>
>> Thanks for any hints!
>>
>> regards
>>
>> Felix
>>
>>
>>
>>
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>

--
Felix Erlacher

ccs-labs.org/~erlacher
Key-ID:4EAC0959

Mailing List Archive

Attached Files:

Attached Files:

Attached Files: