Mailing List Archive

nProbe dynamic blacklist / --max-num-flows
Hi ntop team, I have a couple of nProbe questions for you:

We had an incident where a badly behaved host increased the number of flow
records being generated by nProbe by a factor of 10 and really stressed our
downstream processing. I ended up restarting our nProbe processes with an
added *--black-list x.x.x.x/32* option to ignore that host. That led me to
wonder, is there any way to dynamically change the blacklist configuration
so that in the future I could add a host or network without having to
restart nProbe? Doing so without restarting would be preferable since
restarting will result in some data loss across all the monitored traffic.
I didn't see anything in the documentation, but thought it would be worth
checking here.

On a related note, I wonder about the --max-num-flows option which limits
the number of active flows in the case of DoS, etc. In the event that the
maximum number of flows is exceeded, what flows will get discarded? Any
new flows above the limit, or is there a more selective algorithm?

Thank you!
Peter
Re: nProbe dynamic blacklist / --max-num-flows [ In reply to ]
Hi Peter
changing them on the fly is not supported. Better if you filter the host with -f so avoid processing packets at all, instead of discarding egress flows

Yes of there is a DoS, flows exceeding the threshold are dropped, this to avoid to DoS also nProbe. What is the algorithm you have in mind exactly?

Regards Luca

> On 21 Feb 2019, at 01:03, Peter Giles <gilesp@uw.edu> wrote:
>
> Hi ntop team, I have a couple of nProbe questions for you:
>
> We had an incident where a badly behaved host increased the number of flow records being generated by nProbe by a factor of 10 and really stressed our downstream processing. I ended up restarting our nProbe processes with an added --black-list x.x.x.x/32 option to ignore that host. That led me to wonder, is there any way to dynamically change the blacklist configuration so that in the future I could add a host or network without having to restart nProbe? Doing so without restarting would be preferable since restarting will result in some data loss across all the monitored traffic. I didn't see anything in the documentation, but thought it would be worth checking here.
>
> On a related note, I wonder about the --max-num-flows option which limits the number of active flows in the case of DoS, etc. In the event that the maximum number of flows is exceeded, what flows will get discarded? Any new flows above the limit, or is there a more selective algorithm?
>
> Thank you!
> Peter
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
Re: nProbe dynamic blacklist / --max-num-flows [ In reply to ]
The -f tip is good to know.

Regarding --max-num-flows, while I could imagine different approaches, I
really just wanted to understand what the implemented behavior is.

Back to the wish for dynamic blacklisting, could Lua scripting be an option
for this? It seems there is no API for the Lua script to tell nProbe to
ignore a flow though.

Thanks again,
Peter

On Fri, Feb 22, 2019 at 3:01 AM ntop-misc-request@listgateway.unipi.it <
ntop-misc-request@listgateway.unipi.it> wrote:

> Send Ntop-misc mailing list submissions to
> ntop-misc@listgateway.unipi.it
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> or, via email, send a message with subject or body 'help' to
> ntop-misc-request@listgateway.unipi.it
>
> You can reach the person managing the list at
> ntop-misc-owner@listgateway.unipi.it
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Ntop-misc digest..."
>
>
> Today's Topics:
>
> 1. Re: nProbe dynamic blacklist / --max-num-flows (Luca Deri)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 21 Feb 2019 11:39:48 +0100
> From: Luca Deri <deri@ntop.org>
> To: ntop-misc@listgateway.unipi.it
> Subject: Re: [Ntop-misc] nProbe dynamic blacklist / --max-num-flows
> Message-ID: <80B34F5E-EF6D-41A7-ADA9-AE4A6F64B19C@ntop.org>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Peter
> changing them on the fly is not supported. Better if you filter the host
> with -f so avoid processing packets at all, instead of discarding egress
> flows
>
> Yes of there is a DoS, flows exceeding the threshold are dropped, this to
> avoid to DoS also nProbe. What is the algorithm you have in mind exactly?
>
> Regards Luca
>
> > On 21 Feb 2019, at 01:03, Peter Giles <gilesp@uw.edu> wrote:
> >
> > Hi ntop team, I have a couple of nProbe questions for you:
> >
> > We had an incident where a badly behaved host increased the number of
> flow records being generated by nProbe by a factor of 10 and really
> stressed our downstream processing. I ended up restarting our nProbe
> processes with an added --black-list x.x.x.x/32 option to ignore that
> host. That led me to wonder, is there any way to dynamically change the
> blacklist configuration so that in the future I could add a host or network
> without having to restart nProbe? Doing so without restarting would be
> preferable since restarting will result in some data loss across all the
> monitored traffic. I didn't see anything in the documentation, but thought
> it would be worth checking here.
> >
> > On a related note, I wonder about the --max-num-flows option which
> limits the number of active flows in the case of DoS, etc. In the event
> that the maximum number of flows is exceeded, what flows will get
> discarded? Any new flows above the limit, or is there a more selective
> algorithm?
> >
> > Thank you!
> > Peter
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://listgateway.unipi.it/mailman/private/ntop-misc/attachments/20190221/2b2859e4/attachment-0001.htm
> >
>
> ------------------------------
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
> End of Ntop-misc Digest, Vol 175, Issue 2
> *****************************************
>
Re: nProbe dynamic blacklist / --max-num-flows [ In reply to ]
Hi Peter
we have used Lua so far for other activities, even it could be potentially extended to what you need. The problem, is that executing a script during export slows down things a bit.
Please let us know if there is something you would like to implement

Cheers Luca

> On 22 Feb 2019, at 21:14,Peter Giles <gilesp@uw.edu> wrote:
>
> The -f tip is good to know.
>
> Regarding --max-num-flows, while I could imagine different approaches, I really just wanted to understand what the implemented behavior is.
>
> Back to the wish for dynamic blacklisting, could Lua scripting be an option for this? It seems there is no API for the Lua script to tell nProbe to ignore a flow though.
>
> Thanks again,
> Peter
>
> On Fri, Feb 22, 2019 at 3:01 AM ntop-misc-request@listgateway.unipi.it <mailto:ntop-misc-request@listgateway.unipi.it> <ntop-misc-request@listgateway.unipi.it <mailto:ntop-misc-request@listgateway.unipi.it>> wrote:
> Send Ntop-misc mailing list submissions to
> ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> or, via email, send a message with subject or body 'help' to
> ntop-misc-request@listgateway.unipi.it <mailto:ntop-misc-request@listgateway.unipi.it>
>
> You can reach the person managing the list at
> ntop-misc-owner@listgateway.unipi.it <mailto:ntop-misc-owner@listgateway.unipi.it>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Ntop-misc digest..."
>
>
> Today's Topics:
>
> 1. Re: nProbe dynamic blacklist / --max-num-flows (Luca Deri)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 21 Feb 2019 11:39:48 +0100
> From: Luca Deri <deri@ntop.org <mailto:deri@ntop.org>>
> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> Subject: Re: [Ntop-misc] nProbe dynamic blacklist / --max-num-flows
> Message-ID: <80B34F5E-EF6D-41A7-ADA9-AE4A6F64B19C@ntop.org <mailto:80B34F5E-EF6D-41A7-ADA9-AE4A6F64B19C@ntop.org>>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Peter
> changing them on the fly is not supported. Better if you filter the host with -f so avoid processing packets at all, instead of discarding egress flows
>
> Yes of there is a DoS, flows exceeding the threshold are dropped, this to avoid to DoS also nProbe. What is the algorithm you have in mind exactly?
>
> Regards Luca
>
> > On 21 Feb 2019, at 01:03, Peter Giles <gilesp@uw.edu <mailto:gilesp@uw.edu>> wrote:
> >
> > Hi ntop team, I have a couple of nProbe questions for you:
> >
> > We had an incident where a badly behaved host increased the number of flow records being generated by nProbe by a factor of 10 and really stressed our downstream processing. I ended up restarting our nProbe processes with an added --black-list x.x.x.x/32 option to ignore that host. That led me to wonder, is there any way to dynamically change the blacklist configuration so that in the future I could add a host or network without having to restart nProbe? Doing so without restarting would be preferable since restarting will result in some data loss across all the monitored traffic. I didn't see anything in the documentation, but thought it would be worth checking here.
> >
> > On a related note, I wonder about the --max-num-flows option which limits the number of active flows in the case of DoS, etc. In the event that the maximum number of flows is exceeded, what flows will get discarded? Any new flows above the limit, or is there a more selective algorithm?
> >
> > Thank you!
> > Peter
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://listgateway.unipi.it/mailman/private/ntop-misc/attachments/20190221/2b2859e4/attachment-0001.htm <http://listgateway.unipi.it/mailman/private/ntop-misc/attachments/20190221/2b2859e4/attachment-0001.htm>>
>
> ------------------------------
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> End of Ntop-misc Digest, Vol 175, Issue 2
> *****************************************
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc