Mailing List Archive: nProbe / ntopNG config

nProbe / ntopNG config

Oct 12, 2018, 1:52 AM

Post #1 of 11 (1913 views)

Hello,
I'm trying to make nprobe work with IPFIX and ntopng, but data displayed by
ntopng is inconsistent.

Here's the path my netflow packets take :
router -> nprobe:6345 -> ntopNG:6445.
(nprobe and ntopng services are on the same host.)

nprobe runs with : (cat /etc/nprobe/nprobe.conf)
-i=any
-n=none
--collector-port=6345
--zmq tcp://*:6445 %EXPORTER_IPV4_ADDRESS
-T "@NTOPNG@"

ntopng runs with : (cat /etc/ntopng/ntopng.conf)
-i="tcp://127.0.0.1:6445"
-m=<my local subnet>
-F="mysql;/var/run/mysqld/mysqld.sock;ntopng;flows-%Y.%m.%d;ntopng;ntopng"

I have two hosts sending netflow to nprobe. I don't see two interfaces in
ntopng. any reason why ?
Trafic one one of the hosts which sends netflow to nprobe is always
>100mb/s. In ntopng graphs, I do not see this value. It moves between 1 and
10mb/s. why ?

I'm running ntop/nprobe from ntop debian repositories, latest version
(upgraded this morning).

Regards
Cédriic

Re: nProbe / ntopNG config [ In reply to ]

mainardi at ntop

Oct 12, 2018, 10:19 AM

Post #2 of 11 (1911 views)

Permalink

Hello,

> On 12 Oct 2018, at 10:52, BASSAGET Cédric <cedric.bassaget.ml@gmail.com> wrote:
>
> Hello,
> I'm trying to make nprobe work with IPFIX and ntopng, but data displayed by ntopng is inconsistent.
>
> Here's the path my netflow packets take :
> router -> nprobe:6345 -> ntopNG:6445.
> (nprobe and ntopng services are on the same host.)
>
> nprobe runs with : (cat /etc/nprobe/nprobe.conf)
> -i=any

set to

-i=none

> -n=none
> --collector-port=6345
> --zmq tcp://*:6445 <tcp://*:6445>%EXPORTER_IPV4_ADDRESS
> -T "@NTOPNG@"

exporter ipv4 address must go into the template::

-T "@NTOPNG@ %EXPORTER_IPV4_ADDRESS"

>
>
> ntopng runs with : (cat /etc/ntopng/ntopng.conf)
> -i="tcp://127.0.0.1:6445 <http://127.0.0.1:6445/>"
> -m=<my local subnet>
> -F="mysql;/var/run/mysqld/mysqld.sock;ntopng;flows-%Y.%m.%d;ntopng;ntopng"

-F contains duplicated conf. Check that.

>
> I have two hosts sending netflow to nprobe. I don't see two interfaces in ntopng. any reason why ?

Visit ntopng preferences, enable interfaces disaggregation on the basis of the probe ip, and then restart ntopng

> Trafic one one of the hosts which sends netflow to nprobe is always >100mb/s. In ntopng graphs, I do not see this value. It moves between 1 and 10mb/s. why ?

see this explanation: https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928 <https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928>

Regards,
Simone

>
> I'm running ntop/nprobe from ntop debian repositories, latest version (upgraded this morning).
>
> Regards
> Cédriic
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

Re: nProbe / ntopNG config [ In reply to ]

mainardi at ntop

Oct 12, 2018, 10:19 AM

Post #3 of 11 (1911 views)

Permalink

Re: nProbe / ntopNG config [ In reply to ]

cedric.bassaget.ml at gmail

Oct 15, 2018, 2:47 AM

Post #4 of 11 (1906 views)

Permalink

Hi Simone,

Le ven. 12 oct. 2018 à 19:19, Simone Mainardi <mainardi@ntop.org> a écrit :

> Hello,
>
> On 12 Oct 2018, at 10:52, BASSAGET Cédric <cedric.bassaget.ml@gmail.com>
> wrote:
>
> Hello,
> I'm trying to make nprobe work with IPFIX and ntopng, but data displayed
> by ntopng is inconsistent.
>
> Here's the path my netflow packets take :
> router -> nprobe:6345 -> ntopNG:6445.
> (nprobe and ntopng services are on the same host.)
>
> nprobe runs with : (cat /etc/nprobe/nprobe.conf)
> -i=any
>
>
> set to
>
> -i=none
>
> -n=none
> --collector-port=6345
> --zmq tcp://*:6445
>
> %EXPORTER_IPV4_ADDRESS
> -T "@NTOPNG@"
>
>
> exporter ipv4 address must go into the template::
>
> -T "@NTOPNG@ %EXPORTER_IPV4_ADDRESS"
>
@NTOPNG@ already includes %EXPORTER_IPV4_ADDRESS

>
>
>
> ntopng runs with : (cat /etc/ntopng/ntopng.conf)
> -i="tcp://127.0.0.1:6445"
> -m=<my local subnet>
> -F="mysql;/var/run/mysqld/mysqld.sock;ntopng;flows-%Y.%m.%d;ntopng;ntopng"
>
>
> -F contains duplicated conf. Check that.
>
from man page :
Example -F "mysql;localhost;ntopng;flows-%Y.%m.%d;root;".

as the last "ntopng" is my password, I do not see what is duplicated.

>
> I have two hosts sending netflow to nprobe. I don't see two interfaces in
> ntopng. any reason why ?
>
>
> Visit ntopng preferences, enable interfaces disaggregation on the basis of
> the probe ip, and then restart ntopng
>
Done, works fine.

>
> Trafic one one of the hosts which sends netflow to nprobe is always
> >100mb/s. In ntopng graphs, I do not see this value. It moves between 1 and
> 10mb/s. why ?
>
>
> see this explanation:
> https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928
>
I don't think it's related to this, as the host which sends netflows is a
BGP router and handles a lot of trafic from different sources. TCP sessions
may be relatively short.

I'm still seeing a difference between real trafic on my bgp router and data
gathered by nprobe from netflows. My netflow exporter has a samplign rate
defined to 10, so has my ntopng interface.
Running iftoip and other monitoring tools always shows more than 100mb/s RX.
Graph at the bottom of ntopng page shows completely different values (often
around 10Mb/s)
Historical page of interface shows a max value of 54Mb/s but my max value
on host is around 270Mb/s...

My exporter is pmacct, how to check if it sends cumulative counters or not ?
Regards,
Cédric

>
>
> Regards,
> Simone
>
>
> I'm running ntop/nprobe from ntop debian repositories, latest version
> (upgraded this morning).
>
> Regards
> Cédriic
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

Re: nProbe / ntopNG config [ In reply to ]

cedric.bassaget.ml at gmail

Oct 15, 2018, 2:47 AM

Post #5 of 11 (1906 views)

Permalink

Re: nProbe / ntopNG config [ In reply to ]

mainardi at ntop

Oct 22, 2018, 3:53 AM

Post #6 of 11 (1896 views)

Permalink

Cédric,

You mentioned the exporter is doing 1:10 sampling. I am assuming you are talking about the flow collection sampling rate. So I think you have to use option -S in nProbe to upscale the incoming traffic.

-S <pkt rate>:<flow collection rate>:<flow export rate>

In your case:

-S 1:10:1

Have a look at https://www.ntop.org/guides/nProbe/cli_options.html?highlight=sampling <https://www.ntop.org/guides/nProbe/cli_options.html?highlight=sampling> for a detailed description.

Simone

> On 15 Oct 2018, at 11:47, BASSAGET Cédric <cedric.bassaget.ml@gmail.com> wrote:
>
> Hi Simone,
>
>
> Le ven. 12 oct. 2018 à 19:19, Simone Mainardi <mainardi@ntop.org <mailto:mainardi@ntop.org>> a écrit :
> Hello,
>
>> On 12 Oct 2018, at 10:52, BASSAGET Cédric <cedric.bassaget.ml@gmail.com <mailto:cedric.bassaget.ml@gmail.com>> wrote:
>>
>> Hello,
>> I'm trying to make nprobe work with IPFIX and ntopng, but data displayed by ntopng is inconsistent.
>>
>> Here's the path my netflow packets take :
>> router -> nprobe:6345 -> ntopNG:6445.
>> (nprobe and ntopng services are on the same host.)
>>
>> nprobe runs with : (cat /etc/nprobe/nprobe.conf)
>> -i=any
>
> set to
>
> -i=none
>
>> -n=none
>> --collector-port=6345
>> --zmq tcp://*:6445 <>%EXPORTER_IPV4_ADDRESS
>> -T "@NTOPNG@"
>
> exporter ipv4 address must go into the template::
>
> -T "@NTOPNG@ %EXPORTER_IPV4_ADDRESS"
> @NTOPNG@ already includes %EXPORTER_IPV4_ADDRESS
>
>>
>>
>> ntopng runs with : (cat /etc/ntopng/ntopng.conf)
>> -i="tcp://127.0.0.1:6445 <http://127.0.0.1:6445/>"
>> -m=<my local subnet>
>> -F="mysql;/var/run/mysqld/mysqld.sock;ntopng;flows-%Y.%m.%d;ntopng;ntopng"
>
> -F contains duplicated conf. Check that.
> from man page :
> Example -F "mysql;localhost;ntopng;flows-%Y.%m.%d;root;".
>
> as the last "ntopng" is my password, I do not see what is duplicated.
>
>
>>
>> I have two hosts sending netflow to nprobe. I don't see two interfaces in ntopng. any reason why ?
>
> Visit ntopng preferences, enable interfaces disaggregation on the basis of the probe ip, and then restart ntopng
> Done, works fine.
>
>> Trafic one one of the hosts which sends netflow to nprobe is always >100mb/s. In ntopng graphs, I do not see this value. It moves between 1 and 10mb/s. why ?
>
> see this explanation: https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928 <https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928>
> I don't think it's related to this, as the host which sends netflows is a BGP router and handles a lot of trafic from different sources. TCP sessions may be relatively short.
>
> I'm still seeing a difference between real trafic on my bgp router and data gathered by nprobe from netflows. My netflow exporter has a samplign rate defined to 10, so has my ntopng interface.
> Running iftoip and other monitoring tools always shows more than 100mb/s RX.
> Graph at the bottom of ntopng page shows completely different values (often around 10Mb/s)
> Historical page of interface shows a max value of 54Mb/s but my max value on host is around 270Mb/s...
>
> My exporter is pmacct, how to check if it sends cumulative counters or not ?
> Regards,
> Cédric
>
>
> Regards,
> Simone
>
>>
>> I'm running ntop/nprobe from ntop debian repositories, latest version (upgraded this morning).
>>
>> Regards
>> Cédriic
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>_______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>

Re: nProbe / ntopNG config [ In reply to ]

mainardi at ntop

Oct 22, 2018, 3:53 AM

Post #7 of 11 (1896 views)

Permalink

Re: nProbe / ntopNG config [ In reply to ]

cedric.bassaget.ml at gmail

Oct 24, 2018, 7:12 AM

Post #8 of 11 (1895 views)

Permalink

Hello Simone,
If I have multiple exporters which send flows with different sampling rates
to ZMQ nprobe, do I have a solution ?
Regards

Le lun. 22 oct. 2018 à 12:53, Simone Mainardi <mainardi@ntop.org> a écrit :

> Cédric,
>
> You mentioned the exporter is doing 1:10 sampling. I am assuming you are
> talking about the flow collection sampling rate. So I think you have to use
> option -S in nProbe to upscale the incoming traffic.
>
> -S <pkt rate>:<flow collection rate>:<flow export rate>
>
> In your case:
>
> -S 1:10:1
>
> Have a look at
> https://www.ntop.org/guides/nProbe/cli_options.html?highlight=sampling for
> a detailed description.
>
>
> Simone
>
>
>
>
> On 15 Oct 2018, at 11:47, BASSAGET Cédric <cedric.bassaget.ml@gmail.com>
> wrote:
>
> Hi Simone,
>
>
> Le ven. 12 oct. 2018 à 19:19, Simone Mainardi <mainardi@ntop.org> a
> écrit :
>
>> Hello,
>>
>> On 12 Oct 2018, at 10:52, BASSAGET Cédric <cedric.bassaget.ml@gmail.com>
>> wrote:
>>
>> Hello,
>> I'm trying to make nprobe work with IPFIX and ntopng, but data displayed
>> by ntopng is inconsistent.
>>
>> Here's the path my netflow packets take :
>> router -> nprobe:6345 -> ntopNG:6445.
>> (nprobe and ntopng services are on the same host.)
>>
>> nprobe runs with : (cat /etc/nprobe/nprobe.conf)
>> -i=any
>>
>>
>> set to
>>
>> -i=none
>>
>> -n=none
>> --collector-port=6345
>> --zmq tcp://*:6445
>>
>> %EXPORTER_IPV4_ADDRESS
>> -T "@NTOPNG@"
>>
>>
>> exporter ipv4 address must go into the template::
>>
>> -T "@NTOPNG@ %EXPORTER_IPV4_ADDRESS"
>>
> @NTOPNG@ already includes %EXPORTER_IPV4_ADDRESS
>
>>
>>
>>
>> ntopng runs with : (cat /etc/ntopng/ntopng.conf)
>> -i="tcp://127.0.0.1:6445"
>> -m=<my local subnet>
>> -F="mysql;/var/run/mysqld/mysqld.sock;ntopng;flows-%Y.%m.%d;ntopng;ntopng"
>>
>>
>> -F contains duplicated conf. Check that.
>>
> from man page :
> Example -F "mysql;localhost;ntopng;flows-%Y.%m.%d;root;".
>
> as the last "ntopng" is my password, I do not see what is duplicated.
>
>
>>
>> I have two hosts sending netflow to nprobe. I don't see two interfaces in
>> ntopng. any reason why ?
>>
>>
>> Visit ntopng preferences, enable interfaces disaggregation on the basis
>> of the probe ip, and then restart ntopng
>>
> Done, works fine.
>
>>
>> Trafic one one of the hosts which sends netflow to nprobe is always
>> >100mb/s. In ntopng graphs, I do not see this value. It moves between 1 and
>> 10mb/s. why ?
>>
>>
>> see this explanation:
>> https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928
>>
> I don't think it's related to this, as the host which sends netflows is a
> BGP router and handles a lot of trafic from different sources. TCP sessions
> may be relatively short.
>
> I'm still seeing a difference between real trafic on my bgp router and
> data gathered by nprobe from netflows. My netflow exporter has a samplign
> rate defined to 10, so has my ntopng interface.
> Running iftoip and other monitoring tools always shows more than 100mb/s
> RX.
> Graph at the bottom of ntopng page shows completely different values
> (often around 10Mb/s)
> Historical page of interface shows a max value of 54Mb/s but my max value
> on host is around 270Mb/s...
>
> My exporter is pmacct, how to check if it sends cumulative counters or not
> ?
> Regards,
> Cédric
>
>>
>>
>> Regards,
>> Simone
>>
>>
>> I'm running ntop/nprobe from ntop debian repositories, latest version
>> (upgraded this morning).
>>
>> Regards
>> Cédriic
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

Re: nProbe / ntopNG config [ In reply to ]

cedric.bassaget.ml at gmail

Oct 24, 2018, 7:12 AM

Post #9 of 11 (1895 views)

Permalink

Re: nProbe / ntopNG config [ In reply to ]

mainardi at ntop

Oct 25, 2018, 1:12 AM

Post #10 of 11 (1886 views)

Permalink

Cédric,

Currently, we don't handle that for IPFIX. We handle multiple rates only for sFlow as the actual sampling rate is carried right into in the packets. If you need this feature for IPFIX, please file an issue on our nProbe GitHub issue tracker and we'll see if we can accomodate it.

Simone

> On 24 Oct 2018, at 16:12, BASSAGET Cédric <cedric.bassaget.ml@gmail.com> wrote:
>
> Hello Simone,
> If I have multiple exporters which send flows with different sampling rates to ZMQ nprobe, do I have a solution ?
> Regards
>
> Le lun. 22 oct. 2018 à 12:53, Simone Mainardi <mainardi@ntop.org <mailto:mainardi@ntop.org>> a écrit :
> Cédric,
>
> You mentioned the exporter is doing 1:10 sampling. I am assuming you are talking about the flow collection sampling rate. So I think you have to use option -S in nProbe to upscale the incoming traffic.
>
> -S <pkt rate>:<flow collection rate>:<flow export rate>
>
> In your case:
>
> -S 1:10:1
>
> Have a look at https://www.ntop.org/guides/nProbe/cli_options.html?highlight=sampling <https://www.ntop.org/guides/nProbe/cli_options.html?highlight=sampling> for a detailed description.
>
>
> Simone
>
>
>
>
>> On 15 Oct 2018, at 11:47, BASSAGET Cédric <cedric.bassaget.ml@gmail.com <mailto:cedric.bassaget.ml@gmail.com>> wrote:
>>
>> Hi Simone,
>>
>>
>> Le ven. 12 oct. 2018 à 19:19, Simone Mainardi <mainardi@ntop.org <mailto:mainardi@ntop.org>> a écrit :
>> Hello,
>>
>>> On 12 Oct 2018, at 10:52, BASSAGET Cédric <cedric.bassaget.ml@gmail.com <mailto:cedric.bassaget.ml@gmail.com>> wrote:
>>>
>>> Hello,
>>> I'm trying to make nprobe work with IPFIX and ntopng, but data displayed by ntopng is inconsistent.
>>>
>>> Here's the path my netflow packets take :
>>> router -> nprobe:6345 -> ntopNG:6445.
>>> (nprobe and ntopng services are on the same host.)
>>>
>>> nprobe runs with : (cat /etc/nprobe/nprobe.conf)
>>> -i=any
>>
>> set to
>>
>> -i=none
>>
>>> -n=none
>>> --collector-port=6345
>>> --zmq tcp://*:6445 <>%EXPORTER_IPV4_ADDRESS
>>> -T "@NTOPNG@"
>>
>> exporter ipv4 address must go into the template::
>>
>> -T "@NTOPNG@ %EXPORTER_IPV4_ADDRESS"
>> @NTOPNG@ already includes %EXPORTER_IPV4_ADDRESS
>>
>>>
>>>
>>> ntopng runs with : (cat /etc/ntopng/ntopng.conf)
>>> -i="tcp://127.0.0.1:6445 <http://127.0.0.1:6445/>"
>>> -m=<my local subnet>
>>> -F="mysql;/var/run/mysqld/mysqld.sock;ntopng;flows-%Y.%m.%d;ntopng;ntopng"
>>
>> -F contains duplicated conf. Check that.
>> from man page :
>> Example -F "mysql;localhost;ntopng;flows-%Y.%m.%d;root;".
>>
>> as the last "ntopng" is my password, I do not see what is duplicated.
>>
>>
>>>
>>> I have two hosts sending netflow to nprobe. I don't see two interfaces in ntopng. any reason why ?
>>
>> Visit ntopng preferences, enable interfaces disaggregation on the basis of the probe ip, and then restart ntopng
>> Done, works fine.
>>
>>> Trafic one one of the hosts which sends netflow to nprobe is always >100mb/s. In ntopng graphs, I do not see this value. It moves between 1 and 10mb/s. why ?
>>
>> see this explanation: https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928 <https://github.com/ntop/ntopng/issues/1359#issuecomment-320949928>
>> I don't think it's related to this, as the host which sends netflows is a BGP router and handles a lot of trafic from different sources. TCP sessions may be relatively short.
>>
>> I'm still seeing a difference between real trafic on my bgp router and data gathered by nprobe from netflows. My netflow exporter has a samplign rate defined to 10, so has my ntopng interface.
>> Running iftoip and other monitoring tools always shows more than 100mb/s RX.
>> Graph at the bottom of ntopng page shows completely different values (often around 10Mb/s)
>> Historical page of interface shows a max value of 54Mb/s but my max value on host is around 270Mb/s...
>>
>> My exporter is pmacct, how to check if it sends cumulative counters or not ?
>> Regards,
>> Cédric
>>
>>
>> Regards,
>> Simone
>>
>>>
>>> I'm running ntop/nprobe from ntop debian repositories, latest version (upgraded this morning).
>>>
>>> Regards
>>> Cédriic
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>>> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>_______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>_______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop