Mailing List Archive

Gerard,

When you say wildly inaccurate, are you referring to the realtime charts in the dashboard only? I would expect them to be somehow inaccurate due to the nature of NetFlow, but once you visit the historical pages then totals and speeds must be accurate.

Try and visit the historical charts page of the interface, or the same page for any of your local hosts. Exact data must be there.

If you want to mitigate the inaccuracy of the dashboard, you should reduce the refresh rate to a value that is meaningful, on the basis of the NetFlow timeouts you have configured. To change this, visit the interface page, wheel menu entry, and change the setting.

You can also try and disable the nProbe cache to make sure flows are delivered straight to ntopng as soon as they are received. Option is --disable-cache.

Feel free to attach screenshots if necessary.

Simone


> On 14 Jan 2019, at 23:54, Gerard Beekmans <gbeekmans@tsag.net> wrote:
>
> Hi,
>
> I just finished setting up ntong Enterprise along with nprobe Professional. I am seeing traffic rates/speeds that are much higher than what they are, likely due to a misconfiguration on my end or a limitation of the setup.
>
> We’re using an Exinda packet shaper appliance that sends Netflow V9 to nprobe which in turn sends it to ntopng. All the traffic appear to show up in ntopng’s web GUI but with much higher rates than what they are, sometimes 100x faster than what the speeds physically could be.
>
> From our central location we have VPN tunnels established with dozens of remote sites that we provide Internet to. Despite some of those sites only having 2 to 5 Mbps available speed, ntopng reports their flows as hundreds of Mbps.
>
> /etc/ntopng/ntopng.conf configuration:
>
> -G=/var/run/ntopng.pid
> -i=tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556>
>
> /etc/nprobe/nprobe.conf configuration:
>
> -i=none
> -n=none
> -3=2055
> --zmq=tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556>
> -T="@NTOPNG@"
> -V=9
> -d=1
> -t=60
>
> From what I have been able to glean from posts online is that this is in part a limitation of how Netflow works and possible an issue with the idle timeout values? I can’t find an exact timeout number that the Exinda appliance uses. Its documentation mentions that flows are “exported within 10 seconds” and has a 1 minute timeout configured for persistent or long-term flows.
>
> I tried setting the nprobe timeout to both ’10’ and most recently ‘1’ which seems to have made no difference.
>
> It is important to get this information correct. We previously used a programmed called Plixer which did a better job at portraying actual speeds (it also was nowhere near correct but the discrepancy was not significant enough to worry about it too much).
>
> We require this level of precision for reporting and forensics. For example, on a nearly daily basis we will be asked questions such as “why was the Internet slow at our location at X day during Y and Z hours”. I need to be able to piece together what happened on a specific subnet, on certain days during certain timeframes and need to show the data that includes which applications were at fault (often it’s the Netflix and other social media and streaming at fault, or various OS updates).
>
> Any insights and help in fixing this, if possible, would be much appreciated.
> Thanks,
> Gerard Beekmans
> Sr. Network Engineer
> First Nations Technical Services Advisory Group Inc.
> Phone: 780-638-2739 <tel:780-638-2739>
> Fax: 780-483-8632 <tel:780-483-8632>
> Helpdesk: 1-888-999-3356 <tel:1-888-999-3356>
> Email: gbeekmans@tsag.net <mailto:gbeekmans@tsag.net>
> Santa Fe Plaza
> 18232 - 102 Avenue NW
> Edmonton, AB T5S 1S7
> http://www.tsag.net <http://www.tsag.net/>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>

Gerard,

When you say wildly inaccurate, are you referring to the realtime charts in the dashboard only? I would expect them to be somehow inaccurate due to the nature of NetFlow, but once you visit the historical pages then totals and speeds must be accurate.

Try and visit the historical charts page of the interface, or the same page for any of your local hosts. Exact data must be there.

If you want to mitigate the inaccuracy of the dashboard, you should reduce the refresh rate to a value that is meaningful, on the basis of the NetFlow timeouts you have configured. To change this, visit the interface page, wheel menu entry, and change the setting.

You can also try and disable the nProbe cache to make sure flows are delivered straight to ntopng as soon as they are received. Option is --disable-cache.

Feel free to attach screenshots if necessary.

Simone


> On 14 Jan 2019, at 23:54, Gerard Beekmans <gbeekmans@tsag.net> wrote:
>
> Hi,
>
> I just finished setting up ntong Enterprise along with nprobe Professional. I am seeing traffic rates/speeds that are much higher than what they are, likely due to a misconfiguration on my end or a limitation of the setup.
>
> We’re using an Exinda packet shaper appliance that sends Netflow V9 to nprobe which in turn sends it to ntopng. All the traffic appear to show up in ntopng’s web GUI but with much higher rates than what they are, sometimes 100x faster than what the speeds physically could be.
>
> From our central location we have VPN tunnels established with dozens of remote sites that we provide Internet to. Despite some of those sites only having 2 to 5 Mbps available speed, ntopng reports their flows as hundreds of Mbps.
>
> /etc/ntopng/ntopng.conf configuration:
>
> -G=/var/run/ntopng.pid
> -i=tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556>
>
> /etc/nprobe/nprobe.conf configuration:
>
> -i=none
> -n=none
> -3=2055
> --zmq=tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556>
> -T="@NTOPNG@"
> -V=9
> -d=1
> -t=60
>
> From what I have been able to glean from posts online is that this is in part a limitation of how Netflow works and possible an issue with the idle timeout values? I can’t find an exact timeout number that the Exinda appliance uses. Its documentation mentions that flows are “exported within 10 seconds” and has a 1 minute timeout configured for persistent or long-term flows.
>
> I tried setting the nprobe timeout to both ’10’ and most recently ‘1’ which seems to have made no difference.
>
> It is important to get this information correct. We previously used a programmed called Plixer which did a better job at portraying actual speeds (it also was nowhere near correct but the discrepancy was not significant enough to worry about it too much).
>
> We require this level of precision for reporting and forensics. For example, on a nearly daily basis we will be asked questions such as “why was the Internet slow at our location at X day during Y and Z hours”. I need to be able to piece together what happened on a specific subnet, on certain days during certain timeframes and need to show the data that includes which applications were at fault (often it’s the Netflix and other social media and streaming at fault, or various OS updates).
>
> Any insights and help in fixing this, if possible, would be much appreciated.
> Thanks,
> Gerard Beekmans
> Sr. Network Engineer
> First Nations Technical Services Advisory Group Inc.
> Phone: 780-638-2739 <tel:780-638-2739>
> Fax: 780-483-8632 <tel:780-483-8632>
> Helpdesk: 1-888-999-3356 <tel:1-888-999-3356>
> Email: gbeekmans@tsag.net <mailto:gbeekmans@tsag.net>
> Santa Fe Plaza
> 18232 - 102 Avenue NW
> Edmonton, AB T5S 1S7
> http://www.tsag.net <http://www.tsag.net/>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>

Hi Simone,

Yes, I was primarily referring to the real-time charts as being inaccurate but I am also seeing large inaccuracies in the historic charts.

As of writing this, my actual bandwidth throughput as shown by looking at the router’s interfaces, is about 30 Mbps because it’s so early in the morning and nobody is open for business yet. However, ntopng is showing over 500 Mbps in current traffic with spikes over 1 Gbps.

I can live with the real-time being inaccurate because I can look that up on appliances themselves. I’m mostly interested in historic data for reporting purposes.

The historic data, however, isn’t accurate at the moment either. Here’s what I’ve tried to do.


· In the ntopng web interface, I go to Interfaces -> select the Netflow interface

· I click on the “graph” icon (page=historical)

· For a time range I selected to start yesterday at 16:00 which is the start of our off-hours so it should show little traffic until this morning when business picks up again.

The appliances themselves (router and packet shaper both show the same data) is showing bandwidth after 16:00 yesterday being about a constant 50 Mbps throughout the night. This is a normal pattern.

Ntopng’s shows an average of around 270 Mbps and actual traffic wildly swings up to 700 Mbps which is still not a possible speed based on the ISP capacities involved.

What should the refresh rate on the “config” page be set to? It’s set to 3 seconds by default. Should this match the nprobe timeout I configured or another timeout?

This next graph might give a clue as well. Looking at real time traffic, there is currently a 20 Mbps stream going on to iCloud. It’s correctly showing up as “Apple” protocol in ntopng but it’s not showing a flat line. Attached a screenshot of it, too.

It is going at a steady 20 Mbps for the last 1 hour at least. What I’m seeing in ntop is large peaks instead of a constant flat graph. If this were to be all averaged out, it might be showing close to the actual 20 Mbps expected. The “average” line shows 280 Mbps which is also incorrect. The location this traffic is going to is capped at 20 Mbps (their local ISP limit).

Lastly, the total bytes transferred also doesn’t make much sense to me.

As you can see in the graph for the Apple protocol, it shows in the last 30 minutes a total transfer of 16.85 GB. I can’t reconcile this. A rate of 20 Mbps can only result in about 4.5 GB/minute.

A side-effect of our physical setup might show double the actual numbers due to the fact that traffic is “seen” twice by our packet shaper (the device that is sending Netflow data). Traffic is received from the Internet, shaped, then it’s sent out over a private WAN connection to the remote locations which needs to also flow through the same packet shaper (on a different VLAN but physically travels through the same devices due to the way it’s cabled in-line).

The result is that every upload stream has a matching download stream of the same amount. Even with the doubling-up in mind, I would expect to see double the speed and amounts listed and I can accept that as being a limitation of our setup. But I’m seeing 4x to 10x actual numbers so things still do not make much sense.

Any more insights and tips would be greatly appreciated

Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739
Fax: 780-483-8632
Helpdesk: 1-888-999-3356
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

From: ntop-bounces@listgateway.unipi.it <ntop-bounces@listgateway.unipi.it> On Behalf Of Simone Mainardi
Sent: January 15, 2019 01:49
To: ntop@unipi.it
Cc: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Traffic rates shown are higher than physically possible

Gerard,

When you say wildly inaccurate, are you referring to the realtime charts in the dashboard only? I would expect them to be somehow inaccurate due to the nature of NetFlow, but once you visit the historical pages then totals and speeds must be accurate.

Try and visit the historical charts page of the interface, or the same page for any of your local hosts. Exact data must be there.

If you want to mitigate the inaccuracy of the dashboard, you should reduce the refresh rate to a value that is meaningful, on the basis of the NetFlow timeouts you have configured. To change this, visit the interface page, wheel menu entry, and change the setting.

You can also try and disable the nProbe cache to make sure flows are delivered straight to ntopng as soon as they are received. Option is --disable-cache.

Feel free to attach screenshots if necessary.

Simone



On 14 Jan 2019, at 23:54, Gerard Beekmans <gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>> wrote:

Hi,

I just finished setting up ntong Enterprise along with nprobe Professional. I am seeing traffic rates/speeds that are much higher than what they are, likely due to a misconfiguration on my end or a limitation of the setup.

We’re using an Exinda packet shaper appliance that sends Netflow V9 to nprobe which in turn sends it to ntopng. All the traffic appear to show up in ntopng’s web GUI but with much higher rates than what they are, sometimes 100x faster than what the speeds physically could be.

From our central location we have VPN tunnels established with dozens of remote sites that we provide Internet to. Despite some of those sites only having 2 to 5 Mbps available speed, ntopng reports their flows as hundreds of Mbps.

/etc/ntopng/ntopng.conf configuration:

-G=/var/run/ntopng.pid
-i=tcp://127.0.0.1:5556

/etc/nprobe/nprobe.conf configuration:

-i=none
-n=none
-3=2055
--zmq=tcp://127.0.0.1:5556
-T="@NTOPNG@"
-V=9
-d=1
-t=60

From what I have been able to glean from posts online is that this is in part a limitation of how Netflow works and possible an issue with the idle timeout values? I can’t find an exact timeout number that the Exinda appliance uses. Its documentation mentions that flows are “exported within 10 seconds” and has a 1 minute timeout configured for persistent or long-term flows.

I tried setting the nprobe timeout to both ’10’ and most recently ‘1’ which seems to have made no difference.

It is important to get this information correct. We previously used a programmed called Plixer which did a better job at portraying actual speeds (it also was nowhere near correct but the discrepancy was not significant enough to worry about it too much).

We require this level of precision for reporting and forensics. For example, on a nearly daily basis we will be asked questions such as “why was the Internet slow at our location at X day during Y and Z hours”. I need to be able to piece together what happened on a specific subnet, on certain days during certain timeframes and need to show the data that includes which applications were at fault (often it’s the Netflix and other social media and streaming at fault, or various OS updates).

Any insights and help in fixing this, if possible, would be much appreciated.
Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739<tel:780-638-2739>
Fax: 780-483-8632<tel:780-483-8632>
Helpdesk: 1-888-999-3356<tel:1-888-999-3356>
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop

Hi Simone,

Yes, I was primarily referring to the real-time charts as being inaccurate but I am also seeing large inaccuracies in the historic charts.

As of writing this, my actual bandwidth throughput as shown by looking at the router’s interfaces, is about 30 Mbps because it’s so early in the morning and nobody is open for business yet. However, ntopng is showing over 500 Mbps in current traffic with spikes over 1 Gbps.

I can live with the real-time being inaccurate because I can look that up on appliances themselves. I’m mostly interested in historic data for reporting purposes.

The historic data, however, isn’t accurate at the moment either. Here’s what I’ve tried to do.


· In the ntopng web interface, I go to Interfaces -> select the Netflow interface

· I click on the “graph” icon (page=historical)

· For a time range I selected to start yesterday at 16:00 which is the start of our off-hours so it should show little traffic until this morning when business picks up again.

The appliances themselves (router and packet shaper both show the same data) is showing bandwidth after 16:00 yesterday being about a constant 50 Mbps throughout the night. This is a normal pattern.

Ntopng’s shows an average of around 270 Mbps and actual traffic wildly swings up to 700 Mbps which is still not a possible speed based on the ISP capacities involved.

What should the refresh rate on the “config” page be set to? It’s set to 3 seconds by default. Should this match the nprobe timeout I configured or another timeout?

This next graph might give a clue as well. Looking at real time traffic, there is currently a 20 Mbps stream going on to iCloud. It’s correctly showing up as “Apple” protocol in ntopng but it’s not showing a flat line. Attached a screenshot of it, too.

It is going at a steady 20 Mbps for the last 1 hour at least. What I’m seeing in ntop is large peaks instead of a constant flat graph. If this were to be all averaged out, it might be showing close to the actual 20 Mbps expected. The “average” line shows 280 Mbps which is also incorrect. The location this traffic is going to is capped at 20 Mbps (their local ISP limit).

Lastly, the total bytes transferred also doesn’t make much sense to me.

As you can see in the graph for the Apple protocol, it shows in the last 30 minutes a total transfer of 16.85 GB. I can’t reconcile this. A rate of 20 Mbps can only result in about 4.5 GB/minute.

A side-effect of our physical setup might show double the actual numbers due to the fact that traffic is “seen” twice by our packet shaper (the device that is sending Netflow data). Traffic is received from the Internet, shaped, then it’s sent out over a private WAN connection to the remote locations which needs to also flow through the same packet shaper (on a different VLAN but physically travels through the same devices due to the way it’s cabled in-line).

The result is that every upload stream has a matching download stream of the same amount. Even with the doubling-up in mind, I would expect to see double the speed and amounts listed and I can accept that as being a limitation of our setup. But I’m seeing 4x to 10x actual numbers so things still do not make much sense.

Any more insights and tips would be greatly appreciated

Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739
Fax: 780-483-8632
Helpdesk: 1-888-999-3356
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

From: ntop-bounces@listgateway.unipi.it <ntop-bounces@listgateway.unipi.it> On Behalf Of Simone Mainardi
Sent: January 15, 2019 01:49
To: ntop@unipi.it
Cc: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Traffic rates shown are higher than physically possible

Gerard,

When you say wildly inaccurate, are you referring to the realtime charts in the dashboard only? I would expect them to be somehow inaccurate due to the nature of NetFlow, but once you visit the historical pages then totals and speeds must be accurate.

Try and visit the historical charts page of the interface, or the same page for any of your local hosts. Exact data must be there.

If you want to mitigate the inaccuracy of the dashboard, you should reduce the refresh rate to a value that is meaningful, on the basis of the NetFlow timeouts you have configured. To change this, visit the interface page, wheel menu entry, and change the setting.

You can also try and disable the nProbe cache to make sure flows are delivered straight to ntopng as soon as they are received. Option is --disable-cache.

Feel free to attach screenshots if necessary.

Simone



On 14 Jan 2019, at 23:54, Gerard Beekmans <gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>> wrote:

Hi,

I just finished setting up ntong Enterprise along with nprobe Professional. I am seeing traffic rates/speeds that are much higher than what they are, likely due to a misconfiguration on my end or a limitation of the setup.

We’re using an Exinda packet shaper appliance that sends Netflow V9 to nprobe which in turn sends it to ntopng. All the traffic appear to show up in ntopng’s web GUI but with much higher rates than what they are, sometimes 100x faster than what the speeds physically could be.

From our central location we have VPN tunnels established with dozens of remote sites that we provide Internet to. Despite some of those sites only having 2 to 5 Mbps available speed, ntopng reports their flows as hundreds of Mbps.

/etc/ntopng/ntopng.conf configuration:

-G=/var/run/ntopng.pid
-i=tcp://127.0.0.1:5556

/etc/nprobe/nprobe.conf configuration:

-i=none
-n=none
-3=2055
--zmq=tcp://127.0.0.1:5556
-T="@NTOPNG@"
-V=9
-d=1
-t=60

From what I have been able to glean from posts online is that this is in part a limitation of how Netflow works and possible an issue with the idle timeout values? I can’t find an exact timeout number that the Exinda appliance uses. Its documentation mentions that flows are “exported within 10 seconds” and has a 1 minute timeout configured for persistent or long-term flows.

I tried setting the nprobe timeout to both ’10’ and most recently ‘1’ which seems to have made no difference.

It is important to get this information correct. We previously used a programmed called Plixer which did a better job at portraying actual speeds (it also was nowhere near correct but the discrepancy was not significant enough to worry about it too much).

We require this level of precision for reporting and forensics. For example, on a nearly daily basis we will be asked questions such as “why was the Internet slow at our location at X day during Y and Z hours”. I need to be able to piece together what happened on a specific subnet, on certain days during certain timeframes and need to show the data that includes which applications were at fault (often it’s the Netflix and other social media and streaming at fault, or various OS updates).

Any insights and help in fixing this, if possible, would be much appreciated.
Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739<tel:780-638-2739>
Fax: 780-483-8632<tel:780-483-8632>
Helpdesk: 1-888-999-3356<tel:1-888-999-3356>
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop

Hi all,

In an attempt to see if my issues are with the Netflow data itself I deployed an evaluation version of a product called Scrutinizer by Plixer. Without changing any settings, it is reporting traffic the way one would expect to. I have attached a screenshot. It’s able to deal with the duplicate flows as traffic flows through the appliance twice but is able to handle it gracefully by keeping track of the interface pairs at play from the appliance.

As you can see in the screenshot, it shows sane Mbps rates for the various applications and protocols and filters allow me to only see the subnets or individual hosts I’m interested in.

Real-time data is also reporting correctly (showing a few hundred Mbps aggregate throughput vs the several Gbps).

I would very much prefer to use ntopng and nprobe software in the spirit of Open Source. I would be very keen on getting ntopng/nprobe to work similarly so that I can get the same viewing capabilities. I’m happy to devote resources to make this project a success.



Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739
Fax: 780-483-8632
Helpdesk: 1-888-999-3356
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

From: ntop-bounces@listgateway.unipi.it <ntop-bounces@listgateway.unipi.it> On Behalf Of Gerard Beekmans
Sent: January 15, 2019 08:37
To: ntop@unipi.it
Cc: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Traffic rates shown are higher than physically possible

Hi Simone,

Yes, I was primarily referring to the real-time charts as being inaccurate but I am also seeing large inaccuracies in the historic charts.

As of writing this, my actual bandwidth throughput as shown by looking at the router’s interfaces, is about 30 Mbps because it’s so early in the morning and nobody is open for business yet. However, ntopng is showing over 500 Mbps in current traffic with spikes over 1 Gbps.

I can live with the real-time being inaccurate because I can look that up on appliances themselves. I’m mostly interested in historic data for reporting purposes.

The historic data, however, isn’t accurate at the moment either. Here’s what I’ve tried to do.


· In the ntopng web interface, I go to Interfaces -> select the Netflow interface

· I click on the “graph” icon (page=historical)

· For a time range I selected to start yesterday at 16:00 which is the start of our off-hours so it should show little traffic until this morning when business picks up again.

The appliances themselves (router and packet shaper both show the same data) is showing bandwidth after 16:00 yesterday being about a constant 50 Mbps throughout the night. This is a normal pattern.

Ntopng’s shows an average of around 270 Mbps and actual traffic wildly swings up to 700 Mbps which is still not a possible speed based on the ISP capacities involved.

What should the refresh rate on the “config” page be set to? It’s set to 3 seconds by default. Should this match the nprobe timeout I configured or another timeout?

This next graph might give a clue as well. Looking at real time traffic, there is currently a 20 Mbps stream going on to iCloud. It’s correctly showing up as “Apple” protocol in ntopng but it’s not showing a flat line. Attached a screenshot of it, too.

It is going at a steady 20 Mbps for the last 1 hour at least. What I’m seeing in ntop is large peaks instead of a constant flat graph. If this were to be all averaged out, it might be showing close to the actual 20 Mbps expected. The “average” line shows 280 Mbps which is also incorrect. The location this traffic is going to is capped at 20 Mbps (their local ISP limit).

Lastly, the total bytes transferred also doesn’t make much sense to me.

As you can see in the graph for the Apple protocol, it shows in the last 30 minutes a total transfer of 16.85 GB. I can’t reconcile this. A rate of 20 Mbps can only result in about 4.5 GB/minute.

A side-effect of our physical setup might show double the actual numbers due to the fact that traffic is “seen” twice by our packet shaper (the device that is sending Netflow data). Traffic is received from the Internet, shaped, then it’s sent out over a private WAN connection to the remote locations which needs to also flow through the same packet shaper (on a different VLAN but physically travels through the same devices due to the way it’s cabled in-line).

The result is that every upload stream has a matching download stream of the same amount. Even with the doubling-up in mind, I would expect to see double the speed and amounts listed and I can accept that as being a limitation of our setup. But I’m seeing 4x to 10x actual numbers so things still do not make much sense.

Any more insights and tips would be greatly appreciated

Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739
Fax: 780-483-8632
Helpdesk: 1-888-999-3356
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> <ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it>> On Behalf Of Simone Mainardi
Sent: January 15, 2019 01:49
To: ntop@unipi.it<mailto:ntop@unipi.it>
Cc: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>
Subject: Re: [Ntop] Traffic rates shown are higher than physically possible

Gerard,

When you say wildly inaccurate, are you referring to the realtime charts in the dashboard only? I would expect them to be somehow inaccurate due to the nature of NetFlow, but once you visit the historical pages then totals and speeds must be accurate.

Try and visit the historical charts page of the interface, or the same page for any of your local hosts. Exact data must be there.

If you want to mitigate the inaccuracy of the dashboard, you should reduce the refresh rate to a value that is meaningful, on the basis of the NetFlow timeouts you have configured. To change this, visit the interface page, wheel menu entry, and change the setting.

You can also try and disable the nProbe cache to make sure flows are delivered straight to ntopng as soon as they are received. Option is --disable-cache.

Feel free to attach screenshots if necessary.

Simone


On 14 Jan 2019, at 23:54, Gerard Beekmans <gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>> wrote:

Hi,

I just finished setting up ntong Enterprise along with nprobe Professional. I am seeing traffic rates/speeds that are much higher than what they are, likely due to a misconfiguration on my end or a limitation of the setup.

We’re using an Exinda packet shaper appliance that sends Netflow V9 to nprobe which in turn sends it to ntopng. All the traffic appear to show up in ntopng’s web GUI but with much higher rates than what they are, sometimes 100x faster than what the speeds physically could be.

From our central location we have VPN tunnels established with dozens of remote sites that we provide Internet to. Despite some of those sites only having 2 to 5 Mbps available speed, ntopng reports their flows as hundreds of Mbps.

/etc/ntopng/ntopng.conf configuration:

-G=/var/run/ntopng.pid
-i=tcp://127.0.0.1:5556

/etc/nprobe/nprobe.conf configuration:

-i=none
-n=none
-3=2055
--zmq=tcp://127.0.0.1:5556
-T="@NTOPNG@"
-V=9
-d=1
-t=60

From what I have been able to glean from posts online is that this is in part a limitation of how Netflow works and possible an issue with the idle timeout values? I can’t find an exact timeout number that the Exinda appliance uses. Its documentation mentions that flows are “exported within 10 seconds” and has a 1 minute timeout configured for persistent or long-term flows.

I tried setting the nprobe timeout to both ’10’ and most recently ‘1’ which seems to have made no difference.

It is important to get this information correct. We previously used a programmed called Plixer which did a better job at portraying actual speeds (it also was nowhere near correct but the discrepancy was not significant enough to worry about it too much).

We require this level of precision for reporting and forensics. For example, on a nearly daily basis we will be asked questions such as “why was the Internet slow at our location at X day during Y and Z hours”. I need to be able to piece together what happened on a specific subnet, on certain days during certain timeframes and need to show the data that includes which applications were at fault (often it’s the Netflix and other social media and streaming at fault, or various OS updates).

Any insights and help in fixing this, if possible, would be much appreciated.
Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739<tel:780-638-2739>
Fax: 780-483-8632<tel:780-483-8632>
Helpdesk: 1-888-999-3356<tel:1-888-999-3356>
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop

Hi all,

In an attempt to see if my issues are with the Netflow data itself I deployed an evaluation version of a product called Scrutinizer by Plixer. Without changing any settings, it is reporting traffic the way one would expect to. I have attached a screenshot. It’s able to deal with the duplicate flows as traffic flows through the appliance twice but is able to handle it gracefully by keeping track of the interface pairs at play from the appliance.

As you can see in the screenshot, it shows sane Mbps rates for the various applications and protocols and filters allow me to only see the subnets or individual hosts I’m interested in.

Real-time data is also reporting correctly (showing a few hundred Mbps aggregate throughput vs the several Gbps).

I would very much prefer to use ntopng and nprobe software in the spirit of Open Source. I would be very keen on getting ntopng/nprobe to work similarly so that I can get the same viewing capabilities. I’m happy to devote resources to make this project a success.



Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739
Fax: 780-483-8632
Helpdesk: 1-888-999-3356
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

From: ntop-bounces@listgateway.unipi.it <ntop-bounces@listgateway.unipi.it> On Behalf Of Gerard Beekmans
Sent: January 15, 2019 08:37
To: ntop@unipi.it
Cc: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Traffic rates shown are higher than physically possible

Hi Simone,

Yes, I was primarily referring to the real-time charts as being inaccurate but I am also seeing large inaccuracies in the historic charts.

As of writing this, my actual bandwidth throughput as shown by looking at the router’s interfaces, is about 30 Mbps because it’s so early in the morning and nobody is open for business yet. However, ntopng is showing over 500 Mbps in current traffic with spikes over 1 Gbps.

I can live with the real-time being inaccurate because I can look that up on appliances themselves. I’m mostly interested in historic data for reporting purposes.

The historic data, however, isn’t accurate at the moment either. Here’s what I’ve tried to do.


· In the ntopng web interface, I go to Interfaces -> select the Netflow interface

· I click on the “graph” icon (page=historical)

· For a time range I selected to start yesterday at 16:00 which is the start of our off-hours so it should show little traffic until this morning when business picks up again.

The appliances themselves (router and packet shaper both show the same data) is showing bandwidth after 16:00 yesterday being about a constant 50 Mbps throughout the night. This is a normal pattern.

Ntopng’s shows an average of around 270 Mbps and actual traffic wildly swings up to 700 Mbps which is still not a possible speed based on the ISP capacities involved.

What should the refresh rate on the “config” page be set to? It’s set to 3 seconds by default. Should this match the nprobe timeout I configured or another timeout?

This next graph might give a clue as well. Looking at real time traffic, there is currently a 20 Mbps stream going on to iCloud. It’s correctly showing up as “Apple” protocol in ntopng but it’s not showing a flat line. Attached a screenshot of it, too.

It is going at a steady 20 Mbps for the last 1 hour at least. What I’m seeing in ntop is large peaks instead of a constant flat graph. If this were to be all averaged out, it might be showing close to the actual 20 Mbps expected. The “average” line shows 280 Mbps which is also incorrect. The location this traffic is going to is capped at 20 Mbps (their local ISP limit).

Lastly, the total bytes transferred also doesn’t make much sense to me.

As you can see in the graph for the Apple protocol, it shows in the last 30 minutes a total transfer of 16.85 GB. I can’t reconcile this. A rate of 20 Mbps can only result in about 4.5 GB/minute.

A side-effect of our physical setup might show double the actual numbers due to the fact that traffic is “seen” twice by our packet shaper (the device that is sending Netflow data). Traffic is received from the Internet, shaped, then it’s sent out over a private WAN connection to the remote locations which needs to also flow through the same packet shaper (on a different VLAN but physically travels through the same devices due to the way it’s cabled in-line).

The result is that every upload stream has a matching download stream of the same amount. Even with the doubling-up in mind, I would expect to see double the speed and amounts listed and I can accept that as being a limitation of our setup. But I’m seeing 4x to 10x actual numbers so things still do not make much sense.

Any more insights and tips would be greatly appreciated

Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739
Fax: 780-483-8632
Helpdesk: 1-888-999-3356
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> <ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it>> On Behalf Of Simone Mainardi
Sent: January 15, 2019 01:49
To: ntop@unipi.it<mailto:ntop@unipi.it>
Cc: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>
Subject: Re: [Ntop] Traffic rates shown are higher than physically possible

Gerard,

When you say wildly inaccurate, are you referring to the realtime charts in the dashboard only? I would expect them to be somehow inaccurate due to the nature of NetFlow, but once you visit the historical pages then totals and speeds must be accurate.

Try and visit the historical charts page of the interface, or the same page for any of your local hosts. Exact data must be there.

If you want to mitigate the inaccuracy of the dashboard, you should reduce the refresh rate to a value that is meaningful, on the basis of the NetFlow timeouts you have configured. To change this, visit the interface page, wheel menu entry, and change the setting.

You can also try and disable the nProbe cache to make sure flows are delivered straight to ntopng as soon as they are received. Option is --disable-cache.

Feel free to attach screenshots if necessary.

Simone


On 14 Jan 2019, at 23:54, Gerard Beekmans <gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>> wrote:

Hi,

I just finished setting up ntong Enterprise along with nprobe Professional. I am seeing traffic rates/speeds that are much higher than what they are, likely due to a misconfiguration on my end or a limitation of the setup.

We’re using an Exinda packet shaper appliance that sends Netflow V9 to nprobe which in turn sends it to ntopng. All the traffic appear to show up in ntopng’s web GUI but with much higher rates than what they are, sometimes 100x faster than what the speeds physically could be.

From our central location we have VPN tunnels established with dozens of remote sites that we provide Internet to. Despite some of those sites only having 2 to 5 Mbps available speed, ntopng reports their flows as hundreds of Mbps.

/etc/ntopng/ntopng.conf configuration:

-G=/var/run/ntopng.pid
-i=tcp://127.0.0.1:5556

/etc/nprobe/nprobe.conf configuration:

-i=none
-n=none
-3=2055
--zmq=tcp://127.0.0.1:5556
-T="@NTOPNG@"
-V=9
-d=1
-t=60

From what I have been able to glean from posts online is that this is in part a limitation of how Netflow works and possible an issue with the idle timeout values? I can’t find an exact timeout number that the Exinda appliance uses. Its documentation mentions that flows are “exported within 10 seconds” and has a 1 minute timeout configured for persistent or long-term flows.

I tried setting the nprobe timeout to both ’10’ and most recently ‘1’ which seems to have made no difference.

It is important to get this information correct. We previously used a programmed called Plixer which did a better job at portraying actual speeds (it also was nowhere near correct but the discrepancy was not significant enough to worry about it too much).

We require this level of precision for reporting and forensics. For example, on a nearly daily basis we will be asked questions such as “why was the Internet slow at our location at X day during Y and Z hours”. I need to be able to piece together what happened on a specific subnet, on certain days during certain timeframes and need to show the data that includes which applications were at fault (often it’s the Netflix and other social media and streaming at fault, or various OS updates).

Any insights and help in fixing this, if possible, would be much appreciated.
Thanks,
Gerard Beekmans
Sr. Network Engineer
First Nations Technical Services Advisory Group Inc.
Phone: 780-638-2739<tel:780-638-2739>
Fax: 780-483-8632<tel:780-483-8632>
Helpdesk: 1-888-999-3356<tel:1-888-999-3356>
Email: gbeekmans@tsag.net<mailto:gbeekmans@tsag.net>
Santa Fe Plaza
18232 - 102 Avenue NW
Edmonton, AB T5S 1S7
http://www.tsag.net<http://www.tsag.net/>

_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop