Mailing List Archive

Traffic sent and traffic received in historical view in ntopng
Hi,

Can someone please tell me what I need to do to get differentiated traffic
sent and received in the 'historical view' on an interface?

Thanks in advance.

Andrew
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Hi Andrew,
Currently ntopng saves the interface RRD data as the total number of
bytes, so it not possible to get a differentiated sent/received view.
However, you can get such differentiated metrics from the Local Networks
page.

Emanuele

On 05/12/2017 08:02 PM, Andrew Hilborne wrote:
> Hi,
>
> Can someone please tell me what I need to do to get differentiated
> traffic sent and received in the 'historical view' on an interface?
>
> Thanks in advance.
>
> Andrew
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
I believe I know what you mean:
http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/ shows
exactly this information. However my own installation shows a similar
graphic, but without the 'traffic (Sent)' and 'traffic (Recvd)'
designations.

Andrew

On 15 May 2017 at 10:18, Emanuele Faranda faranda-at-ntop.org |ntop-flugle|
<rrjzg9nx7t@sneakemail.com> wrote:

> Hi Andrew,
> Currently ntopng saves the interface RRD data as the total number of
> bytes, so it not possible to get a differentiated sent/received view.
> However, you can get such differentiated metrics from the Local Networks
> page.
>
> Emanuele
>
>
> On 05/12/2017 08:02 PM, Andrew Hilborne wrote:
>
> Hi,
>
> Can someone please tell me what I need to do to get differentiated traffic
> sent and received in the 'historical view' on an interface?
>
> Thanks in advance.
>
> Andrew
>
>
> _______________________________________________
> Ntop mailing listNtop@listgateway.unipi.ithttp://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
The graph at the link shows *host* traffic, which indeed is split
between sent and received. The *interface* graph instead does not make
such a distinction.

In ntopng we have graphs for:
- interfaces: total bytes
- local hosts: sent, received bytes
- local networks: ingress, egress, inner bytes
- .. and others

If you really want *interface* traffic to be split, please open a
feature request on our github page https://github.com/ntop/ntopng.

Regards,
Emanuele

On 05/15/2017 04:11 PM, Andrew Hilborne wrote:
> I believe I know what you mean:
> http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/
> shows exactly this information. However my own installation shows a
> similar graphic, but without the 'traffic (Sent)' and 'traffic
> (Recvd)' designations.
>
> Andrew
>
> On 15 May 2017 at 10:18, Emanuele Faranda faranda-at-ntop.org
> <http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com
> <mailto:rrjzg9nx7t@sneakemail.com>> wrote:
>
> Hi Andrew,
> Currently ntopng saves the interface RRD data as the total number
> of bytes, so it not possible to get a differentiated sent/received
> view.
> However, you can get such differentiated metrics from the Local
> Networks page.
>
> Emanuele
>
>
> On 05/12/2017 08:02 PM, Andrew Hilborne wrote:
>> Hi,
>>
>> Can someone please tell me what I need to do to get
>> differentiated traffic sent and received in the 'historical view'
>> on an interface?
>>
>> Thanks in advance.
>>
>> Andrew
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>> <http://listgateway.unipi.it/mailman/listinfo/ntop>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop
> <http://listgateway.unipi.it/mailman/listinfo/ntop>
>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
It's the ingress/egress bytes *per local network* which I am interested in.
Thank you - I have found this now.

However, it is showing no data. I think this is a licensed feature?

Andrew

On 15 May 2017 at 15:21, Emanuele Faranda faranda-at-ntop.org |ntop-flugle|
<rrjzg9nx7t@sneakemail.com> wrote:

> The graph at the link shows *host* traffic, which indeed is split between
> sent and received. The *interface* graph instead does not make such a
> distinction.
>
> In ntopng we have graphs for:
> - interfaces: total bytes
> - local hosts: sent, received bytes
> - local networks: ingress, egress, inner bytes
> - .. and others
>
> If you really want *interface* traffic to be split, please open a feature
> request on our github page https://github.com/ntop/ntopng.
>
> Regards,
> Emanuele
>
>
> On 05/15/2017 04:11 PM, Andrew Hilborne wrote:
>
> I believe I know what you mean: http://www.ntop.org/
> ntopng/exploring-historical-data-using-ntopng/ shows exactly this
> information. However my own installation shows a similar graphic, but
> without the 'traffic (Sent)' and 'traffic (Recvd)' designations.
>
> Andrew
>
> On 15 May 2017 at 10:18, Emanuele Faranda faranda-at-ntop.org
> |ntop-flugle| <rrjzg9nx7t@sneakemail.com> wrote:
>
>> Hi Andrew,
>> Currently ntopng saves the interface RRD data as the total number of
>> bytes, so it not possible to get a differentiated sent/received view.
>> However, you can get such differentiated metrics from the Local Networks
>> page.
>>
>> Emanuele
>>
>>
>> On 05/12/2017 08:02 PM, Andrew Hilborne wrote:
>>
>> Hi,
>>
>> Can someone please tell me what I need to do to get differentiated
>> traffic sent and received in the 'historical view' on an interface?
>>
>> Thanks in advance.
>>
>> Andrew
>>
>>
>> _______________________________________________
>> Ntop mailing listNtop@listgateway.unipi.ithttp://listgateway.unipi.it/mailman/listinfo/ntop
>>
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>
>
>
> _______________________________________________
> Ntop mailing listNtop@listgateway.unipi.ithttp://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
No, it is also available in the community version.
It should be enabled by default, please verify that the "Traffic"
preference under "Data Retention" settings is enabled.

Note: local networks data is dumped to disk each 5 minutes.

Emanuele

On 05/15/2017 04:38 PM, Andrew Hilborne wrote:
> It's the ingress/egress bytes _per local network_ which I am
> interested in. Thank you - I have found this now.
>
> However, it is showing no data. I think this is a licensed feature?
>
> Andrew
>
> On 15 May 2017 at 15:21, Emanuele Faranda faranda-at-ntop.org
> <http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com
> <mailto:rrjzg9nx7t@sneakemail.com>> wrote:
>
> The graph at the link shows *host* traffic, which indeed is split
> between sent and received. The *interface* graph instead does not
> make such a distinction.
>
> In ntopng we have graphs for:
> - interfaces: total bytes
> - local hosts: sent, received bytes
> - local networks: ingress, egress, inner bytes
> - .. and others
>
> If you really want *interface* traffic to be split, please open a
> feature request on our github page https://github.com/ntop/ntopng.
>
> Regards,
> Emanuele
>
>
> On 05/15/2017 04:11 PM, Andrew Hilborne wrote:
>> I believe I know what you mean:
>> http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/
>> <http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/>
>> shows exactly this information. However my own installation shows
>> a similar graphic, but without the 'traffic (Sent)' and 'traffic
>> (Recvd)' designations.
>>
>> Andrew
>>
>> On 15 May 2017 at 10:18, Emanuele Faranda faranda-at-ntop.org
>> <http://faranda-at-ntop.org> |ntop-flugle|
>> <rrjzg9nx7t@sneakemail.com <mailto:rrjzg9nx7t@sneakemail.com>> wrote:
>>
>> Hi Andrew,
>> Currently ntopng saves the interface RRD data as the total
>> number of bytes, so it not possible to get a differentiated
>> sent/received view.
>> However, you can get such differentiated metrics from the
>> Local Networks page.
>>
>> Emanuele
>>
>>
>> On 05/12/2017 08:02 PM, Andrew Hilborne wrote:
>>> Hi,
>>>
>>> Can someone please tell me what I need to do to get
>>> differentiated traffic sent and received in the 'historical
>>> view' on an interface?
>>>
>>> Thanks in advance.
>>>
>>> Andrew
>>>
>>>
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>> <http://listgateway.unipi.it/mailman/listinfo/ntop>
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>> <http://listgateway.unipi.it/mailman/listinfo/ntop>
>>
>>
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>> <http://listgateway.unipi.it/mailman/listinfo/ntop>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop
> <http://listgateway.unipi.it/mailman/listinfo/ntop>
>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
On 15 May 2017 at 15:44, Emanuele Faranda faranda-at-ntop.org |ntop-flugle|
<rrjzg9nx7t@sneakemail.com> wrote:

> No, it is also available in the community version.
> It should be enabled by default, please verify that the "Traffic"
> preference under "Data Retention" settings is enabled.
>
> Note: local networks data is dumped to disk each 5 minutes.
>

?My SQL Database > Data retention is set to 30 days.

There is no traffic being measured at the moment, but all I can see at
localhost:3000/lua/network_stats.lua is "No results found".

Are per-interface stats collected with better granularity than per-network
stats?? My impression was that per-network stats are possibly updated every
minute - which is much better for detecting peaks, though could still be
better.

Andrew
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Hi Andrew,

You are right, network stats are calculated every minute, whereas
interface stats are updated each second.

Please note that these stats are dumped to RRD files, not to the MySQL
database.

For the "No results found" issue please open an issue on github with
detailed information on software version and configuration used.


Note: the fact that in/out breakdown for interface traffic is not
available is already tracked by issue
https://github.com/ntop/ntopng/issues/1114

Regards,
Emanuele

On 05/15/2017 05:08 PM, Andrew Hilborne wrote:
> On 15 May 2017 at 15:44, Emanuele Faranda faranda-at-ntop.org
> <http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com
> <mailto:rrjzg9nx7t@sneakemail.com>>wrote:
>
> No, it is also available in the community version.
> It should be enabled by default, please verify that the "Traffic"
> preference under "Data Retention" settings is enabled.
>
> Note: local networks data is dumped to disk each 5 minutes.
>
>
> ?My SQL Database > Data retentionis set to 30 days.
>
> There is no traffic being measured at the moment, but all I can see at
> localhost:3000/lua/network_stats.luais "No results found".
>
> Are per-interface stats collected with better granularity than
> per-network stats?? My impression was that per-network stats are
> possibly updated every minute - which is much better for detecting
> peaks, though could still be better.
>
> Andrew
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
On 15 May 2017 at 17:10, Emanuele Faranda faranda-at-ntop.org |ntop-flugle|
<rrjzg9nx7t@sneakemail.com> wrote:

> You are right, network stats are calculated every minute, whereas
> interface stats are updated each second.
>
> Please note that these stats are dumped to RRD files, not to the MySQL
> database.
>
?Would it be possible t change this? Is the issue storage space in the
MySQL database? This is what I want to know (initially):

- How much traffic is coming into and leaving my local network?
- What is the traffic breakdown?

I don't believe that this is an uncommon requirement. In the first place,
looking at per-host flows isn't nearly so informative.

Andrew
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
I'm also interested in this. I can get from my ISP daily totals for our internet usage. I would like ntopng to be able to replicate those daily totals (to give confidence our data is correct), and then analyse the totals to see which devices contributed. Eg. If we have an above average daily total, I want to know why.

Peter Shute

Sent from my iPad

On 16 May 2017, at 4:01 am, Andrew Hilborne <ntop-flugle@snkmail.com<mailto:ntop-flugle@snkmail.com>> wrote:

On 15 May 2017 at 17:10, Emanuele Faranda faranda-at-ntop.org<http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com<mailto:rrjzg9nx7t@sneakemail.com>> wrote:

You are right, network stats are calculated every minute, whereas interface stats are updated each second.

Please note that these stats are dumped to RRD files, not to the MySQL database.

?Would it be possible t change this? Is the issue storage space in the MySQL database? This is what I want to know (initially):

* How much traffic is coming into and leaving my local network?
* What is the traffic breakdown?

I don't believe that this is an uncommon requirement. In the first place, looking at per-host flows isn't nearly so informative.

Andrew


_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Hi Peter and Andrew,

Please see below.


On 05/16/2017 01:33 PM, Peter Shute wrote:
> I'm also interested in this. I can get from my ISP daily totals for our internet usage. I would like ntopng to be able to replicate those daily totals (to give confidence our data is correct), and then analyse the totals to see which devices contributed. Eg. If we have an above average daily total, I want to know why.
Ntopng can actually produce a traffic report where it shows the top
local/remote talkers for a specified time frame (e.g. a day), but this
is a pro only feature.

In enterprise version we also have a per host/network/interface report
showing the total traffic in a day/week/month with comparison to the
previous day/week/month.
Note: the traffic is not split into sent/received.

> Peter Shute
>
> Sent from my iPad
>
> On 16 May 2017, at 4:01 am, Andrew Hilborne <ntop-flugle@snkmail.com<mailto:ntop-flugle@snkmail.com>> wrote:
>
> On 15 May 2017 at 17:10, Emanuele Faranda faranda-at-ntop.org<http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com<mailto:rrjzg9nx7t@sneakemail.com>> wrote:
>
> You are right, network stats are calculated every minute, whereas interface stats are updated each second.
>
> Please note that these stats are dumped to RRD files, not to the MySQL database.
>
> ?Would it be possible t change this? Is the issue storage space in the MySQL database? This is what I want to know (initially):
No, this is something different. MySQL database exports /flows/ as data,
whereas RRD is /timeseries/ database, so they play different roles.
If you export to MySQL using the -F option, you can do your own
computations and aggregations on data.
> * How much traffic is coming into and leaving my local network?
> * What is the traffic breakdown?
Right now you have the graphs on the network which show you the traffic.
If you hover the graph with the mouse, a table on the right will appear
and it reports the "Total Traffic" for the specified time frame (1d in
your case).
But the graphs totals are not split into ingress/egress, so please open
a feature request so that we can evaluate the proposal.

Regards,
Emanuele
>
> I don't believe that this is an uncommon requirement. In the first place, looking at per-host flows isn't nearly so informative.
>
> Andrew
>
>
> On 05/15/2017 08:00 PM, Andrew Hilborne wrote:
>
> You are right, network stats are calculated every minute, whereas
> interface stats are updated each second.
>
> Please note that these stats are dumped to RRD files, not to the
> MySQL database.
>
> ?Would it be possible t change this? Is the issue storage space in the
> MySQL database? This is what I want to know (initially):
>
> * How much traffic is coming into and leaving my local network?
> * What is the traffic breakdown?
>
> I don't believe that this is an uncommon requirement. In the first
> place, looking at per-host flows isn't nearly so informative.
>
> Andrew
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Where do I find that report, please? Do I have access to it? The GUI says I'm running "ntopng Pro [Small Business Edition] v.2.5.170323".

Peter Shute

> -----Original Message-----
> From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
> bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
> Sent: Wednesday, 17 May 2017 4:29 AM
> To: ntop@listgateway.unipi.it
> Subject: Re: [Ntop] Traffic sent and traffic received in historical view in
> ntopng

> Ntopng can actually produce a traffic report where it shows the top
> local/remote talkers for a specified time frame (e.g. a day), but this is a pro
> only feature.
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Please see this image


Emanuele

On 05/17/2017 12:05 AM, Peter Shute wrote:
> Where do I find that report, please? Do I have access to it? The GUI says I'm running "ntopng Pro [Small Business Edition] v.2.5.170323".
>
> Peter Shute
>
>> -----Original Message-----
>> From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
>> bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
>> Sent: Wednesday, 17 May 2017 4:29 AM
>> To: ntop@listgateway.unipi.it
>> Subject: Re: [Ntop] Traffic sent and traffic received in historical view in
>> ntopng
>> Ntopng can actually produce a traffic report where it shows the top
>> local/remote talkers for a specified time frame (e.g. a day), but this is a pro
>> only feature.
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
OK, thanks, I didn’t spot those right down the bottom of the report.

One problem I have with it is that it takes so much work to run it for a calendar day – midnight to midnight.

It also doesn’t resolve remote ip addresses like some other pages do.



From: ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
Sent: Wednesday, 17 May 2017 8:11 AM
To: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Traffic sent and traffic received in historical view in ntopng


Please see this image



[cid:image002.png@01D2CEFC.785C48C0]
Emanuele
On 05/17/2017 12:05 AM, Peter Shute wrote:

Where do I find that report, please? Do I have access to it? The GUI says I'm running "ntopng Pro [Small Business Edition] v.2.5.170323".



Peter Shute



-----Original Message-----

From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> [mailto:ntop-

bounces@listgateway.unipi.it<mailto:bounces@listgateway.unipi.it>] On Behalf Of Emanuele Faranda

Sent: Wednesday, 17 May 2017 4:29 AM

To: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>

Subject: Re: [Ntop] Traffic sent and traffic received in historical view in

ntopng



Ntopng can actually produce a traffic report where it shows the top

local/remote talkers for a specified time frame (e.g. a day), but this is a pro

only feature.

_______________________________________________

Ntop mailing list

Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>

http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Peter,


On Wed, May 17, 2017 at 3:52 AM, Peter Shute <pshute@nuw.org.au> wrote:

> OK, thanks, I didn’t spot those right down the bottom of the report.
>
>
>
> One problem I have with it is that it takes so much work to run it for a
> calendar day – midnight to midnight.
>

We implemented several optimizations that make report gereration faster,
are you using the latest dev build?


>
> It also doesn’t resolve remote ip addresses like some other pages do.
>
>
>
>
> *From:* ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@
> listgateway.unipi.it] *On Behalf Of *Emanuele Faranda
> *Sent:* Wednesday, 17 May 2017 8:11 AM
> *To:* ntop@listgateway.unipi.it
> *Subject:* Re: [Ntop] Traffic sent and traffic received in historical
> view in ntopng
>
>
>
> Please see this image
>
>
>
> Emanuele
>
> On 05/17/2017 12:05 AM, Peter Shute wrote:
>
> Where do I find that report, please? Do I have access to it? The GUI says I'm running "ntopng Pro [Small Business Edition] v.2.5.170323".
>
>
>
> Peter Shute
>
>
>
> -----Original Message-----
>
> From: ntop-bounces@listgateway.unipi.it [mailto:ntop <ntop>-
>
> bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
>
> Sent: Wednesday, 17 May 2017 4:29 AM
>
> To: ntop@listgateway.unipi.it
>
> Subject: Re: [Ntop] Traffic sent and traffic received in historical view in
>
> ntopng
>
>
>
> Ntopng can actually produce a traffic report where it shows the top
>
> local/remote talkers for a specified time frame (e.g. a day), but this is a pro
>
> only feature.
>
> _______________________________________________
>
> Ntop mailing list
>
> Ntop@listgateway.unipi.it
>
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
On 16 May 2017 at 19:29, Emanuele Faranda faranda-at-ntop.org |ntop-flugle|
<rrjzg9nx7t@sneakemail.com> wrote:

> Hi Peter and Andrew,
>
> Please see below.
>
>> On 05/16/2017 01:33 PM, Peter Shute wrote:
>> I'm also interested in this. I can get from my ISP daily totals for our
>> internet usage. I would like ntopng to be able to replicate those daily
>> totals (to give confidence our data is correct), and then analyse the
>> totals to see which devices contributed. Eg. If we have an above average
>> daily total, I want to know why.
>> ??
>>
>
> Ntopng can actually produce a traffic report where it shows the top
> local/remote talkers for a specified time frame (e.g. a day), but this is a
> pro only feature.
>

?I don't object to paying for the license, but this still doesn't get me
what I want, I think. I want to look back over a historical graph (or jump
to a time and day, if I believe there was a problem at that time) and drill
down to see the protocols invloved, and the hosts. Top 10 talkers may not
include the information I want.?

> On 16 May 2017, at 4:01 am, Andrew Hilborne <ntop-flugle@snkmail.com<mailto:ntop-flugle@snkmail.com> <ntop-flugle@snkmail.com>> wrote:
> ??
>
> On 15 May 2017 at 17:10, Emanuele Faranda faranda-at-ntop.org<http://faranda-at-ntop.org> <http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com<mailto:rrjzg9nx7t@sneakemail.com> <rrjzg9nx7t@sneakemail.com>> wrote:
>
> You are right, network stats are calculated every minute, whereas interface stats are updated each second.
>
> Please note that these stats are dumped to RRD files, not to the MySQL database.
>
> ?Would it be possible t change this? Is the issue storage space in the MySQL database? This is what I want to know (initially):
>
> No, this is something different. MySQL database exports *flows* as data,
> whereas RRD is *timeseries* database, so they play different roles.
>

?I do understand the difference between flows and and the RDD timeseries.
However, typical 5-minute RDD data is useless for investigating traffic
peaks; I think you may know this, because n2disk can now detect
'micro-bursts'?. I don't suggest that storing sufficient information to
provide a near real-time breakdown of traffic is easy, but it would be
interesting. If I am reduced to going back to using RDDtool type data,
there are better tools than ntopng for that purpose.

Maybe I'm not using ntopng properly?

Andrew
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Andrew,

Please see below.


On 05/17/2017 07:17 PM, Andrew Hilborne wrote:
> On 16 May 2017 at 19:29, Emanuele Faranda faranda-at-ntop.org
> <http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com
> <mailto:rrjzg9nx7t@sneakemail.com>>wrote:
>
> Hi Peter and Andrew,
>
> Please see below.
>
> On 05/16/2017 01:33 PM, Peter Shute wrote:
> I'm also interested in this. I can get from my ISP daily
> totals for our internet usage. I would like ntopng to be able
> to replicate those daily totals (to give confidence our data
> is correct), and then analyse the totals to see which devices
> contributed. Eg. If we have an above average daily total, I
> want to know why.
> ??
>
> Ntopng can actually produce a traffic report where it shows the
> top local/remote talkers for a specified time frame (e.g. a day),
> but this is a pro only feature.
>
>
> ?I don't object to paying for the license, but this still doesn't get
> me what I want, I think. I want to look back over a historical graph
> (or jump to a time and day, if I believe there was a problem at that
> time) and drill down to see the protocols invloved, and the hosts. Top
> 10 talkers may not include the information I want.?
The most accurate information you can get is via MySQL data (-F option).

I take into account your use use case: you view a local network traffic
graph and see a peak at 5 am. of the last day and want to know which
hosts are involved. You double click on the graph to restrict the time
frame so that a 10 minutes range is selected and 5 am is centered on the
graph.

Now, if hover the mouse on the graph you will see the top talkers at 5
am. From the top talkers panel, you can click the historical icon
(http://fontawesome.io/icon/history/) to access the MySQL data specific
to that host, and drill down its flows and protocols for that particular
time frame.

You can also click on the graph historical icon to get an overview of
all the flows, but you cannot aggregate per host in this way.

What I feel is missing is:
1) an aggregated view of the top protocols on the graph
2) an easy way from the historical explorer to aggregate per host or per
protocol to be able to see and sort bewteen accurate statistics

>> On 16 May 2017, at 4:01 am, Andrew Hilborne <ntop-flugle@snkmail.com
>> <mailto:ntop-flugle@snkmail.com><mailto:ntop-flugle@snkmail.com>
>> <mailto:ntop-flugle@snkmail.com>> wrote:
>> ??
>>
>> On 15 May 2017 at 17:10, Emanuele Farandafaranda-at-ntop.org
>> <http://faranda-at-ntop.org><http://faranda-at-ntop.org>
>> <http://faranda-at-ntop.org> |ntop-flugle| <rrjzg9nx7t@sneakemail.com
>> <mailto:rrjzg9nx7t@sneakemail.com><mailto:rrjzg9nx7t@sneakemail.com>
>> <mailto:rrjzg9nx7t@sneakemail.com>> wrote:
>>
>> You are right, network stats are calculated every minute, whereas interface stats are updated each second.
>>
>> Please note that these stats are dumped to RRD files, not to the MySQL database.
>>
>> ?Would it be possible t change this? Is the issue storage space in the MySQL database? This is what I want to know (initially):
> No, this is something different. MySQL database exports /flows/ as
> data, whereas RRD is /timeseries/ database, so they play different
> roles.
>
>
> ?I do understand the difference between flows and and the RDD
> timeseries. However, typical 5-minute RDD data is useless for
> investigating traffic peaks; I think you may know this, because n2disk
> can now detect 'micro-bursts'?. I don't suggest that storing
> sufficient information to provide a near real-time breakdown of
> traffic is easy, but it would be interesting. If I am reduced to going
> back to using RDDtool type data, there are better tools than ntopng
> for that purpose.
>
> Maybe I'm not using ntopng properly?
Interface traffic statistics are stored with 1 second resolution,
whereas network traffic statistics with 1 minute resolution
(ingress/egress not the protocols, which are dumped each 5 minutes).
It's a trade off between space/time taken for data dump and time
resolution you get. The idea is that raw data is kept in MySQL database,
so this is where you land when you need precise data.

We know there is room for improvements, and we appreciate our users
feedback. So please, if you feel there is a use case interesting which
is not covered/could be better implemented into ntopng, open a feature
request on our github page https://github.com/ntop/ntopng .

Please see also these links:
http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng/
http://www.ntop.org/ntopng/exploring-historical-data-using-ntopng-part-2/

Emanuele

>
> Andrew
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Thanks for the detailed response.

On 18 May 2017 at 11:28, Emanuele Faranda faranda-at-ntop.org |ntop-flugle|
<rrjzg9nx7t@sneakemail.com> wrote:

>
> On 05/17/2017 07:17 PM, Andrew Hilborne wrote:
>> Top 10 talkers may not include the information I want.?
>
> ??
>

The most accurate information you can get is via MySQL data (-F option).
>
> I take into account your use use case: you view a local network traffic
> graph and see a peak at 5 am. of the last day and want to know which hosts
> are involved. You double click on the graph to restrict the time frame so
> that a 10 minutes range is selected and 5 am is centered on the graph.
>
> Now, if hover the mouse on the graph you will see the top talkers at 5 am.
> From the top talkers panel, you can click the historical icon (
> http://fontawesome.io/icon/history/) to access the MySQL data specific to
> that host, and drill down its flows and protocols for that particular time
> frame.
>
> You can also click on the graph historical icon to get an overview of all
> the flows, but you cannot aggregate per host in this way.
>
> What I feel is missing is:
> 1) an aggregated view of the top protocols on the graph
> 2) an easy way from the historical explorer to aggregate per host or per
> protocol to be able to see and sort bewteen accurate statistics
>

?I think this is about right.? Have you seen any Cisco Meraki traffic
graphs? Here's a little (silent) movie which shows some nice rollover
effects and drill-downs: youtu.be/cktxZdR8A3w. Sadly, like nearly everyone
else, Meraki indulge in RDDtool-type averaging, even over a single day, so
they could be a lot better.

?[ S N I P ]

Interface traffic statistics are stored with 1 second resolution, whereas
> network traffic statistics with 1 minute resolution (ingress/egress not the
> protocols, which are dumped each 5 minutes). It's a trade off between
> space/time taken for data dump and time resolution you get. The idea is
> that raw data is kept in MySQL database, so this is where you land when you
> need precise data.
>
> We know there is room for improvements, and we appreciate our users
> feedback. So please, if you feel there is a use case interesting which is
> not covered/could be better implemented into ntopng, open a feature request
>

?I don't really think I can adequately describe what's needed, in terms
which fit into the current program. I would urge you to think about it and
create something yourself.

Andrew
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
Apologies for hijacking Andrew's thread, but maybe what I want could turn out to be helpful for others.

Looking though the available graphs and Andrew's Meraki video, I think where I'm struggling is that all the graphs show traffic in Mbit/s. That's ideal for identifying causes of peak bandwidth usage if you have problems with contention. It doesn't help if you're trying to identify reasons for high total usage, e.g. exceeding a monthly download quota.

I would like to have the option to see the traffic in, say, MB/hour or GB/day. The only way I can see to do that now is to manually enter date ranges and ignore the charts and just look at the totals. I've attached a sample GB/day chart from our ISP's website. I'm imagining being able to hover over the high usage day of 26 April and see the top talkers for that day. Or click on it and show just that day in MB/hour.

Would a feature like this help you, Andrew, or anyone?

Peter Shute

> -----Original Message-----
> From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
> bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
> Sent: Thursday, 18 May 2017 8:29 PM
> To: ntop@listgateway.unipi.it
> Subject: Re: [Ntop] Traffic sent and traffic received in historical view in
> ntopng

> What I feel is missing is:
> 1) an aggregated view of the top protocols on the graph
> 2) an easy way from the historical explorer to aggregate per host or per
> protocol to be able to see and sort bewteen accurate statistics
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
I forgot to mention the reason for needing hourly or daily totals is that things like users streaming radio, etc, can contribute greatly to daily totals, but never show up in a MBits/s graph because they're low level, but all day.

Peter Shute

> -----Original Message-----
> From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
> bounces@listgateway.unipi.it] On Behalf Of Peter Shute
> Sent: Friday, 19 May 2017 9:42 AM
> To: 'ntop@unipi.it' <ntop@unipi.it>
> Subject: Re: [Ntop] Traffic sent and traffic received in historical view in
> ntopng
>
> Apologies for hijacking Andrew's thread, but maybe what I want could turn
> out to be helpful for others.
>
> Looking though the available graphs and Andrew's Meraki video, I think
> where I'm struggling is that all the graphs show traffic in Mbit/s. That's ideal
> for identifying causes of peak bandwidth usage if you have problems with
> contention. It doesn't help if you're trying to identify reasons for high total
> usage, e.g. exceeding a monthly download quota.
>
> I would like to have the option to see the traffic in, say, MB/hour or GB/day.
> The only way I can see to do that now is to manually enter date ranges and
> ignore the charts and just look at the totals. I've attached a sample GB/day
> chart from our ISP's website. I'm imagining being able to hover over the high
> usage day of 26 April and see the top talkers for that day. Or click on it and
> show just that day in MB/hour.
>
> Would a feature like this help you, Andrew, or anyone?
>
> Peter Shute
>
> > -----Original Message-----
> > From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
> > bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
> > Sent: Thursday, 18 May 2017 8:29 PM
> > To: ntop@listgateway.unipi.it
> > Subject: Re: [Ntop] Traffic sent and traffic received in historical
> > view in ntopng
>
> > What I feel is missing is:
> > 1) an aggregated view of the top protocols on the graph
> > 2) an easy way from the historical explorer to aggregate per host or
> > per protocol to be able to see and sort bewteen accurate statistics
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop
Re: Traffic sent and traffic received in historical view in ntopng [ In reply to ]
I'm in favor of this. I myself need to see a 30 day report of all hosts, not
just the top 10. Using the Hosts/Hosts report and sorting by Traffic does
not let you select a time period, only "seen since". If I could select a
time period instead, I'd have everything I need to see a 30 day total of all
hosts.
In my preferences, I have both Idle and Active Local Hosts cache turned on
and "Active Local Host Cache Interval" and "Local Hosts Cache Duration"
both set to 30 days


-----Original Message-----
From: ntop-bounces@listgateway.unipi.it
[mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Peter Shute
Sent: Thursday, May 18, 2017 4:59 PM
To: 'ntop@unipi.it' <ntop@unipi.it>
Subject: Re: [Ntop] Traffic sent and traffic received in historical view in
ntopng

I forgot to mention the reason for needing hourly or daily totals is that
things like users streaming radio, etc, can contribute greatly to daily
totals, but never show up in a MBits/s graph because they're low level, but
all day.

Peter Shute

> -----Original Message-----
> From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
> bounces@listgateway.unipi.it] On Behalf Of Peter Shute
> Sent: Friday, 19 May 2017 9:42 AM
> To: 'ntop@unipi.it' <ntop@unipi.it>
> Subject: Re: [Ntop] Traffic sent and traffic received in historical
> view in ntopng
>
> Apologies for hijacking Andrew's thread, but maybe what I want could
> turn out to be helpful for others.
>
> Looking though the available graphs and Andrew's Meraki video, I think
> where I'm struggling is that all the graphs show traffic in Mbit/s.
> That's ideal for identifying causes of peak bandwidth usage if you
> have problems with contention. It doesn't help if you're trying to
> identify reasons for high total usage, e.g. exceeding a monthly download
quota.
>
> I would like to have the option to see the traffic in, say, MB/hour or
GB/day.
> The only way I can see to do that now is to manually enter date ranges
> and ignore the charts and just look at the totals. I've attached a
> sample GB/day chart from our ISP's website. I'm imagining being able
> to hover over the high usage day of 26 April and see the top talkers
> for that day. Or click on it and show just that day in MB/hour.
>
> Would a feature like this help you, Andrew, or anyone?
>
> Peter Shute
>
> > -----Original Message-----
> > From: ntop-bounces@listgateway.unipi.it [mailto:ntop-
> > bounces@listgateway.unipi.it] On Behalf Of Emanuele Faranda
> > Sent: Thursday, 18 May 2017 8:29 PM
> > To: ntop@listgateway.unipi.it
> > Subject: Re: [Ntop] Traffic sent and traffic received in historical
> > view in ntopng
>
> > What I feel is missing is:
> > 1) an aggregated view of the top protocols on the graph
> > 2) an easy way from the historical explorer to aggregate per host or
> > per protocol to be able to see and sort bewteen accurate statistics
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop


--


_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop