Mailing List Archive

Six second ticks - real time
Hi,

I have a network with about 50000 IP addresses, 2Gbps of traffic. A lot of these ordered into groups, and groups of groups.

I'd like to generate some real time statistics for these, and by real time, I mean every 6 seconds. I don't need flows just RX/TX bytes/packets.

I need these statistics by IP address and aggregated up the group hierarchy. I'd like "something" to do this aggregation. If it can be done at the collection point, that would efficient.

I would like to be able to stream/push some of these statistics every six seconds to applications which are interested. These applications certainly would not want all statistics all of the time. Maybe the aggregations all of the time, and the specific IP addresses some of the time.

So the question is, from a ntop and associated architecture point of view


1. Were and how do I do the aggregation ?
2. Can I selectivity push the aggregations ?

The simple answer is to dump all current statistics to disk every 6 seconds, and batch process the aggregation, and then push the statistics - meaning I do all the work. 50000x4 data points every six seconds to disk does not sound very efficient.

Is there a better way in the ntop architecture ?

Thanks

john
Re: Six second ticks - real time [ In reply to ]
Jonh,
in short the current RRD architecture does NOT allow this, but we have introduced batching and Influx/Prometheus that can do that. However we need to make some changes in the code to enable this.

Would you please open a ticket on github and explain the exact requirements you have, as well include some detailed examples?

Thanks Luca

> On 22 Apr 2018, at 20:10, John Bourke <john.bourke@mobileinternet.com> wrote:
>
> Hi,
>
> I have a network with about 50000 IP addresses, 2Gbps of traffic. A lot of these ordered into groups, and groups of groups.
>
> I’d like to generate some real time statistics for these, and by real time, I mean every 6 seconds. I don’t need flows just RX/TX bytes/packets.
>
> I need these statistics by IP address and aggregated up the group hierarchy. I’d like “something” to do this aggregation. If it can be done at the collection point, that would efficient.
>
> I would like to be able to stream/push some of these statistics every six seconds to applications which are interested. These applications certainly would not want all statistics all of the time. Maybe the aggregations all of the time, and the specific IP addresses some of the time.
>
> So the question is, from a ntop and associated architecture point of view
>
> Were and how do I do the aggregation ?
> Can I selectivity push the aggregations ?
>
> The simple answer is to dump all current statistics to disk every 6 seconds, and batch process the aggregation, and then push the statistics – meaning I do all the work. 50000x4 data points every six seconds to disk does not sound very efficient.
>
> Is there a better way in the ntop architecture ?
>
> Thanks
>
> john
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>
Re: Six second ticks - real time [ In reply to ]
Jonh,
in short the current RRD architecture does NOT allow this, but we have introduced batching and Influx/Prometheus that can do that. However we need to make some changes in the code to enable this.

Would you please open a ticket on github and explain the exact requirements you have, as well include some detailed examples?

Thanks Luca

> On 22 Apr 2018, at 20:10, John Bourke <john.bourke@mobileinternet.com> wrote:
>
> Hi,
>
> I have a network with about 50000 IP addresses, 2Gbps of traffic. A lot of these ordered into groups, and groups of groups.
>
> I’d like to generate some real time statistics for these, and by real time, I mean every 6 seconds. I don’t need flows just RX/TX bytes/packets.
>
> I need these statistics by IP address and aggregated up the group hierarchy. I’d like “something” to do this aggregation. If it can be done at the collection point, that would efficient.
>
> I would like to be able to stream/push some of these statistics every six seconds to applications which are interested. These applications certainly would not want all statistics all of the time. Maybe the aggregations all of the time, and the specific IP addresses some of the time.
>
> So the question is, from a ntop and associated architecture point of view
>
> Were and how do I do the aggregation ?
> Can I selectivity push the aggregations ?
>
> The simple answer is to dump all current statistics to disk every 6 seconds, and batch process the aggregation, and then push the statistics – meaning I do all the work. 50000x4 data points every six seconds to disk does not sound very efficient.
>
> Is there a better way in the ntop architecture ?
>
> Thanks
>
> john
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>