Mailing List Archive

NetFlow questions
Hello,

I would like to set up Ntopng as a NetFlow collector to monitor a network of virtual machines. I used VirtualBox to create the network (see enclosed image “Network 2.png”). My goal is to display NetFlow data relative to the flows passing through an Open vSwitch (OVS) installed on a CentOS 7 server. I have installed Ntopng in a CentOS 7 VM in Community mode. The load generators are used to simulate traffic through the OVS with Iperf.

I put the IP address 10.0.0.4/28 in the OVS to enable it to export NetFlow packets to the Ntopng VM:
ip addr add 10.0.0.4/28 dev ovs2
ip link set ovs2 up

According to the documentation of Open vSwitch (http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.pdf), the NetFlow configuration in the OVS is as follows:
ovs?vsctl ?? set Bridge ovs2 netflow=@nf ?? ??id=@nf create NetFlow targets=\"10.0.0.3:2055\" active?timeout=60

To collect NetFlow data, the configuration of nProbe in /etc/nprobe/nprobe.conf is the following:
-i none
-n none
-3 2055
--zmq "tcp://*:5556"
-T %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %L4_SRC_PORT %L4_DST_PORT %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %L4_SRV_PORT %L4_SRV_PORT_MAP %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_SRC_MASK %IPV4_DST_MASK %TOTAL_BYTES_EXP %TOTAL_PKTS_EXP %TOTAL_FLOWS_EXP %IN_SRC_MAC %OUT_DST_MAC %SRC_VLAN %DST_VLAN %INPUT_SNMP %OUTPUT_SNMP %IP_PROTOCOL_VERSION %PROTOCOL %PROTOCOL_MAP %DIRECTION %APPLICATION_ID %APPLICATION_NAME %EXPORTER_IPV4_ADDRESS %FLOW_START_SEC %FLOW_END_SEC %PACKETS_OBSERVED

That of Ntopng in /etc/ntopng/ntopng.conf is:
-i "tcp://127.0.0.1:5556"
-m "10.0.0.0/28"

With these configurations, I get some NetFlow data in Ntopng. The Timeseries graphs work fine and I can see the number of bits/s correctly for the Hosts, Networks and the tcp://127.0.0.1:5556 interface. But I would rather plot data every minute than with a 5 minute interval. Is it possible?

My main problem is that I don’t manage to see data relative to packets, ports and protocols for the hosts:

- I can sometimes see the client ports of the Iperf traffic (see enclosed image “Client ports.png”) for about 2s. But most of the time, information disappears for minutes and only “100% Other” is displayed for Client and Server ports (see enclosed image “Ports.png”), as if NetFlow collected data is rarely understood by Ntopng.

- The data relative to protocols is never processed by Ntopng, the legends “Other”, “Unknown”, “Unrated” or “Unspecified” are displayed (see enclosed image “Protocol overviews.png”). The same problem appears for the section Protocols of the tcp://127.0.0.1:5556 interface.

- I cannot see data in the Packets section at all, the Sent and Received distributions are always empty (see enclosed image “Packets.png”)

Could you explain to me what I should do to solve these issues please?

Moreover, I would like to have the NetFlow data in Grafana by means of its Ntopng plugin.
Referring to Grafana plugin presentation, I would be able to plot:
Interface Metrics:

- Traffic rates, in bits and packets per second

- Traffic totals, both in Bytes and packets

- Application protocol rates, in bits per second
Host Metrics:

- Traffic rate in bits per second

- Traffic total in Bytes

- Application protocol rates in bits per second.

However, I currently manage to visualize only Traffic rate in bits per second for the hosts (host_10.0.0.1_interface_tcp://127.0.0.1:5556_traffic_bps and host_10.0.0.2_interface_tcp://127.0.0.2:5556_traffic_bps).
For host_10.0.0.X_interface_tcp://127.0.0.1:5556_traffic_total_bytes, the graph displays “Data points outside time range”.
For host_10.0.0.X_interface_tcp://127.0.0.1:5556_allprotocols_bps and host_10.0.0.X_interface_tcp://127.0.0.1:5556_allcategories_bps, the graphs display “No data points”.

Besides, no interface metric is available.
Do these issues come from Ntopng or Grafana? And how can they be solved please?

Thank you very much in advance for answering my questions.

Best regards,

Arnaud POLOSSAT
Airbus Defence and Space
1, Bvd Jean Moulin, CS 40001
78 996 Elancourt Cedex, France
E-mail: arnaud.polossat@fr.airbus.com<mailto:arnaud.polossat@fr.airbus.com>


***************************************************************
Ce courriel (incluant ses eventuelles pieces jointes) peut contenir des informations confidentielles et/ou protegees ou dont la diffusion est restreinte. Si vous avez recu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu a quiconque. Merci d'en avertir immediatement l'expediteur et d'effacer ce courriel de votre systeme. Airbus Defence and Space et les sociétés Airbus Group declinent toute responsabilite en cas de corruption par virus, d'alteration ou de falsification de ce courriel lors de sa transmission par voie electronique.
This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus Defence and Space and Airbus Group companies disclaim any and all liability if this email transmission was virus corrupted, altered or falsified.
---------------------------------------------------------------------
Airbus Defence and Space SAS (393 341 516 RCS Toulouse) - Capital: 29.821.072 EUR - Siege social: 31 rue des Cosmonautes, ZI du Palays, 31402 Toulouse cedex 4, France
Re: NetFlow questions [ In reply to ]
Hi,



> On 13 Sep 2018, at 11:31, Polossat, Arnaud [FR] <arnaud.polossat@fr.airbus.com> wrote:
>
> Hello,
>
> I would like to set up Ntopng as a NetFlow collector to monitor a network of virtual machines. I used VirtualBox to create the network (see enclosed image “Network 2.png”). My goal is to display NetFlow data relative to the flows passing through an Open vSwitch (OVS) installed on a CentOS 7 server. I have installed Ntopng in a CentOS 7 VM in Community mode. The load generators are used to simulate traffic through the OVS with Iperf.
>
> I put the IP address 10.0.0.4/28 in the OVS to enable it to export NetFlow packets to the Ntopng VM:
> ip addr add 10.0.0.4/28 dev ovs2
> ip link set ovs2 up
>
> According to the documentation of Open vSwitch (http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.pdf <http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.pdf>), the NetFlow configuration in the OVS is as follows:
> ovs?vsctl ?? set Bridge ovs2 netflow=@nf ?? ??id=@nf create NetFlow targets=\"10.0.0.3:2055\" active?timeout=60
>
> To collect NetFlow data, the configuration of nProbe in /etc/nprobe/nprobe.conf is the following:
> -i none
> -n none
> -3 2055
> --zmq "tcp://*:5556 <tcp://*:5556>"
> -T %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %L4_SRC_PORT %L4_DST_PORT %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %L4_SRV_PORT %L4_SRV_PORT_MAP %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_SRC_MASK %IPV4_DST_MASK %TOTAL_BYTES_EXP %TOTAL_PKTS_EXP %TOTAL_FLOWS_EXP %IN_SRC_MAC %OUT_DST_MAC %SRC_VLAN %DST_VLAN %INPUT_SNMP %OUTPUT_SNMP %IP_PROTOCOL_VERSION %PROTOCOL %PROTOCOL_MAP %DIRECTION %APPLICATION_ID %APPLICATION_NAME %EXPORTER_IPV4_ADDRESS %FLOW_START_SEC %FLOW_END_SEC %PACKETS_OBSERVED
>
> That of Ntopng in /etc/ntopng/ntopng.conf is:
> -i "tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556>"
> -m "10.0.0.0/28"
>
> With these configurations, I get some NetFlow data in Ntopng. The Timeseries graphs work fine and I can see the number of bits/s correctly for the Hosts, Networks and the tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556> interface. But I would rather plot data every minute than with a 5 minute interval. Is it possible?
>
> My main problem is that I don’t manage to see data relative to packets, ports and protocols for the hosts:
> - I can sometimes see the client ports of the Iperf traffic (see enclosed image “Client ports.png”) for about 2s. But most of the time, information disappears for minutes and only “100% Other” is displayed for Client and Server ports (see enclosed image “Ports.png”), as if NetFlow collected data is rarely understood by Ntopng.
> - The data relative to protocols is never processed by Ntopng, the legends “Other”, “Unknown”, “Unrated” or “Unspecified” are displayed (see enclosed image “Protocol overviews.png”). The same problem appears for the section Protocols of the tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556> interface.

When collecting NetFlow, neither nProbe nor ntopng have access to the raw packets, they just get the flows from NetFlow. For this reason they can't run nDPI and thus they have to resort to the naive port-based protocol detection. So my guess is that, in your case, iperf is generating traffic using unknown/ephemeral ports that are not associated to any well-known port. This causes the naive port-based protocol detection to just return 'unknown'.

> - I cannot see data in the Packets section at all, the Sent and Received distributions are always empty (see enclosed image “Packets.png”)

I am not sure the NetFlow contains all the information. Can you enclose a pcap capture of the NetFlow?

>
> Could you explain to me what I should do to solve these issues please?
>
> Moreover, I would like to have the NetFlow data in Grafana by means of its Ntopng plugin.
> Referring to Grafana plugin presentation, I would be able to plot:
> Interface Metrics:
> - Traffic rates, in bits and packets per second
> - Traffic totals, both in Bytes and packets
> - Application protocol rates, in bits per second
> Host Metrics:
> - Traffic rate in bits per second
> - Traffic total in Bytes
> - Application protocol rates in bits per second.
>
> However, I currently manage to visualize only Traffic rate in bits per second for the hosts (host_10.0.0.1_interface_tcp://127.0.0.1:5556_traffic_bps and host_10.0.0.2_interface_tcp://127.0.0.2:5556_traffic_bps).
> For host_10.0.0.X_interface_tcp://127.0.0.1:5556_traffic_total_bytes, the graph displays “Data points outside time range”.
> For host_10.0.0.X_interface_tcp://127.0.0.1:5556_allprotocols_bps and host_10.0.0.X_interface_tcp://127.0.0.1:5556_allcategories_bps, the graphs display “No data points”.
>
> Besides, no interface metric is available.
> Do these issues come from Ntopng or Grafana? And how can they be solved please?

Use the latest 3.7 version. Make sure l7 protocol timeseries generation for local hosts is enabled.


Simone

>
> Thank you very much in advance for answering my questions.
>
> Best regards,
>
> Arnaud POLOSSAT
> Airbus Defence and Space
> 1, Bvd Jean Moulin, CS 40001
> 78 996 Elancourt Cedex, France
> E-mail: arnaud.polossat@fr.airbus.com <mailto:arnaud.polossat@fr.airbus.com>
>
> ***************************************************************
> Ce courriel (incluant ses eventuelles pieces jointes) peut contenir des informations confidentielles et/ou protegees ou dont la diffusion est restreinte. Si vous avez recu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu a quiconque. Merci d'en avertir immediatement l'expediteur et d'effacer ce courriel de votre systeme. Airbus Defence and Space et les sociétés Airbus Group declinent toute responsabilite en cas de corruption par virus, d'alteration ou de falsification de ce courriel lors de sa transmission par voie electronique.
> This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus Defence and Space and Airbus Group companies disclaim any and all liability if this email transmission was virus corrupted, altered or falsified.
> ---------------------------------------------------------------------
> Airbus Defence and Space SAS (393 341 516 RCS Toulouse) - Capital: 29.821.072 EUR - Siege social: 31 rue des Cosmonautes, ZI du Palays, 31402 Toulouse cedex 4, France
> <Network 2.png><Client ports.png><Ports.png><Packets.png><Protocol overviews.png>_______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>
Re: NetFlow questions [ In reply to ]
Hi,



> On 13 Sep 2018, at 11:31, Polossat, Arnaud [FR] <arnaud.polossat@fr.airbus.com> wrote:
>
> Hello,
>
> I would like to set up Ntopng as a NetFlow collector to monitor a network of virtual machines. I used VirtualBox to create the network (see enclosed image “Network 2.png”). My goal is to display NetFlow data relative to the flows passing through an Open vSwitch (OVS) installed on a CentOS 7 server. I have installed Ntopng in a CentOS 7 VM in Community mode. The load generators are used to simulate traffic through the OVS with Iperf.
>
> I put the IP address 10.0.0.4/28 in the OVS to enable it to export NetFlow packets to the Ntopng VM:
> ip addr add 10.0.0.4/28 dev ovs2
> ip link set ovs2 up
>
> According to the documentation of Open vSwitch (http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.pdf <http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.pdf>), the NetFlow configuration in the OVS is as follows:
> ovs?vsctl ?? set Bridge ovs2 netflow=@nf ?? ??id=@nf create NetFlow targets=\"10.0.0.3:2055\" active?timeout=60
>
> To collect NetFlow data, the configuration of nProbe in /etc/nprobe/nprobe.conf is the following:
> -i none
> -n none
> -3 2055
> --zmq "tcp://*:5556 <tcp://*:5556>"
> -T %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %L4_SRC_PORT %L4_DST_PORT %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %L4_SRV_PORT %L4_SRV_PORT_MAP %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_SRC_MASK %IPV4_DST_MASK %TOTAL_BYTES_EXP %TOTAL_PKTS_EXP %TOTAL_FLOWS_EXP %IN_SRC_MAC %OUT_DST_MAC %SRC_VLAN %DST_VLAN %INPUT_SNMP %OUTPUT_SNMP %IP_PROTOCOL_VERSION %PROTOCOL %PROTOCOL_MAP %DIRECTION %APPLICATION_ID %APPLICATION_NAME %EXPORTER_IPV4_ADDRESS %FLOW_START_SEC %FLOW_END_SEC %PACKETS_OBSERVED
>
> That of Ntopng in /etc/ntopng/ntopng.conf is:
> -i "tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556>"
> -m "10.0.0.0/28"
>
> With these configurations, I get some NetFlow data in Ntopng. The Timeseries graphs work fine and I can see the number of bits/s correctly for the Hosts, Networks and the tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556> interface. But I would rather plot data every minute than with a 5 minute interval. Is it possible?
>
> My main problem is that I don’t manage to see data relative to packets, ports and protocols for the hosts:
> - I can sometimes see the client ports of the Iperf traffic (see enclosed image “Client ports.png”) for about 2s. But most of the time, information disappears for minutes and only “100% Other” is displayed for Client and Server ports (see enclosed image “Ports.png”), as if NetFlow collected data is rarely understood by Ntopng.
> - The data relative to protocols is never processed by Ntopng, the legends “Other”, “Unknown”, “Unrated” or “Unspecified” are displayed (see enclosed image “Protocol overviews.png”). The same problem appears for the section Protocols of the tcp://127.0.0.1:5556 <tcp://127.0.0.1:5556> interface.

When collecting NetFlow, neither nProbe nor ntopng have access to the raw packets, they just get the flows from NetFlow. For this reason they can't run nDPI and thus they have to resort to the naive port-based protocol detection. So my guess is that, in your case, iperf is generating traffic using unknown/ephemeral ports that are not associated to any well-known port. This causes the naive port-based protocol detection to just return 'unknown'.

> - I cannot see data in the Packets section at all, the Sent and Received distributions are always empty (see enclosed image “Packets.png”)

I am not sure the NetFlow contains all the information. Can you enclose a pcap capture of the NetFlow?

>
> Could you explain to me what I should do to solve these issues please?
>
> Moreover, I would like to have the NetFlow data in Grafana by means of its Ntopng plugin.
> Referring to Grafana plugin presentation, I would be able to plot:
> Interface Metrics:
> - Traffic rates, in bits and packets per second
> - Traffic totals, both in Bytes and packets
> - Application protocol rates, in bits per second
> Host Metrics:
> - Traffic rate in bits per second
> - Traffic total in Bytes
> - Application protocol rates in bits per second.
>
> However, I currently manage to visualize only Traffic rate in bits per second for the hosts (host_10.0.0.1_interface_tcp://127.0.0.1:5556_traffic_bps and host_10.0.0.2_interface_tcp://127.0.0.2:5556_traffic_bps).
> For host_10.0.0.X_interface_tcp://127.0.0.1:5556_traffic_total_bytes, the graph displays “Data points outside time range”.
> For host_10.0.0.X_interface_tcp://127.0.0.1:5556_allprotocols_bps and host_10.0.0.X_interface_tcp://127.0.0.1:5556_allcategories_bps, the graphs display “No data points”.
>
> Besides, no interface metric is available.
> Do these issues come from Ntopng or Grafana? And how can they be solved please?

Use the latest 3.7 version. Make sure l7 protocol timeseries generation for local hosts is enabled.


Simone

>
> Thank you very much in advance for answering my questions.
>
> Best regards,
>
> Arnaud POLOSSAT
> Airbus Defence and Space
> 1, Bvd Jean Moulin, CS 40001
> 78 996 Elancourt Cedex, France
> E-mail: arnaud.polossat@fr.airbus.com <mailto:arnaud.polossat@fr.airbus.com>
>
> ***************************************************************
> Ce courriel (incluant ses eventuelles pieces jointes) peut contenir des informations confidentielles et/ou protegees ou dont la diffusion est restreinte. Si vous avez recu ce courriel par erreur, vous ne devez ni le copier, ni l'utiliser, ni en divulguer le contenu a quiconque. Merci d'en avertir immediatement l'expediteur et d'effacer ce courriel de votre systeme. Airbus Defence and Space et les sociétés Airbus Group declinent toute responsabilite en cas de corruption par virus, d'alteration ou de falsification de ce courriel lors de sa transmission par voie electronique.
> This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Airbus Defence and Space and Airbus Group companies disclaim any and all liability if this email transmission was virus corrupted, altered or falsified.
> ---------------------------------------------------------------------
> Airbus Defence and Space SAS (393 341 516 RCS Toulouse) - Capital: 29.821.072 EUR - Siege social: 31 rue des Cosmonautes, ZI du Palays, 31402 Toulouse cedex 4, France
> <Network 2.png><Client ports.png><Ports.png><Packets.png><Protocol overviews.png>_______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop <http://listgateway.unipi.it/mailman/listinfo/ntop>