Mailing List Archive

Actually, interesting phaenomon:

I reinstalled again (with whiped database) and logged in with admin/admin.

I had flows ...

It seems to be getting flows on startup (about 800 in my case), then
just stops, lets the flows expire, but does not add any new ones anymore.

After about 5 minutes of runtime all flows are gone.

This does not happen, when ntopng is run, where it listens on an
interface itself. Only when it's used with ntopng.

Kind regards,
Martin List-Petersen

On 08/01/17 16:32, Martin List-Petersen wrote:
> After upgrading to 2.5.170108-2130 I have no flows in the flows view ..
> at all.
>
> I have tried to downgrade to 2.5.170106 as I had a copy of that lying
> around on a host, that listens on a different network without the use of
> nprobe and it has flows.
>
> But when used together with nprobe even that version has no flows.
>
> I then downgraded to 2.4 stable and I have flows again, as I previously
> had with the 2.5 releases from November 2016.
>
> This what I installed today:
> ntopng 2.5.170108-2130
> ntopng-data 2.5.170108
> nprobe 7.5.170108-5578
> pfring 6.5.0-1089
> pfring-dkms 6.5.0
>
> nprobe is started like this:
> nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -V 9
>
> ntopng is started like this:
> ntopng /etc/ntopng/ntopng.conf --local-networks xxxx -i
> tcp://127.0.0.1:1234 -x 400000 -X 800000
>
> nprobe gets SNMP from 6 routers with a total stream of about 1.4-1.7
> Gbit/s data flow, so it's not the lack of flows, that's the problem. And
> the configuration nor the startup parameters have been changed from
> before the upgrade, where I had flows.
>
> Kind regards,
> Martin List-Petersen


--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair
http://www.airwire.ie
Phone: 091-865 968
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
Ireland No. 508961
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop

Martin,

On Sun, Jan 8, 2017 at 5:32 PM, Martin List-Petersen <martin@airwire.ie>
wrote:

> After upgrading to 2.5.170108-2130 I have no flows in the flows view .. at
> all.
>
> I have tried to downgrade to 2.5.170106 as I had a copy of that lying
> around on a host, that listens on a different network without the use of
> nprobe and it has flows.
>
> But when used together with nprobe even that version has no flows.
>
> I then downgraded to 2.4 stable and I have flows again, as I previously
> had with the 2.5 releases from November 2016.
>
> This what I installed today:
> ntopng 2.5.170108-2130
> ntopng-data 2.5.170108
> nprobe 7.5.170108-5578
> pfring 6.5.0-1089
> pfring-dkms 6.5.0
>
> nprobe is started like this:
> nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -V 9
>
> ntopng is started like this:
> ntopng /etc/ntopng/ntopng.conf --local-networks xxxx -i tcp://
> 127.0.0.1:1234 -x 400000 -X 800000
>

What are the contents of /etc/ntopng/ntopng.conf? Please don't mix a
configuration file with command line arguments.

Post both the ntopng and the nprobe output. You may also want to run ntopng
and nProbe with verbose/debug flags to see the path of the flows. That is,
if they correctly reach the nProbe and if they are correctly sent to the
ntopng.


>
> nprobe gets SNMP from 6 routers with a total stream of about 1.4-1.7
> Gbit/s data flow, so it's not the lack of flows, that's the problem. And
> the configuration nor the startup parameters have been changed from before
> the upgrade, where I had flows.
>
> Kind regards,
> Martin List-Petersen
> --
> Airwire Ltd. - Ag Nascadh Pobail an Iarthair
> http://www.airwire.ie
> Phone: 091-865 968
> Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
> Ireland No. 508961
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>

Hi,

ntopng.conf only contains:

-G=/var/run/ntopng.pid

nothing more.

nprobe output:
09/Jan/2017 10:01:03 [nprobe.c:3492] Valid nProbe license found
09/Jan/2017 10:01:03 [nprobe.c:5201] WARNING: The output interfaceId is
set to 0: did you forget to use -Q perhaps ?
09/Jan/2017 10:01:03 [nprobe.c:5204] WARNING: The input interfaceId is
set to 0: did you forget to use -u perhaps ?
09/Jan/2017 10:01:03 [nprobe.c:5304] Welcome to nProbe v.7.5.170108
($Revision: 5578 $) for x86_64-unknown-linux-gnu with native PF_RING
acceleration
09/Jan/2017 10:01:03 [nprobe.c:5314] Running on Debian GNU/Linux 8.2
(jessie)
09/Jan/2017 10:01:03 [nprobe.c:5325] [LICENSE] nProbe SystemId:
1B71ED8609B0B927
09/Jan/2017 10:01:03 [nprobe.c:7680] Welcome to nProbe v.7.5.170108 for
x86_64-unknown-linux-gnu
09/Jan/2017 10:01:03 [nprobe.c:6757] WARNING: You selected v9/IPFIX
without specifying a template (-T).
09/Jan/2017 10:01:03 [nprobe.c:6758] WARNING: The default template will
be used
09/Jan/2017 10:01:03 [nprobe.c:6763] Using NetFlow Packet Payload Len: 1472
09/Jan/2017 10:01:03 [plugin.c:1078] 0 plugin(s) enabled
09/Jan/2017 10:01:03 [nprobe.c:7176] Each flow is 89 bytes long
09/Jan/2017 10:01:03 [nprobe.c:7177] The # packets per flow has been set
to 15
09/Jan/2017 10:01:03 [nprobe.c:7180] IP TOS is accounted
09/Jan/2017 10:01:03 [nprobe.c:7206] Non IPv4/v6 traffic is discarded
according to the template
09/Jan/2017 10:01:03 [util.c:430] GeoIP: loaded AS config file
/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
09/Jan/2017 10:01:03 [util.c:441] GeoIP: loaded AS IPv6 config file
/usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
09/Jan/2017 10:01:03 [nprobe.c:8005] Not capturing packet from interface
(collector mode)
09/Jan/2017 10:01:03 [util.c:4043] Initializing ZMQ as server
09/Jan/2017 10:01:03 [util.c:4086] Succesfully created ZMQ endpoint
tcp://127.0.0.1:1234
09/Jan/2017 10:01:03 [collect.c:143] Flow collector listening on port
2055 (IPv4/v6)
09/Jan/2017 10:01:03 [nprobe.c:8230] nProbe started successfully

ntopng output:
09/Jan/2017 10:04:22 [Redis.cpp:108] Successfully connected to redis
127.0.0.1:6379@0
09/Jan/2017 10:04:22 [NtopPro.cpp:118] [LICENSE] Read license from Redis
[XXXXX]
09/Jan/2017 10:04:22 [Ntop.cpp:1236] Registered interface
tcp://127.0.0.1:1234 [id: 0]
09/Jan/2017 10:04:22 [main.cpp:248] PID stored in file /var/run/ntopng.pid
09/Jan/2017 10:04:22 [HTTPserver.cpp:507] Please read
https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to
enable SSL.
09/Jan/2017 10:04:22 [Utils.cpp:367] User changed to nobody
09/Jan/2017 10:04:22 [HTTPserver.cpp:552] Web server dirs
[/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
09/Jan/2017 10:04:22 [HTTPserver.cpp:555] HTTP server listening on port 3000
09/Jan/2017 10:04:22 [main.cpp:309] Working directory: /var/tmp/ntopng
09/Jan/2017 10:04:22 [main.cpp:311] Scripts/HTML pages directory:
/usr/share/ntopng
09/Jan/2017 10:04:22 [Ntop.cpp:268] Welcome to ntopng x86_64
v.2.5.170108 - (C) 1998-2016 ntop.org
09/Jan/2017 10:04:22 [Ntop.cpp:273] Built on Debian GNU/Linux 8.2 (jessie)
09/Jan/2017 10:04:22 [PeriodicActivities.cpp:55] Started periodic
activities loop...
09/Jan/2017 10:04:22 [NtopPro.cpp:262] [LICENSE] ntopng systemId:
1B71ED8609B0B927
09/Jan/2017 10:04:22 [NtopPro.cpp:273] [LICENSE] ntopng license:
F94DBEB4F844679D6B490B2830E3072715076388282F622A26
09/Jan/2017 10:04:22 [NtopPro.cpp:294] [LICENSE] Maintenance is
available until Tue Oct 10 13:33:48 2017 [274 days left]
09/Jan/2017 10:04:22 [Ntop.cpp:559] Local Interface Addresses (System Host)
09/Jan/2017 10:04:22 [Ntop.cpp:561] Local Networks
09/Jan/2017 10:04:22 [AddressTree.cpp:134] [AddressTree] XXXXXXX
09/Jan/2017 10:04:22 [NetworkInterface.cpp:1797] Started packet polling
on interface tcp://127.0.0.1:1234 [id: 0]...
09/Jan/2017 10:04:23 [CollectorInterface.cpp:115] Collecting flows on
tcp://127.0.0.1:1234

And as I said, my configuration has not changed. I've upgraded from the
November build to the January build. That's the only difference.

This was working perfectly up until then.

Kind regards,
Martin List-Petersen
Airwire Ltd.


On 09/01/17 09:10, Simone Mainardi wrote:
> Martin,
>
> On Sun, Jan 8, 2017 at 5:32 PM, Martin List-Petersen <martin@airwire.ie>
> wrote:
>
>> After upgrading to 2.5.170108-2130 I have no flows in the flows view .. at
>> all.
>>
>> I have tried to downgrade to 2.5.170106 as I had a copy of that lying
>> around on a host, that listens on a different network without the use of
>> nprobe and it has flows.
>>
>> But when used together with nprobe even that version has no flows.
>>
>> I then downgraded to 2.4 stable and I have flows again, as I previously
>> had with the 2.5 releases from November 2016.
>>
>> This what I installed today:
>> ntopng 2.5.170108-2130
>> ntopng-data 2.5.170108
>> nprobe 7.5.170108-5578
>> pfring 6.5.0-1089
>> pfring-dkms 6.5.0
>>
>> nprobe is started like this:
>> nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -V 9
>>
>> ntopng is started like this:
>> ntopng /etc/ntopng/ntopng.conf --local-networks xxxx -i tcp://
>> 127.0.0.1:1234 -x 400000 -X 800000
>>
>
> What are the contents of /etc/ntopng/ntopng.conf? Please don't mix a
> configuration file with command line arguments.
>
> Post both the ntopng and the nprobe output. You may also want to run ntopng
> and nProbe with verbose/debug flags to see the path of the flows. That is,
> if they correctly reach the nProbe and if they are correctly sent to the
> ntopng.
>
>
>>
>> nprobe gets SNMP from 6 routers with a total stream of about 1.4-1.7
>> Gbit/s data flow, so it's not the lack of flows, that's the problem. And
>> the configuration nor the startup parameters have been changed from before
>> the upgrade, where I had flows.
>>
>> Kind regards,
>> Martin List-Petersen
>> --
>> Airwire Ltd. - Ag Nascadh Pobail an Iarthair
>> http://www.airwire.ie
>> Phone: 091-865 968
>> Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
>> Ireland No. 508961
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>


_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop

Hi,

upgraded to 2.5.170111 after a ntopng 2.5.170108 failing with
segmentation faults a couple of times.

The issue with flows only being shown after a restart, then disappearing
remains.

When I downgrade to 2.4, flows work perfectly.

The same configuration in both cases.

Kind regards,
Martin List-Petersen
Airwire Ltd.


On 09/01/17 10:07, Martin List-Petersen wrote:
> Hi,
>
> ntopng.conf only contains:
>
> -G=/var/run/ntopng.pid
>
> nothing more.
>
> nprobe output:
> 09/Jan/2017 10:01:03 [nprobe.c:3492] Valid nProbe license found
> 09/Jan/2017 10:01:03 [nprobe.c:5201] WARNING: The output interfaceId is
> set to 0: did you forget to use -Q perhaps ?
> 09/Jan/2017 10:01:03 [nprobe.c:5204] WARNING: The input interfaceId is
> set to 0: did you forget to use -u perhaps ?
> 09/Jan/2017 10:01:03 [nprobe.c:5304] Welcome to nProbe v.7.5.170108
> ($Revision: 5578 $) for x86_64-unknown-linux-gnu with native PF_RING
> acceleration
> 09/Jan/2017 10:01:03 [nprobe.c:5314] Running on Debian GNU/Linux 8.2
> (jessie)
> 09/Jan/2017 10:01:03 [nprobe.c:5325] [LICENSE] nProbe SystemId:
> 1B71ED8609B0B927
> 09/Jan/2017 10:01:03 [nprobe.c:7680] Welcome to nProbe v.7.5.170108 for
> x86_64-unknown-linux-gnu
> 09/Jan/2017 10:01:03 [nprobe.c:6757] WARNING: You selected v9/IPFIX
> without specifying a template (-T).
> 09/Jan/2017 10:01:03 [nprobe.c:6758] WARNING: The default template will
> be used
> 09/Jan/2017 10:01:03 [nprobe.c:6763] Using NetFlow Packet Payload Len: 1472
> 09/Jan/2017 10:01:03 [plugin.c:1078] 0 plugin(s) enabled
> 09/Jan/2017 10:01:03 [nprobe.c:7176] Each flow is 89 bytes long
> 09/Jan/2017 10:01:03 [nprobe.c:7177] The # packets per flow has been set
> to 15
> 09/Jan/2017 10:01:03 [nprobe.c:7180] IP TOS is accounted
> 09/Jan/2017 10:01:03 [nprobe.c:7206] Non IPv4/v6 traffic is discarded
> according to the template
> 09/Jan/2017 10:01:03 [util.c:430] GeoIP: loaded AS config file
> /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
> 09/Jan/2017 10:01:03 [util.c:441] GeoIP: loaded AS IPv6 config file
> /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
> 09/Jan/2017 10:01:03 [nprobe.c:8005] Not capturing packet from interface
> (collector mode)
> 09/Jan/2017 10:01:03 [util.c:4043] Initializing ZMQ as server
> 09/Jan/2017 10:01:03 [util.c:4086] Succesfully created ZMQ endpoint
> tcp://127.0.0.1:1234
> 09/Jan/2017 10:01:03 [collect.c:143] Flow collector listening on port
> 2055 (IPv4/v6)
> 09/Jan/2017 10:01:03 [nprobe.c:8230] nProbe started successfully
>
> ntopng output:
> 09/Jan/2017 10:04:22 [Redis.cpp:108] Successfully connected to redis
> 127.0.0.1:6379@0
> 09/Jan/2017 10:04:22 [NtopPro.cpp:118] [LICENSE] Read license from Redis
> [XXXXX]
> 09/Jan/2017 10:04:22 [Ntop.cpp:1236] Registered interface
> tcp://127.0.0.1:1234 [id: 0]
> 09/Jan/2017 10:04:22 [main.cpp:248] PID stored in file /var/run/ntopng.pid
> 09/Jan/2017 10:04:22 [HTTPserver.cpp:507] Please read
> https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to
> enable SSL.
> 09/Jan/2017 10:04:22 [Utils.cpp:367] User changed to nobody
> 09/Jan/2017 10:04:22 [HTTPserver.cpp:552] Web server dirs
> [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
> 09/Jan/2017 10:04:22 [HTTPserver.cpp:555] HTTP server listening on port
> 3000
> 09/Jan/2017 10:04:22 [main.cpp:309] Working directory: /var/tmp/ntopng
> 09/Jan/2017 10:04:22 [main.cpp:311] Scripts/HTML pages directory:
> /usr/share/ntopng
> 09/Jan/2017 10:04:22 [Ntop.cpp:268] Welcome to ntopng x86_64
> v.2.5.170108 - (C) 1998-2016 ntop.org
> 09/Jan/2017 10:04:22 [Ntop.cpp:273] Built on Debian GNU/Linux 8.2 (jessie)
> 09/Jan/2017 10:04:22 [PeriodicActivities.cpp:55] Started periodic
> activities loop...
> 09/Jan/2017 10:04:22 [NtopPro.cpp:262] [LICENSE] ntopng systemId:
> 1B71ED8609B0B927
> 09/Jan/2017 10:04:22 [NtopPro.cpp:273] [LICENSE] ntopng license:
> F94DBEB4F844679D6B490B2830E3072715076388282F622A26
> 09/Jan/2017 10:04:22 [NtopPro.cpp:294] [LICENSE] Maintenance is
> available until Tue Oct 10 13:33:48 2017 [274 days left]
> 09/Jan/2017 10:04:22 [Ntop.cpp:559] Local Interface Addresses (System Host)
> 09/Jan/2017 10:04:22 [Ntop.cpp:561] Local Networks
> 09/Jan/2017 10:04:22 [AddressTree.cpp:134] [AddressTree] XXXXXXX
> 09/Jan/2017 10:04:22 [NetworkInterface.cpp:1797] Started packet polling
> on interface tcp://127.0.0.1:1234 [id: 0]...
> 09/Jan/2017 10:04:23 [CollectorInterface.cpp:115] Collecting flows on
> tcp://127.0.0.1:1234
>
> And as I said, my configuration has not changed. I've upgraded from the
> November build to the January build. That's the only difference.
>
> This was working perfectly up until then.
>
> Kind regards,
> Martin List-Petersen
> Airwire Ltd.
>
>
> On 09/01/17 09:10, Simone Mainardi wrote:
>> Martin,
>>
>> On Sun, Jan 8, 2017 at 5:32 PM, Martin List-Petersen <martin@airwire.ie>
>> wrote:
>>
>>> After upgrading to 2.5.170108-2130 I have no flows in the flows view
>>> .. at
>>> all.
>>>
>>> I have tried to downgrade to 2.5.170106 as I had a copy of that lying
>>> around on a host, that listens on a different network without the use of
>>> nprobe and it has flows.
>>>
>>> But when used together with nprobe even that version has no flows.
>>>
>>> I then downgraded to 2.4 stable and I have flows again, as I previously
>>> had with the 2.5 releases from November 2016.
>>>
>>> This what I installed today:
>>> ntopng 2.5.170108-2130
>>> ntopng-data 2.5.170108
>>> nprobe 7.5.170108-5578
>>> pfring 6.5.0-1089
>>> pfring-dkms 6.5.0
>>>
>>> nprobe is started like this:
>>> nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -V 9
>>>
>>> ntopng is started like this:
>>> ntopng /etc/ntopng/ntopng.conf --local-networks xxxx -i tcp://
>>> 127.0.0.1:1234 -x 400000 -X 800000
>>>
>>
>> What are the contents of /etc/ntopng/ntopng.conf? Please don't mix a
>> configuration file with command line arguments.
>>
>> Post both the ntopng and the nprobe output. You may also want to run
>> ntopng
>> and nProbe with verbose/debug flags to see the path of the flows. That
>> is,
>> if they correctly reach the nProbe and if they are correctly sent to the
>> ntopng.
>>
>>
>>>
>>> nprobe gets SNMP from 6 routers with a total stream of about 1.4-1.7
>>> Gbit/s data flow, so it's not the lack of flows, that's the problem. And
>>> the configuration nor the startup parameters have been changed from
>>> before
>>> the upgrade, where I had flows.
>>>
>>> Kind regards,
>>> Martin List-Petersen
>>> --
>>> Airwire Ltd. - Ag Nascadh Pobail an Iarthair
>>> http://www.airwire.ie
>>> Phone: 091-865 968
>>> Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
>>> Ireland No. 508961
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>
>>
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop


--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair
http://www.airwire.ie
Phone: 091-865 968
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
Ireland No. 508961
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop

Ok,

after further testing, the problem seems to be with nprobe.

Basically, I'm collecting netflow data from various routers (v5 and v9)
and send these to the collector, which is a licensed nprobe.

The netflow data could be collected from a Cisco router, it could come
from a Linux box running pmacctd or it could come from a Linux box
running nprobe.

nprobe.conf example for one of those boxes that collect netflows and
send them on as for example netflow v9:
-i=bond0
-g=/var/run/nprobe-zmq.pid
-n=XXX.XXX.XXX.XXX:2055
-V=9
-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP
%OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED
%L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS
%IPV4_SRC_MASK %IPV4_DST_MASK %IPV6_SRC_ADDR %IPV6_DST_ADDR
%IPV6_NEXT_HOP %IPV6_SRC_MASK %IPV6_DST_MASK %IP_PROTOCOL_VERSION
%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS %FLOW_ID %FLOW_START_SEC
%FLOW_END_SEC %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%BIFLOW_DIRECTION"

My collector nprobe is configured like this:
-i none
-n none
-3 2055
--zmq tcp://127.0.0.1:1234
-V 9
-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP
%OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED
%L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS
%IPV4_SRC_MASK %IPV4_DST_MASK %IPV6_SRC_ADDR %IPV6_DST_ADDR
%IPV6_NEXT_HOP %IPV6_SRC_MASK %IPV6_DST_MASK %IP_PROTOCOL_VERSION
%EXPORTER_IPV4_ADDRESS %EXPORTER_IPV6_ADDRESS %FLOW_ID %FLOW_START_SEC
%FLOW_END_SEC %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS
%BIFLOW_DIRECTION"

And this then hands the data off to ntopng, which is configured like this:
-G=/var/run/ntopng.pid
-m=XXXX/X
-i=tcp://127.0.0.1:1234
-x=400000
-X=800000

Flows are present on startup, after the redis-database is cleared but
then stop coming in and flows view empties after a short while and stays
empty.

When I downgrade to v2.4 this setup works perfectly and I have flows.
With the builds of v2.5 from November, this also worked perfectly.

I spend the evening to strip it all down and where it fails is at the
point where the individual netflows are collected by the collector on
port 2055.

If i for example take the host collecting netflows on interface bond0
above and send the netflows using zmq directly to ntopng, then I have
flows .. and it works perfectly.

When I collect flows via the Netflows/sFlow/IPFix port, it's broken.

I hope this helps to pinpoint the problem.

Tested tonight with:
nprobe 7.5.170112-5587
pfring 6.5.0-1094
pfring-dkms 6.5.0
ntopng 2.5.170112-2154
ntopng-data 2.5.170112

Kind regards,
Martin List-Petersen


On 11/01/17 22:36, Martin List-Petersen wrote:
> Hi,
>
> upgraded to 2.5.170111 after a ntopng 2.5.170108 failing with
> segmentation faults a couple of times.
>
> The issue with flows only being shown after a restart, then disappearing
> remains.
>
> When I downgrade to 2.4, flows work perfectly.
>
> The same configuration in both cases.
>
> Kind regards,
> Martin List-Petersen
> Airwire Ltd.
>
>
> On 09/01/17 10:07, Martin List-Petersen wrote:
>> Hi,
>>
>> ntopng.conf only contains:
>>
>> -G=/var/run/ntopng.pid
>>
>> nothing more.
>>
>> nprobe output:
>> 09/Jan/2017 10:01:03 [nprobe.c:3492] Valid nProbe license found
>> 09/Jan/2017 10:01:03 [nprobe.c:5201] WARNING: The output interfaceId is
>> set to 0: did you forget to use -Q perhaps ?
>> 09/Jan/2017 10:01:03 [nprobe.c:5204] WARNING: The input interfaceId is
>> set to 0: did you forget to use -u perhaps ?
>> 09/Jan/2017 10:01:03 [nprobe.c:5304] Welcome to nProbe v.7.5.170108
>> ($Revision: 5578 $) for x86_64-unknown-linux-gnu with native PF_RING
>> acceleration
>> 09/Jan/2017 10:01:03 [nprobe.c:5314] Running on Debian GNU/Linux 8.2
>> (jessie)
>> 09/Jan/2017 10:01:03 [nprobe.c:5325] [LICENSE] nProbe SystemId:
>> 1B71ED8609B0B927
>> 09/Jan/2017 10:01:03 [nprobe.c:7680] Welcome to nProbe v.7.5.170108 for
>> x86_64-unknown-linux-gnu
>> 09/Jan/2017 10:01:03 [nprobe.c:6757] WARNING: You selected v9/IPFIX
>> without specifying a template (-T).
>> 09/Jan/2017 10:01:03 [nprobe.c:6758] WARNING: The default template will
>> be used
>> 09/Jan/2017 10:01:03 [nprobe.c:6763] Using NetFlow Packet Payload Len:
>> 1472
>> 09/Jan/2017 10:01:03 [plugin.c:1078] 0 plugin(s) enabled
>> 09/Jan/2017 10:01:03 [nprobe.c:7176] Each flow is 89 bytes long
>> 09/Jan/2017 10:01:03 [nprobe.c:7177] The # packets per flow has been set
>> to 15
>> 09/Jan/2017 10:01:03 [nprobe.c:7180] IP TOS is accounted
>> 09/Jan/2017 10:01:03 [nprobe.c:7206] Non IPv4/v6 traffic is discarded
>> according to the template
>> 09/Jan/2017 10:01:03 [util.c:430] GeoIP: loaded AS config file
>> /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
>> 09/Jan/2017 10:01:03 [util.c:441] GeoIP: loaded AS IPv6 config file
>> /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
>> 09/Jan/2017 10:01:03 [nprobe.c:8005] Not capturing packet from interface
>> (collector mode)
>> 09/Jan/2017 10:01:03 [util.c:4043] Initializing ZMQ as server
>> 09/Jan/2017 10:01:03 [util.c:4086] Succesfully created ZMQ endpoint
>> tcp://127.0.0.1:1234
>> 09/Jan/2017 10:01:03 [collect.c:143] Flow collector listening on port
>> 2055 (IPv4/v6)
>> 09/Jan/2017 10:01:03 [nprobe.c:8230] nProbe started successfully
>>
>> ntopng output:
>> 09/Jan/2017 10:04:22 [Redis.cpp:108] Successfully connected to redis
>> 127.0.0.1:6379@0
>> 09/Jan/2017 10:04:22 [NtopPro.cpp:118] [LICENSE] Read license from Redis
>> [XXXXX]
>> 09/Jan/2017 10:04:22 [Ntop.cpp:1236] Registered interface
>> tcp://127.0.0.1:1234 [id: 0]
>> 09/Jan/2017 10:04:22 [main.cpp:248] PID stored in file
>> /var/run/ntopng.pid
>> 09/Jan/2017 10:04:22 [HTTPserver.cpp:507] Please read
>> https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to
>> enable SSL.
>> 09/Jan/2017 10:04:22 [Utils.cpp:367] User changed to nobody
>> 09/Jan/2017 10:04:22 [HTTPserver.cpp:552] Web server dirs
>> [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
>> 09/Jan/2017 10:04:22 [HTTPserver.cpp:555] HTTP server listening on port
>> 3000
>> 09/Jan/2017 10:04:22 [main.cpp:309] Working directory: /var/tmp/ntopng
>> 09/Jan/2017 10:04:22 [main.cpp:311] Scripts/HTML pages directory:
>> /usr/share/ntopng
>> 09/Jan/2017 10:04:22 [Ntop.cpp:268] Welcome to ntopng x86_64
>> v.2.5.170108 - (C) 1998-2016 ntop.org
>> 09/Jan/2017 10:04:22 [Ntop.cpp:273] Built on Debian GNU/Linux 8.2
>> (jessie)
>> 09/Jan/2017 10:04:22 [PeriodicActivities.cpp:55] Started periodic
>> activities loop...
>> 09/Jan/2017 10:04:22 [NtopPro.cpp:262] [LICENSE] ntopng systemId:
>> 1B71ED8609B0B927
>> 09/Jan/2017 10:04:22 [NtopPro.cpp:273] [LICENSE] ntopng license:
>> F94DBEB4F844679D6B490B2830E3072715076388282F622A26
>> 09/Jan/2017 10:04:22 [NtopPro.cpp:294] [LICENSE] Maintenance is
>> available until Tue Oct 10 13:33:48 2017 [274 days left]
>> 09/Jan/2017 10:04:22 [Ntop.cpp:559] Local Interface Addresses (System
>> Host)
>> 09/Jan/2017 10:04:22 [Ntop.cpp:561] Local Networks
>> 09/Jan/2017 10:04:22 [AddressTree.cpp:134] [AddressTree] XXXXXXX
>> 09/Jan/2017 10:04:22 [NetworkInterface.cpp:1797] Started packet polling
>> on interface tcp://127.0.0.1:1234 [id: 0]...
>> 09/Jan/2017 10:04:23 [CollectorInterface.cpp:115] Collecting flows on
>> tcp://127.0.0.1:1234
>>
>> And as I said, my configuration has not changed. I've upgraded from the
>> November build to the January build. That's the only difference.
>>
>> This was working perfectly up until then.
>>
>> Kind regards,
>> Martin List-Petersen
>> Airwire Ltd.
>>
>>
>> On 09/01/17 09:10, Simone Mainardi wrote:
>>> Martin,
>>>
>>> On Sun, Jan 8, 2017 at 5:32 PM, Martin List-Petersen <martin@airwire.ie>
>>> wrote:
>>>
>>>> After upgrading to 2.5.170108-2130 I have no flows in the flows view
>>>> .. at
>>>> all.
>>>>
>>>> I have tried to downgrade to 2.5.170106 as I had a copy of that lying
>>>> around on a host, that listens on a different network without the
>>>> use of
>>>> nprobe and it has flows.
>>>>
>>>> But when used together with nprobe even that version has no flows.
>>>>
>>>> I then downgraded to 2.4 stable and I have flows again, as I previously
>>>> had with the 2.5 releases from November 2016.
>>>>
>>>> This what I installed today:
>>>> ntopng 2.5.170108-2130
>>>> ntopng-data 2.5.170108
>>>> nprobe 7.5.170108-5578
>>>> pfring 6.5.0-1089
>>>> pfring-dkms 6.5.0
>>>>
>>>> nprobe is started like this:
>>>> nprobe -i none -n none -3 2055 --zmq tcp://127.0.0.1:1234 -V 9
>>>>
>>>> ntopng is started like this:
>>>> ntopng /etc/ntopng/ntopng.conf --local-networks xxxx -i tcp://
>>>> 127.0.0.1:1234 -x 400000 -X 800000
>>>>
>>>
>>> What are the contents of /etc/ntopng/ntopng.conf? Please don't mix a
>>> configuration file with command line arguments.
>>>
>>> Post both the ntopng and the nprobe output. You may also want to run
>>> ntopng
>>> and nProbe with verbose/debug flags to see the path of the flows. That
>>> is,
>>> if they correctly reach the nProbe and if they are correctly sent to the
>>> ntopng.
>>>
>>>
>>>>
>>>> nprobe gets SNMP from 6 routers with a total stream of about 1.4-1.7
>>>> Gbit/s data flow, so it's not the lack of flows, that's the problem.
>>>> And
>>>> the configuration nor the startup parameters have been changed from
>>>> before
>>>> the upgrade, where I had flows.
>>>>
>>>> Kind regards,
>>>> Martin List-Petersen
>>>> --
>>>> Airwire Ltd. - Ag Nascadh Pobail an Iarthair
>>>> http://www.airwire.ie
>>>> Phone: 091-865 968
>>>> Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 -
>>>> Registered in
>>>> Ireland No. 508961
>>>> _______________________________________________
>>>> Ntop mailing list
>>>> Ntop@listgateway.unipi.it
>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>


--
Airwire Ltd. - Ag Nascadh Pobail an Iarthair
http://www.airwire.ie
Phone: 091-865 968
Registered Office: Moy, Kinvara, Co. Galway, 091-865 968 - Registered in
Ireland No. 508961
_______________________________________________
Ntop mailing list
Ntop@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop