Thank you, Luca. In the help output, it indicates these aggregation fields:
<VLAN Id>/<proto>/<IP>/<port>/<TOS>/<SCTP StreamId>
We are not using VLANs in our network and we are not using SCTP. So, I
assume then that ONLY the following fields will be used for aggregations:
<proto>/<IP>/<port>/<TOS>
You said "Please pay attention to the nprobe startup log" but I do not see
anything here indicating anything about the aggregation bit mappings that
are in effect.
I am concerned that nprobe, in our setup, is going to perform unintended
aggregates so I would have to design around that. Let me explain. We will
be acting as a network provider in a multi-tenant environment. Each tenant
can have overlapping private IP networks. I was thinking about sending
flows from many routers - across many tenants - to a single nprobe
instance, as many as that instance could handle, CPU-wise. I would stand up
N instances then to scale out to handle all tenant flows.
I believe - as I understand the aggregation feature - that the following
could occur:
Assume <proto>/<IP>/<port>/<TOS> is configured and TOS is constant and
protocol=TCP and, by coincidence, the connection 4-tuple just happens to
overlap between two tenant networks.
1. Router A (running in tenant network X with EXPORTER_IPV4_ADDRESS
172.10.10.1) sends a flow with tuples 10.2.3.4:5555 -> 10.6.7.8:443
2. Router B (running in tenant network Y with EXPORTER_IPV4_ADDRESS
172.11.11.1) sends a flow with tuples 10.2.3.4:5555 -> 10.6.7.8:443
Question:
Even though these are coming from two different tenants (two different
routers), am I correct in concluding that these two flow records would be
aggregated in the same aggregation? I believe this is the case because the
EXPORTER_IPV4_ADDRESS is not part of the composite key used for
aggregations. I obviously don't want this to happen so I would have to
design my collection system to avoid this behavior if this is the case
--interface=none
--collector=none
--collector-port=2055
--verbose=1
--lifetime-timeout=120
--idle-timeout=30
--queue-timeout=30
--flow-version=9
--hash-size=256000
--kafka-add-timestamp
--kafka="kafka01:9092;netflow-raw;1"
-T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK
%IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES
%FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION
%EXPORTER_IPV4_ADDRESS"
08/Jan/2018 18:08:42 [plugin.c:187] No plugins found in ./plugins
08/Jan/2018 18:08:42 [plugin.c:195] Loading 23 plugins [.so] from
/usr/local/lib/nprobe/plugins
08/Jan/2018 18:08:42 [nprobe.c:3784] ERROR: Invalid nProbe license
(/etc/nprobe.license) [Missing license file]
08/Jan/2018 18:08:42 [nprobe.c:3791] ERROR:
*****************************************************
08/Jan/2018 18:08:42 [nprobe.c:3792] ERROR: **
**
08/Jan/2018 18:08:42 [nprobe.c:3793] ERROR: ** Switching to DEMO MODE
(missing valid license) **
08/Jan/2018 18:08:42 [nprobe.c:3794] ERROR: **
**
08/Jan/2018 18:08:42 [nprobe.c:3795] ERROR: ** Purchase your nProbe
license at **
08/Jan/2018 18:08:42 [nprobe.c:3796] ERROR: **
https://shop.ntop.org/ **
08/Jan/2018 18:08:42 [nprobe.c:3797] ERROR: **
**
08/Jan/2018 18:08:42 [nprobe.c:3798] ERROR:
*****************************************************
08/Jan/2018 18:08:42 [nprobe.c:4809] WARNING: If you want to preserve the
-M value, please specify -w before -M
08/Jan/2018 18:08:42 [nprobe.c:5755] WARNING: The output interfaceId is set
to 0: did you forget to use -Q perhaps ?
08/Jan/2018 18:08:42 [nprobe.c:5758] WARNING: The input interfaceId is set
to 0: did you forget to use -u perhaps ?
08/Jan/2018 18:08:42 [nprobe.c:5859] Welcome to nProbe v.8.2.171214
($Revision: 5982 $) for x86_64-unknown-linux-gnu with native PF_RING
acceleration
08/Jan/2018 18:08:42 [nprobe.c:5869] Running on CentOS Linux release
7.4.1708 (Core)
08/Jan/2018 18:08:42 [nprobe.c:5880] [LICENSE] nProbe SystemId:
68A2B43E76056A7E
08/Jan/2018 18:08:42 [nprobe.c:5993] Sample rate [packet: 1][flow
collection/export: 1/1]
08/Jan/2018 18:08:42 [nprobe.c:8432] ERROR:
***************************************************************
08/Jan/2018 18:08:42 [nprobe.c:8433] ERROR: * NOTE: This is a DEMO version
limited to 25000 flows export. *
08/Jan/2018 18:08:42 [nprobe.c:8434] ERROR:
***************************************************************
08/Jan/2018 18:08:42 [exportPlugin.c:397] Trying to acquire metadata
information from kafka brokers. This could take several seconds.
08/Jan/2018 18:08:42 [exportPlugin.c:413] Succesfully acquired metadata
information from broker(s)
08/Jan/2018 18:08:42 [exportPlugin.c:425] 1 partions found
08/Jan/2018 18:08:42 [nprobe.c:8440] Welcome to nProbe v.8.2.171214 for
x86_64-unknown-linux-gnu
08/Jan/2018 18:08:42 [nprobe.c:7468] Using NetFlow Packet Payload Len: 1472
08/Jan/2018 18:08:42 [plugin.c:1155] 1 plugin(s) enabled
08/Jan/2018 18:08:42 [nprobe.c:7907] Each flow is 98 bytes long
08/Jan/2018 18:08:42 [nprobe.c:7908] The # flows per packet has been set to
14
08/Jan/2018 18:08:42 [nprobe.c:7911] IP TOS is accounted
08/Jan/2018 18:08:42 [nprobe.c:7937] Non IPv4/v6 traffic is discarded
according to the template
08/Jan/2018 18:08:42 [util.c:440] GeoIP: loaded AS config file
/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
08/Jan/2018 18:08:42 [util.c:451] GeoIP: loaded AS IPv6 config file
/usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
08/Jan/2018 18:08:42 [nprobe.c:8772] Not capturing packet from interface
(collector mode)
08/Jan/2018 18:08:42 [util.c:3591] nProbe changed user to 'nobody'
08/Jan/2018 18:08:42 [plugin.c:900] Enabling plugin Export Plugin
08/Jan/2018 18:08:42 [collect.c:144] Flow collector listening on port 2055
(IPv4/v6)
08/Jan/2018 18:08:42 [nprobe.c:8989] nProbe started successfully
08/Jan/2018 18:08:43 [nprobe.c:3201] ---------------------------------
08/Jan/2018 18:08:43 [nprobe.c:3202] Average traffic: [0.00 pps][All
Traffic 0 b/sec][IP Traffic 0 b/sec][ratio -nan]
08/Jan/2018 18:08:43 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec]
08/Jan/2018 18:08:43 [nprobe.c:3216] Current flow export rate: [0.0
flows/sec]
08/Jan/2018 18:08:43 [nprobe.c:3219] Flow drops: [export queue too
long=0][too many flows=0][ELK queue flow drops=0]
08/Jan/2018 18:08:43 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %]
08/Jan/2018 18:08:43 [nprobe.c:3229] Flow Buckets:
[active=13395][allocated=13395][toBeExported=0]
08/Jan/2018 18:08:43 [nprobe.c:3235] Kafka [flows exported=0/0.0
flows/sec][msgs sent=0/0.0 flows/msg][send errors=0]
08/Jan/2018 18:08:43 [nprobe.c:3260] Collector Threads: [757 pkts@0]
08/Jan/2018 18:08:43 [nprobe.c:3052] Processed packets: 0 (max bucket
search: 7)
08/Jan/2018 18:08:43 [nprobe.c:3035] Fragment queue length: 0
08/Jan/2018 18:08:43 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0
flows/0 pkts sent]
08/Jan/2018 18:08:43 [nprobe.c:3068] Flow collection: [collected pkts:
757][processed flows: 20160]
08/Jan/2018 18:08:43 [nprobe.c:3071] Flow drop stats: [0 bytes/0 pkts][0
flows]
08/Jan/2018 18:08:43 [nprobe.c:3076] Total flow stats: [0 bytes/0 pkts][0
flows/0 pkts sent]
08/Jan/2018 18:08:43 [nprobe.c:3087] Kafka [flows exported=0][msgs
sent=0/0.0 flows/msg][send errors=0]
08/Jan/2018 18:09:13 [nprobe.c:3201] ---------------------------------
08/Jan/2018 18:09:13 [nprobe.c:3202] Average traffic: [0.00 pps][All
Traffic 0 b/sec][IP Traffic 0 b/sec][ratio -nan]
08/Jan/2018 18:09:13 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec]
08/Jan/2018 18:09:13 [nprobe.c:3216] Current flow export rate: [27.4
flows/sec]
08/Jan/2018 18:09:13 [nprobe.c:3219] Flow drops: [export queue too
long=0][too many flows=0][ELK queue flow drops=0]
08/Jan/2018 18:09:13 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %]
08/Jan/2018 18:09:13 [nprobe.c:3229] Flow Buckets:
[active=167763][allocated=167763][toBeExported=0]
08/Jan/2018 18:09:13 [nprobe.c:3235] Kafka [flows exported=822/27.4
flows/sec][msgs sent=822/1.0 flows/msg][send errors=0]
08/Jan/2018 18:09:13 [nprobe.c:3260] Collector Threads: [28566 pkts@0]
08/Jan/2018 18:09:13 [nprobe.c:3052] Processed packets: 0 (max bucket
search: 8)
08/Jan/2018 18:09:13 [nprobe.c:3035] Fragment queue length: 0
08/Jan/2018 18:09:13 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0
flows/0 pkts sent]
08/Jan/2018 18:09:13 [nprobe.c:3068] Flow collection: [collected pkts:
28566][processed flows: 765143]
08/Jan/2018 18:09:13 [nprobe.c:3071] Flow drop stats: [0 bytes/0 pkts][0
flows]
08/Jan/2018 18:09:13 [nprobe.c:3076] Total flow stats: [0 bytes/0 pkts][0
flows/0 pkts sent]
08/Jan/2018 18:09:13 [nprobe.c:3087] Kafka [flows exported=822][msgs
sent=822/1.0 flows/msg][send errors=0]
08/Jan/2018 18:09:43 [nprobe.c:3201] ---------------------------------
08/Jan/2018 18:09:43 [nprobe.c:3202] Average traffic: [0.00 pps][All
Traffic 0 b/sec][IP Traffic 0 b/sec][ratio -nan]
08/Jan/2018 18:09:43 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec]
08/Jan/2018 18:09:43 [nprobe.c:3216] Current flow export rate: [4333.8
flows/sec]
08/Jan/2018 18:09:43 [nprobe.c:3219] Flow drops: [export queue too
long=0][too many flows=0][ELK queue flow drops=0]
08/Jan/2018 18:09:43 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %]
08/Jan/2018 18:09:43 [nprobe.c:3229] Flow Buckets:
[active=96146][allocated=96146][toBeExported=0]
08/Jan/2018 18:09:43 [nprobe.c:3235] Kafka [flows exported=130835/4333.8
flows/sec][msgs sent=130835/1.0 flows/msg][send errors=0]
08/Jan/2018 18:09:43 [nprobe.c:3260] Collector Threads: [50988 pkts@0]
08/Jan/2018 18:09:43 [nprobe.c:3052] Processed packets: 0 (max bucket
search: 8)
08/Jan/2018 18:09:43 [nprobe.c:3035] Fragment queue length: 0
08/Jan/2018 18:09:43 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0
flows/0 pkts sent]
08/Jan/2018 18:09:43 [nprobe.c:3068] Flow collection: [collected pkts:
50988][processed flows: 1376945]
08/Jan/2018 18:09:43 [nprobe.c:3071] Flow drop stats: [0 bytes/0 pkts][0
flows]
08/Jan/2018 18:09:43 [nprobe.c:3076] Total flow stats: [0 bytes/0 pkts][0
flows/0 pkts sent]
08/Jan/2018 18:09:43 [nprobe.c:3087] Kafka [flows exported=130835][msgs
sent=130835/1.0 flows/msg][send errors=0]
On Mon, Jan 8, 2018 at 1:10 PM, Luca Deri <deri@ntop.org> wrote:
> Mark
> the default is 1/1/1/1/1/1 but please note that depending on the template
> some fields will be set to 0. Please pay attention to the nprobe startup log
>
> Thanks Luca
>
> On 8 Jan 2018, at 19:01, Mark Petronic <markpetronic@gmail.com> wrote:
>
> Some indicate the default in the -h output and some do not. Can some
> please tell me the default value for --aggregation in v8.2? Thank you!
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>