Mailing List Archive: UDP flow collection / nprobe question

Hello ntop team,

I've been trying to troubleshoot a flow collection issue where an old
version of nprobe was collecting a high flow volume, but collecting from
the same stream of UDP flow packets with a newer version produced
inconsistent flow collection rates. I was getting ready to write up a bug
report, but decided to update to the latest stable nprobe version and try
again first. Now, when I run nprobe the log output is full of lines like
these:

12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined
[bucket_id: 28][num: 256]: skipping
12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined
[bucket_id: 29][num: 256]: skipping
12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined
[bucket_id: 30][num: 256]: skipping
12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined
[bucket_id: 25][num: 256]: skipping

I'm not sure I understand what this is telling me. Wild guess: Are there
duplicate IPFIX templates for the same observation domain that nprobe isn't
sure what to do with?

My nprobe command line is like this:

/usr/bin/nprobe --collector-port=2155 --verbose 1 --max-log-lines=100000
--dump-path=/u01/flow/raw/2056 --collector=none --disable-cache
--dump-format=t --dont-nest-dump-dirs --dont-drop-privileges
--smart-udp-frags -V 10

And the version I'm running:

$ nprobe --version

Welcome to nProbe v.9.2.201112 (r6993) for x86_64-pc-linux-gnu
with native PF_RING acceleration.
Copyright 2002-20 ntop.org

Build OS: Ubuntu 20.04.1 LTS
...

Thank you,
Peter

--
Peter Giles | Senior Developer & Data Analyst | Office of the CISO |
University of Washington

Hi Peter
the problem is that your nProbe is receiving templates from multiple routers and they exceed the number of 256. You should see a log like

13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=257][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=1]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=258][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=2]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=259][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=3]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=260][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=4]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=261][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=5]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=262][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=6]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition [id=263][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=7]

to show the templates defined.

We have introduced new checks as in some installations people was sending too many templates and nProbe did not operate properly, So it might be this has broken something on your case.

I would appreciate if you can contact me directly and provide me information for troubleshooting this issue.

Thanks Luca

> On 12 Nov 2020, at 23:06, Peter Giles <gilesp@uw.edu> wrote:
>
> Hello ntop team,
>
> I've been trying to troubleshoot a flow collection issue where an old version of nprobe was collecting a high flow volume, but collecting from the same stream of UDP flow packets with a newer version produced inconsistent flow collection rates. I was getting ready to write up a bug report, but decided to update to the latest stable nprobe version and try again first. Now, when I run nprobe the log output is full of lines like these:
>
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined [bucket_id: 28][num: 256]: skipping
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined [bucket_id: 29][num: 256]: skipping
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined [bucket_id: 30][num: 256]: skipping
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined [bucket_id: 25][num: 256]: skipping
>
> I'm not sure I understand what this is telling me. Wild guess: Are there duplicate IPFIX templates for the same observation domain that nprobe isn't sure what to do with?
>
> My nprobe command line is like this:
>
> /usr/bin/nprobe --collector-port=2155 --verbose 1 --max-log-lines=100000 --dump-path=/u01/flow/raw/2056 --collector=none --disable-cache --dump-format=t --dont-nest-dump-dirs --dont-drop-privileges --smart-udp-frags -V 10
>
> And the version I'm running:
>
> $ nprobe --version
>
> Welcome to nProbe v.9.2.201112 (r6993) for x86_64-pc-linux-gnu
> with native PF_RING acceleration.
> Copyright 2002-20 ntop.org <http://ntop.org/>
>
> Build OS: Ubuntu 20.04.1 LTS
> ...
>
> Thank you,
> Peter
>
> --
> Peter Giles | Senior Developer & Data Analyst | Office of the CISO | University of Washington
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc