Mailing List Archive: Kentik nprobes sampling reporting excessive packets

Kentik nprobes sampling reporting excessive packets

Mar 6, 2017, 9:54 AM

Post #1 of 11 (773 views)

Hello,

We are using the version of nprobe to work with Kentik (nprobes) and we are seeing packets being reported incorrectly when using sampling.

$sudo nprobe -v

Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu

Copyright 2002-17 ntop.org

Build OS: CentOS release 6.8 (Final)

Example script being used:

-q=A.A.A.A:6650

-n=X.X.X.X:9995

-n=Y.Y.Y.Y:2056

-V=5

-a=

-i=myri1-1

-S=160:1

-t=60

--if-networks=@/etc/cento/networks

-b=1

-Q=1

-u=1

--dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt

With the above, when I perform a controlled bandwidth test using a traffic generator, the packets per second being reported to Kentik is roughly 350K. The traffic generator is sending 2140pps of udp packets with 1460 byte payloads at a payload rate of 25 Mbps, so 25.9 Mbps with IP header.

We also sent the same data to another collector at the same time, and we saw exact same issue in the nfacctd data.

We saw that when we changed the sample rate, the pps was multiplied by that value, so in this case 160x (2140x160=342,400).

[cid:image002.jpg@01D29670.6A8C9F70]

[cid:image003.jpg@01D29670.6A8C9F70]

This is also evident in the attached png image of wireshark output from a captured packet.

[cid:image001.png@01D2966F.6430ED30]

Can you please advise if this is a bug, and if so, the best course of action (use a previous version?), or if I am doing something wrong.

I also attached the images in case they are stripped out.

Kind regards,

Jesse

Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 8, 2017, 4:21 AM

Post #2 of 11 (768 views)

Permalink

Anything? ...

From: ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Jesse Alexander
Sent: Monday, March 06, 2017 11:55 AM
To: ntop@listgateway.unipi.it
Subject: [Ntop] Kentik nprobes sampling reporting excessive packets

Hello,

We are using the version of nprobe to work with Kentik (nprobes) and we are seeing packets being reported incorrectly when using sampling.

$sudo nprobe -v

Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu

Copyright 2002-17 ntop.org

Build OS: CentOS release 6.8 (Final)

Example script being used:

-q=A.A.A.A:6650

-n=X.X.X.X:9995

-n=Y.Y.Y.Y:2056

-V=5

-a=

-i=myri1-1

-S=160:1

-t=60

--if-networks=@/etc/cento/networks

-b=1

-Q=1

-u=1

--dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt

With the above, when I perform a controlled bandwidth test using a traffic generator, the packets per second being reported to Kentik is roughly 350K. The traffic generator is sending 2140pps of udp packets with 1460 byte payloads at a payload rate of 25 Mbps, so 25.9 Mbps with IP header.

We also sent the same data to another collector at the same time, and we saw exact same issue in the nfacctd data.

We saw that when we changed the sample rate, the pps was multiplied by that value, so in this case 160x (2140x160=342,400).

[cid:image002.jpg@01D297D4.4428A120]

[cid:image003.jpg@01D297D4.4428A120]

This is also evident in the attached png image of wireshark output from a captured packet.

[cid:image004.png@01D297D4.4428A120]

Can you please advise if this is a bug, and if so, the best course of action (use a previous version?), or if I am doing something wrong.

I also attached the images in case they are stripped out.

Kind regards,

Jesse

Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 8, 2017, 4:21 AM

Post #3 of 11 (768 views)

Permalink

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 28, 2017, 4:54 AM

Post #4 of 11 (747 views)

Permalink

Good morning (depending on where you are),

We are still waiting for a response to this. Is this a known issue, or am I doing something wrong? Or has there been an update with a fix?

Kind regards,

Jesse

From: ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Jesse Alexander
Sent: Monday, March 06, 2017 11:55 AM
To: ntop@listgateway.unipi.it
Subject: [Ntop] Kentik nprobes sampling reporting excessive packets

Hello,

We are using the version of nprobe to work with Kentik (nprobes) and we are seeing packets being reported incorrectly when using sampling.

$sudo nprobe -v

Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu

Copyright 2002-17 ntop.org

Build OS: CentOS release 6.8 (Final)

Example script being used:

-q=A.A.A.A:6650

-n=X.X.X.X:9995

-n=Y.Y.Y.Y:2056

-V=5

-a=

-i=myri1-1

-S=160:1

-t=60

--if-networks=@/etc/cento/networks

-b=1

-Q=1

-u=1

--dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt

With the above, when I perform a controlled bandwidth test using a traffic generator, the packets per second being reported to Kentik is roughly 350K. The traffic generator is sending 2140pps of udp packets with 1460 byte payloads at a payload rate of 25 Mbps, so 25.9 Mbps with IP header.

We also sent the same data to another collector at the same time, and we saw exact same issue in the nfacctd data.

We saw that when we changed the sample rate, the pps was multiplied by that value, so in this case 160x (2140x160=342,400).

[cid:image002.jpg@01D2A790.1C5E0A60]

[cid:image003.jpg@01D2A790.1C5E0A60]

This is also evident in the attached png image of wireshark output from a captured packet.

[cid:image004.png@01D2A790.1C5E0A60]

Can you please advise if this is a bug, and if so, the best course of action (use a previous version?), or if I am doing something wrong.

I also attached the images in case they are stripped out.

Kind regards,

Jesse

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 28, 2017, 4:54 AM

Post #5 of 11 (747 views)

Permalink

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

deri at ntop

Mar 29, 2017, 1:52 AM

Post #6 of 11 (744 views)

Permalink

Jesse,
I have modified this but I have forgot to reply, my fault.

Please download the latest nprobe and it will work. In the new release
traffic upscale (i.e. x 160 multiplication) is not performed unless you
use --upscale-traffic

Regards Luca

On 03/28/2017 01:54 PM, Jesse Alexander wrote:
>
> Good morning (depending on where you are),
>
>
>
> We are still waiting for a response to this. Is this a known issue,
> or am I doing something wrong? Or has there been an update with a fix?
>
>
>
> Kind regards,
>
>
>
> Jesse
>
>
>
> *From:*ntop-bounces@listgateway.unipi.it
> [mailto:ntop-bounces@listgateway.unipi.it] *On Behalf Of *Jesse Alexander
> *Sent:* Monday, March 06, 2017 11:55 AM
> *To:* ntop@listgateway.unipi.it
> *Subject:* [Ntop] Kentik nprobes sampling reporting excessive packets
>
>
>
> Hello,
>
>
>
> We are using the version of nprobe to work with Kentik (nprobes) and
> we are seeing packets being reported incorrectly when using sampling.
>
>
>
> $sudo nprobe -v
>
> Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu
>
> Copyright 2002-17 ntop.org
>
> Build OS: CentOS release 6.8 (Final)
>
>
>
> Example script being used:
>
> -q=A.A.A.A:6650
>
> -n=X.X.X.X:9995
>
> -n=Y.Y.Y.Y:2056
>
> -V=5
>
> -a=
>
> -i=myri1-1
>
> -S=160:1
>
> -t=60
>
> --if-networks=@/etc/cento/networks
>
> -b=1
>
> -Q=1
>
> -u=1
>
> --dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt
>
>
>
> With the above, when I perform a controlled bandwidth test using a
> traffic generator, the packets per second being reported to Kentik is
> roughly 350K. The traffic generator is sending 2140pps of udp packets
> with 1460 byte payloads at a payload rate of 25 Mbps, so 25.9 Mbps
> with IP header.
>
>
>
> We also sent the same data to another collector at the same time, and
> we saw exact same issue in the nfacctd data.
>
>
>
> We saw that when we changed the sample rate, the pps was multiplied by
> that value, so in this case 160x (2140x160=342,400).
>
>
>
>
>
>
>
>
>
>
>
> This is also evident in the attached png image of wireshark output
> from a captured packet.
>
>
>
>
>
> Can you please advise if this is a bug, and if so, the best course of
> action (use a previous version?), or if I am doing something wrong.
>
>
>
> I also attached the images in case they are stripped out.
>
>
>
> Kind regards,
>
>
>
> Jesse
>
>
>
>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 29, 2017, 8:42 AM

Post #7 of 11 (744 views)

Permalink

Luca,

Thank you very much. I performed the nightly update but now I can't start it as a daemon because it removed /etc/init.d/nprobe and didn't add it back.

Welcome to nProbe v.7.5.170329 (r5697) for x86_64-unknown-linux-gnu

Can you please add it back?

Kind regards,

Jesse

From: ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Luca Deri
Sent: Wednesday, March 29, 2017 3:52 AM
To: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Kentik nprobes sampling reporting excessive packets

Jesse,
I have modified this but I have forgot to reply, my fault.

Please download the latest nprobe and it will work. In the new release traffic upscale (i.e. x 160 multiplication) is not performed unless you use --upscale-traffic

Regards Luca

On 03/28/2017 01:54 PM, Jesse Alexander wrote:
Good morning (depending on where you are),

We are still waiting for a response to this. Is this a known issue, or am I doing something wrong? Or has there been an update with a fix?

Kind regards,

Jesse

From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Jesse Alexander
Sent: Monday, March 06, 2017 11:55 AM
To: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>
Subject: [Ntop] Kentik nprobes sampling reporting excessive packets

Hello,

We are using the version of nprobe to work with Kentik (nprobes) and we are seeing packets being reported incorrectly when using sampling.

$sudo nprobe -v

Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu

Copyright 2002-17 ntop.org

Build OS: CentOS release 6.8 (Final)

Example script being used:

-q=A.A.A.A:6650

-n=X.X.X.X:9995

-n=Y.Y.Y.Y:2056

-V=5

-a=

-i=myri1-1

-S=160:1

-t=60

--if-networks=@/etc/cento/networks

-b=1

-Q=1

-u=1

--dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt

With the above, when I perform a controlled bandwidth test using a traffic generator, the packets per second being reported to Kentik is roughly 350K. The traffic generator is sending 2140pps of udp packets with 1460 byte payloads at a payload rate of 25 Mbps, so 25.9 Mbps with IP header.

We also sent the same data to another collector at the same time, and we saw exact same issue in the nfacctd data.

We saw that when we changed the sample rate, the pps was multiplied by that value, so in this case 160x (2140x160=342,400).

[cid:image002.jpg@01D2A878.2297EED0]

[cid:image003.jpg@01D2A878.2297EED0]

This is also evident in the attached png image of wireshark output from a captured packet.

[cid:image004.png@01D2A878.2297EED0]

Can you please advise if this is a bug, and if so, the best course of action (use a previous version?), or if I am doing something wrong.

I also attached the images in case they are stripped out.

Kind regards,

Jesse

_______________________________________________

Ntop mailing list

Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>

http://listgateway.unipi.it/mailman/listinfo/ntop

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 29, 2017, 8:42 AM

Post #8 of 11 (744 views)

Permalink

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

deri at ntop

Mar 30, 2017, 6:42 AM

Post #9 of 11 (743 views)

Permalink

Jesse,
this change was requested by Kentik

Luca

On 03/29/2017 05:42 PM, Jesse Alexander wrote:
>
> Luca,
>
>
>
> Thank you very much. I performed the nightly update but now I can?t
> start it as a daemon because it removed /etc/init.d/nprobe and didn?t
> add it back.
>
>
>
> Welcome to nProbe v.7.5.170329 (r5697) for x86_64-unknown-linux-gnu
>
>
>
> Can you please add it back?
>
>
>
> Kind regards,
>
>
>
> Jesse
>
>
>
>
>
> *From:*ntop-bounces@listgateway.unipi.it
> [mailto:ntop-bounces@listgateway.unipi.it] *On Behalf Of *Luca Deri
> *Sent:* Wednesday, March 29, 2017 3:52 AM
> *To:* ntop@listgateway.unipi.it
> *Subject:* Re: [Ntop] Kentik nprobes sampling reporting excessive packets
>
>
>
> Jesse,
> I have modified this but I have forgot to reply, my fault.
>
> Please download the latest nprobe and it will work. In the new release
> traffic upscale (i.e. x 160 multiplication) is not performed unless
> you use --upscale-traffic
>
> Regards Luca
>
> On 03/28/2017 01:54 PM, Jesse Alexander wrote:
>
> Good morning (depending on where you are),
>
>
>
> We are still waiting for a response to this. Is this a known
> issue, or am I doing something wrong? Or has there been an update
> with a fix?
>
>
>
> Kind regards,
>
>
>
> Jesse
>
>
>
> *From:*ntop-bounces@listgateway.unipi.it
> <mailto:ntop-bounces@listgateway.unipi.it>
> [mailto:ntop-bounces@listgateway.unipi.it] *On Behalf Of *Jesse
> Alexander
> *Sent:* Monday, March 06, 2017 11:55 AM
> *To:* ntop@listgateway.unipi.it <mailto:ntop@listgateway.unipi.it>
> *Subject:* [Ntop] Kentik nprobes sampling reporting excessive packets
>
>
>
> Hello,
>
>
>
> We are using the version of nprobe to work with Kentik (nprobes)
> and we are seeing packets being reported incorrectly when using
> sampling.
>
>
>
> $sudo nprobe -v
>
> Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu
>
> Copyright 2002-17 ntop.org
>
> Build OS: CentOS release 6.8 (Final)
>
>
>
> Example script being used:
>
> -q=A.A.A.A:6650
>
> -n=X.X.X.X:9995
>
> -n=Y.Y.Y.Y:2056
>
> -V=5
>
> -a=
>
> -i=myri1-1
>
> -S=160:1
>
> -t=60
>
> --if-networks=@/etc/cento/networks
>
> -b=1
>
> -Q=1
>
> -u=1
>
> --dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt
>
>
>
> With the above, when I perform a controlled bandwidth test using a
> traffic generator, the packets per second being reported to Kentik
> is roughly 350K. The traffic generator is sending 2140pps of udp
> packets with 1460 byte payloads at a payload rate of 25 Mbps, so
> 25.9 Mbps with IP header.
>
>
>
> We also sent the same data to another collector at the same time,
> and we saw exact same issue in the nfacctd data.
>
>
>
> We saw that when we changed the sample rate, the pps was
> multiplied by that value, so in this case 160x (2140x160=342,400).
>
>
>
>
>
>
>
>
>
>
>
> This is also evident in the attached png image of wireshark output
> from a captured packet.
>
>
>
>
>
> Can you please advise if this is a bug, and if so, the best course
> of action (use a previous version?), or if I am doing something wrong.
>
>
>
> I also attached the images in case they are stripped out.
>
>
>
> Kind regards,
>
>
>
> Jesse
>
>
>
>
>
>
>
>
> _______________________________________________
>
> Ntop mailing list
>
> Ntop@listgateway.unipi.it <mailto:Ntop@listgateway.unipi.it>
>
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop

Re: Kentik nprobes sampling reporting excessive packets [ In reply to ]

jessea at datafoundry

Mar 30, 2017, 6:55 AM

Post #10 of 11 (743 views)

Permalink

Luca,

Interesting. Thanks for the information.

Jesse

From: ntop-bounces@listgateway.unipi.it [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Luca Deri
Sent: Thursday, March 30, 2017 8:43 AM
To: ntop@listgateway.unipi.it
Subject: Re: [Ntop] Kentik nprobes sampling reporting excessive packets

Jesse,
this change was requested by Kentik

Luca

On 03/29/2017 05:42 PM, Jesse Alexander wrote:
Luca,

Thank you very much. I performed the nightly update but now I can't start it as a daemon because it removed /etc/init.d/nprobe and didn't add it back.

Welcome to nProbe v.7.5.170329 (r5697) for x86_64-unknown-linux-gnu

Can you please add it back?

Kind regards,

Jesse

From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Luca Deri
Sent: Wednesday, March 29, 2017 3:52 AM
To: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>
Subject: Re: [Ntop] Kentik nprobes sampling reporting excessive packets

Jesse,
I have modified this but I have forgot to reply, my fault.

Please download the latest nprobe and it will work. In the new release traffic upscale (i.e. x 160 multiplication) is not performed unless you use --upscale-traffic

Regards Luca

On 03/28/2017 01:54 PM, Jesse Alexander wrote:
Good morning (depending on where you are),

We are still waiting for a response to this. Is this a known issue, or am I doing something wrong? Or has there been an update with a fix?

Kind regards,

Jesse

From: ntop-bounces@listgateway.unipi.it<mailto:ntop-bounces@listgateway.unipi.it> [mailto:ntop-bounces@listgateway.unipi.it] On Behalf Of Jesse Alexander
Sent: Monday, March 06, 2017 11:55 AM
To: ntop@listgateway.unipi.it<mailto:ntop@listgateway.unipi.it>
Subject: [Ntop] Kentik nprobes sampling reporting excessive packets

Hello,

We are using the version of nprobe to work with Kentik (nprobes) and we are seeing packets being reported incorrectly when using sampling.

$sudo nprobe -v

Welcome to nProbe v.7.5.170306 (r5675) for x86_64-unknown-linux-gnu

Copyright 2002-17 ntop.org

Build OS: CentOS release 6.8 (Final)

Example script being used:

-q=A.A.A.A:6650

-n=X.X.X.X:9995

-n=Y.Y.Y.Y:2056

-V=5

-a=

-i=myri1-1

-S=160:1

-t=60

--if-networks=@/etc/cento/networks

-b=1

-Q=1

-u=1

--dump-stats=/var/log/nprobe/myri1-1-0_flows_stats.txt

With the above, when I perform a controlled bandwidth test using a traffic generator, the packets per second being reported to Kentik is roughly 350K. The traffic generator is sending 2140pps of udp packets with 1460 byte payloads at a payload rate of 25 Mbps, so 25.9 Mbps with IP header.

We also sent the same data to another collector at the same time, and we saw exact same issue in the nfacctd data.

We saw that when we changed the sample rate, the pps was multiplied by that value, so in this case 160x (2140x160=342,400).

[cid:image001.jpg@01D2A933.599EEEE0]

[cid:image002.jpg@01D2A933.599EEEE0]

This is also evident in the attached png image of wireshark output from a captured packet.

[cid:image003.png@01D2A933.599EEEE0]

Can you please advise if this is a bug, and if so, the best course of action (use a previous version?), or if I am doing something wrong.

I also attached the images in case they are stripped out.

Kind regards,

Jesse

_______________________________________________

Ntop mailing list

Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>

http://listgateway.unipi.it/mailman/listinfo/ntop

_______________________________________________

Ntop mailing list

Ntop@listgateway.unipi.it<mailto:Ntop@listgateway.unipi.it>

http://listgateway.unipi.it/mailman/listinfo/ntop

Mailing List Archive

Attached Files:

Attached Files:

Attached Files:

Attached Files:

Attached Files:

Attached Files:

Attached Files:

Attached Files:

Attached Files: