Mailing List Archive: packet drops are observed in pf

packet drops are observed in pf_ring statistics

Mar 18, 2018, 10:35 PM

Post #1 of 6 (911 views)

Hi,

We are observing some intermittent drops in pf_ring statistics in one of
the production sites. I need your valuable inputs to debug this issue.

Brief description of the setup is -

Fragmentation is enabled at kernel level. Traffic at site is varying from
500Mbps to 1 Gbps. We have observed cpu usage per thread of our application
and it is not exceeding ~20-30 percent and there are no alarms raised by
applications.
So I am suspecting whether kernel fragmentation logic could slow down the
processing and causing pf_ring to drop these packets. Can you please
confirm whether this can cause the drop ?If yes, How shall we debug this
issue.
Please let me know if you need any data from the sites.

Regards,
Gautam

Re: packet drops are observed in pf_ring statistics [ In reply to ]

cardigliano at ntop

Mar 19, 2018, 1:21 AM

Post #2 of 6 (911 views)

Permalink

Hi Gautam
I guess you are using standard drivers, with defragmentation enabled in the pf_ring kernel module,
in that case if you have many fragments, at that rate, that could slow down the processing. Did you
try disabling defragmentation to figure out if the bottleneck is there?

Regards
Alfredo

> On 19 Mar 2018, at 06:35, Chandrika Gautam <chandrika.iitd.rock@gmail.com> wrote:
>
> Hi,
>
> We are observing some intermittent drops in pf_ring statistics in one of the production sites. I need your valuable inputs to debug this issue.
>
> Brief description of the setup is -
>
> Fragmentation is enabled at kernel level. Traffic at site is varying from 500Mbps to 1 Gbps. We have observed cpu usage per thread of our application and it is not exceeding ~20-30 percent and there are no alarms raised by applications.
> So I am suspecting whether kernel fragmentation logic could slow down the processing and causing pf_ring to drop these packets. Can you please confirm whether this can cause the drop ?If yes, How shall we debug this issue.
> Please let me know if you need any data from the sites.
>
>
> Regards,
> Gautam
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: packet drops are observed in pf_ring statistics [ In reply to ]

chandrika.iitd.rock at gmail

Mar 19, 2018, 9:19 PM

Post #3 of 6 (911 views)

Permalink

Hi Alfredo,

Yes its a standard driver (Intel 82599 10 G) and server is hpdl380g9
servers(cpu clock speed 2GHz ).

We have another deployment in which fragmentation logic is enabled in
applications(g8 2.9 GHz) and we are not observing a drop in pf_ring stats.

How can we prove this to customer that enabling fragmentation in pf_ring
kernel could be causing this?
Or can we tune ipfrag related parameters to support more number of
fragmented packets.

On Mon, Mar 19, 2018 at 1:51 PM, Alfredo Cardigliano <cardigliano@ntop.org>
wrote:

> Hi Gautam
> I guess you are using standard drivers, with defragmentation enabled in
> the pf_ring kernel module,
> in that case if you have many fragments, at that rate, that could slow
> down the processing. Did you
> try disabling defragmentation to figure out if the bottleneck is there?
>
> Regards
> Alfredo
>
> > On 19 Mar 2018, at 06:35, Chandrika Gautam <
> chandrika.iitd.rock@gmail.com> wrote:
> >
> > Hi,
> >
> > We are observing some intermittent drops in pf_ring statistics in one of
> the production sites. I need your valuable inputs to debug this issue.
> >
> > Brief description of the setup is -
> >
> > Fragmentation is enabled at kernel level. Traffic at site is varying
> from 500Mbps to 1 Gbps. We have observed cpu usage per thread of our
> application and it is not exceeding ~20-30 percent and there are no alarms
> raised by applications.
> > So I am suspecting whether kernel fragmentation logic could slow down
> the processing and causing pf_ring to drop these packets. Can you please
> confirm whether this can cause the drop ?If yes, How shall we debug this
> issue.
> > Please let me know if you need any data from the sites.
> >
> >
> > Regards,
> > Gautam
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>

Re: packet drops are observed in pf_ring statistics [ In reply to ]

cardigliano at ntop

Mar 20, 2018, 1:53 AM

Post #4 of 6 (909 views)

Permalink

Hi Gautam
pf_ring leverages on the linux kernel defragmentation support, thus we cannot do much on our side for that,
I see that your application is not exceeding 30% of cpu usage, are you sure you do not have spikes that are
hard to detect and your application cannot keep up with processing? Please provide pf_ring statistics in order
to figure out if packet loss is at kernel level or application level.

Alfredo

> On 20 Mar 2018, at 05:19, Chandrika Gautam <chandrika.iitd.rock@gmail.com> wrote:
>
> Hi Alfredo,
>
> Yes its a standard driver (Intel 82599 10 G) and server is hpdl380g9 servers(cpu clock speed 2GHz ).
>
> We have another deployment in which fragmentation logic is enabled in applications(g8 2.9 GHz) and we are not observing a drop in pf_ring stats.
>
> How can we prove this to customer that enabling fragmentation in pf_ring kernel could be causing this?
> Or can we tune ipfrag related parameters to support more number of fragmented packets.
>
>
> On Mon, Mar 19, 2018 at 1:51 PM, Alfredo Cardigliano <cardigliano@ntop.org <mailto:cardigliano@ntop.org>> wrote:
> Hi Gautam
> I guess you are using standard drivers, with defragmentation enabled in the pf_ring kernel module,
> in that case if you have many fragments, at that rate, that could slow down the processing. Did you
> try disabling defragmentation to figure out if the bottleneck is there?
>
> Regards
> Alfredo
>
> > On 19 Mar 2018, at 06:35, Chandrika Gautam <chandrika.iitd.rock@gmail.com <mailto:chandrika.iitd.rock@gmail.com>> wrote:
> >
> > Hi,
> >
> > We are observing some intermittent drops in pf_ring statistics in one of the production sites. I need your valuable inputs to debug this issue.
> >
> > Brief description of the setup is -
> >
> > Fragmentation is enabled at kernel level. Traffic at site is varying from 500Mbps to 1 Gbps. We have observed cpu usage per thread of our application and it is not exceeding ~20-30 percent and there are no alarms raised by applications.
> > So I am suspecting whether kernel fragmentation logic could slow down the processing and causing pf_ring to drop these packets. Can you please confirm whether this can cause the drop ?If yes, How shall we debug this issue.
> > Please let me know if you need any data from the sites.
> >
> >
> > Regards,
> > Gautam
> > _______________________________________________
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: packet drops are observed in pf_ring statistics [ In reply to ]

chandrika.iitd.rock at gmail

Mar 29, 2018, 12:15 AM

Post #5 of 6 (889 views)

Permalink

Hi Alfredo,

Yes I am quite sure that this is due to fragmentation logic in kernel not
in application.
We have one deployment where fragmentation logic is enabled in application
but we do not see any drops in pf_ring statistics also.

Will output of /proc/net/pf_ring/<pid> be enough ? Are you looking for
some other statistics?

Regards,
Chandrika

On Tue, Mar 20, 2018 at 2:23 PM, Alfredo Cardigliano <cardigliano@ntop.org>
wrote:

> Hi Gautam
> pf_ring leverages on the linux kernel defragmentation support, thus we
> cannot do much on our side for that,
> I see that your application is not exceeding 30% of cpu usage, are you
> sure you do not have spikes that are
> hard to detect and your application cannot keep up with processing? Please
> provide pf_ring statistics in order
> to figure out if packet loss is at kernel level or application level.
>
> Alfredo
>
> On 20 Mar 2018, at 05:19, Chandrika Gautam <chandrika.iitd.rock@gmail.com>
> wrote:
>
> Hi Alfredo,
>
> Yes its a standard driver (Intel 82599 10 G) and server is hpdl380g9
> servers(cpu clock speed 2GHz ).
>
> We have another deployment in which fragmentation logic is enabled in
> applications(g8 2.9 GHz) and we are not observing a drop in pf_ring stats.
>
> How can we prove this to customer that enabling fragmentation in pf_ring
> kernel could be causing this?
> Or can we tune ipfrag related parameters to support more number of
> fragmented packets.
>
>
> On Mon, Mar 19, 2018 at 1:51 PM, Alfredo Cardigliano <cardigliano@ntop.org
> > wrote:
>
>> Hi Gautam
>> I guess you are using standard drivers, with defragmentation enabled in
>> the pf_ring kernel module,
>> in that case if you have many fragments, at that rate, that could slow
>> down the processing. Did you
>> try disabling defragmentation to figure out if the bottleneck is there?
>>
>> Regards
>> Alfredo
>>
>> > On 19 Mar 2018, at 06:35, Chandrika Gautam <
>> chandrika.iitd.rock@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > We are observing some intermittent drops in pf_ring statistics in one
>> of the production sites. I need your valuable inputs to debug this issue.
>> >
>> > Brief description of the setup is -
>> >
>> > Fragmentation is enabled at kernel level. Traffic at site is varying
>> from 500Mbps to 1 Gbps. We have observed cpu usage per thread of our
>> application and it is not exceeding ~20-30 percent and there are no alarms
>> raised by applications.
>> > So I am suspecting whether kernel fragmentation logic could slow down
>> the processing and causing pf_ring to drop these packets. Can you please
>> confirm whether this can cause the drop ?If yes, How shall we debug this
>> issue.
>> > Please let me know if you need any data from the sites.
>> >
>> >
>> > Regards,
>> > Gautam
>> > _______________________________________________
>> > Ntop-misc mailing list
>> > Ntop-misc@listgateway.unipi.it
>> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>

Re: packet drops are observed in pf_ring statistics [ In reply to ]

cardigliano at ntop

Mar 29, 2018, 12:59 AM

Post #6 of 6 (889 views)

Permalink

Yes please provide /proc/net/pf_ring/<pid> and ethtool -S <interface>

Alfredo

> On 29 Mar 2018, at 09:15, Chandrika Gautam <chandrika.iitd.rock@gmail.com> wrote:
>
> Hi Alfredo,
>
> Yes I am quite sure that this is due to fragmentation logic in kernel not in application.
> We have one deployment where fragmentation logic is enabled in application but we do not see any drops in pf_ring statistics also.
>
> Will output of /proc/net/pf_ring/<pid> be enough ? Are you looking for some other statistics?
>
> Regards,
> Chandrika
>
> On Tue, Mar 20, 2018 at 2:23 PM, Alfredo Cardigliano <cardigliano@ntop.org <mailto:cardigliano@ntop.org>> wrote:
> Hi Gautam
> pf_ring leverages on the linux kernel defragmentation support, thus we cannot do much on our side for that,
> I see that your application is not exceeding 30% of cpu usage, are you sure you do not have spikes that are
> hard to detect and your application cannot keep up with processing? Please provide pf_ring statistics in order
> to figure out if packet loss is at kernel level or application level.
>
> Alfredo
>
>> On 20 Mar 2018, at 05:19, Chandrika Gautam <chandrika.iitd.rock@gmail.com <mailto:chandrika.iitd.rock@gmail.com>> wrote:
>>
>> Hi Alfredo,
>>
>> Yes its a standard driver (Intel 82599 10 G) and server is hpdl380g9 servers(cpu clock speed 2GHz ).
>>
>> We have another deployment in which fragmentation logic is enabled in applications(g8 2.9 GHz) and we are not observing a drop in pf_ring stats.
>>
>> How can we prove this to customer that enabling fragmentation in pf_ring kernel could be causing this?
>> Or can we tune ipfrag related parameters to support more number of fragmented packets.
>>
>>
>> On Mon, Mar 19, 2018 at 1:51 PM, Alfredo Cardigliano <cardigliano@ntop.org <mailto:cardigliano@ntop.org>> wrote:
>> Hi Gautam
>> I guess you are using standard drivers, with defragmentation enabled in the pf_ring kernel module,
>> in that case if you have many fragments, at that rate, that could slow down the processing. Did you
>> try disabling defragmentation to figure out if the bottleneck is there?
>>
>> Regards
>> Alfredo
>>
>> > On 19 Mar 2018, at 06:35, Chandrika Gautam <chandrika.iitd.rock@gmail.com <mailto:chandrika.iitd.rock@gmail.com>> wrote:
>> >
>> > Hi,
>> >
>> > We are observing some intermittent drops in pf_ring statistics in one of the production sites. I need your valuable inputs to debug this issue.
>> >
>> > Brief description of the setup is -
>> >
>> > Fragmentation is enabled at kernel level. Traffic at site is varying from 500Mbps to 1 Gbps. We have observed cpu usage per thread of our application and it is not exceeding ~20-30 percent and there are no alarms raised by applications.
>> > So I am suspecting whether kernel fragmentation logic could slow down the processing and causing pf_ring to drop these packets. Can you please confirm whether this can cause the drop ?If yes, How shall we debug this issue.
>> > Please let me know if you need any data from the sites.
>> >
>> >
>> > Regards,
>> > Gautam
>> > _______________________________________________
>> > Ntop-misc mailing list
>> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>>
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>
> _______________________________________________
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Mailing List Archive

Attached Files:

Attached Files:

Attached Files: