Mailing List Archive: FCX High CPU Diagnostics

FCX High CPU Diagnostics

Apr 3, 2019, 10:51 AM

Post #1 of 11 (3983 views)

Anyone have any thoughts on how to diagnose intermittent 99% CPU utilization on a FCX624? We're running Layer 3 code and high CPU usage seems to be causing BGP sessions to drop.

I'm able to run sh cpu, but it doesn't give a breakdown of what process is using high cpu.

Port utilization is low, BGP is only taking a single route at this point so re-convergence is not beating it up, and I don't see anything revealing in the logs. The problem has persisted after multiple reloads.

Thanks for the guidance.

Re: FCX High CPU Diagnostics [ In reply to ]

jk at ip-clear

Apr 3, 2019, 11:25 AM

Post #2 of 11 (3983 views)

Permalink

Does
show processes cpu
work on the FCX?

On 3 Apr 2019, at 19:51, Derek Maxwell wrote:

> Anyone have any thoughts on how to diagnose intermittent 99% CPU
> utilization on a FCX624? We're running Layer 3 code and high CPU usage
> seems to be causing BGP sessions to drop.
>
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

derek.maxwell at chosentechgroup

Apr 3, 2019, 11:34 AM

Post #3 of 11 (3983 views)

Permalink

Yes and that's exactly what I was looking for. Thanks!

Will continue to debug from here when it pops up again and see what I find.

-----Original Message-----
From: J?rg Kost <jk@ip-clear.de>
Sent: Wednesday, April 3, 2019 11:26 AM
To: Derek Maxwell <derek.maxwell@chosentechgroup.com>; foundry-nsp <foundry-nsp@puck.nether.net>
Subject: Re: [f-nsp] FCX High CPU Diagnostics

Does
show processes cpu
work on the FCX?

On 3 Apr 2019, at 19:51, Derek Maxwell wrote:

> Anyone have any thoughts on how to diagnose intermittent 99% CPU
> utilization on a FCX624? We're running Layer 3 code and high CPU usage
> seems to be causing BGP sessions to drop.
>
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

derek.maxwell at chosentechgroup

Apr 3, 2019, 12:13 PM

Post #4 of 11 (3983 views)

Permalink

So, something doesn't add up here:

During a high CPU event, this is the output of show processes cpu:

show processes cpu
Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
ARP 0.00 0.00 0.00 0.00 1989
BGP 0.00 0.00 0.00 0.00 2809
DOT1X 0.00 0.00 0.00 0.00 0
GVRP 0.00 0.00 0.00 0.00 0
ICMP 0.00 0.00 0.00 0.00 1690
IP 1.10 1.37 1.21 0.58 128381
OSPF 0.00 0.00 0.00 0.00 0
RIP 0.00 0.00 0.00 0.00 20
STP 0.00 0.00 0.00 0.00 0
VRRP 0.00 0.00 0.00 0.00 0
Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
IPv6 0.00 0.00 0.00 0.01 33559
ICMP6 0.00 0.01 0.01 0.01 31180
ND6 0.00 0.00 0.00 0.00 1461
RIPng 0.00 0.00 0.00 0.00 41
OSPFv3 0.00 0.00 0.00 0.00 0
IPV6_RX 0.00 0.00 0.00 0.00 181

IP is the only high CPU process, but it's only at 1.1-1.37%.

This is the output of show cpu:
show cpu
99 percent busy, from 289 sec ago
1 sec avg: 99 percent busy
5 sec avg: 99 percent busy
60 sec avg: 99 percent busy
300 sec avg: 99 percent busy

It's 99% busy and the control plane is barely responding.

Any thoughts on a hidden process that may be monopolizing the system?

?On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry-nsp-bounces@puck.nether.net on behalf of derek.maxwell@chosentechgroup.com> wrote:

Yes and that's exactly what I was looking for. Thanks!

Will continue to debug from here when it pops up again and see what I find.

-----Original Message-----
From: Jörg Kost <jk@ip-clear.de>
Sent: Wednesday, April 3, 2019 11:26 AM
To: Derek Maxwell <derek.maxwell@chosentechgroup.com>; foundry-nsp <foundry-nsp@puck.nether.net>
Subject: Re: [f-nsp] FCX High CPU Diagnostics

Does
show processes cpu
work on the FCX?

On 3 Apr 2019, at 19:51, Derek Maxwell wrote:

> Anyone have any thoughts on how to diagnose intermittent 99% CPU
> utilization on a FCX624? We're running Layer 3 code and high CPU usage
> seems to be causing BGP sessions to drop.
>
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

timoid at timoid

Apr 3, 2019, 12:26 PM

Post #5 of 11 (3983 views)

Permalink

Are you trying to put more routes than can go into TCAM?

Does 'show ip bgp sum' show any of your BGP sessions doing FIB programming?

I don't know the FCX but a command from mlx/ce[s-r] is 'rcon lp', then run 'show task'. That will give you an idea if the LP is busy, if the command exists.

> -----Original Message-----
> From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On
> Behalf Of Derek Maxwell
> Sent: Thursday, 4 April 2019 5:14 AM
> To: Jörg Kost <jk@ip-clear.de>; foundry-nsp <foundry-
> nsp@puck.nether.net>
> Subject: Re: [f-nsp] FCX High CPU Diagnostics
>
> So, something doesn't add up here:
>
> During a high CPU event, this is the output of show processes cpu:
>
> show processes cpu
> Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> ARP 0.00 0.00 0.00 0.00 1989
> BGP 0.00 0.00 0.00 0.00 2809
> DOT1X 0.00 0.00 0.00 0.00 0
> GVRP 0.00 0.00 0.00 0.00 0
> ICMP 0.00 0.00 0.00 0.00 1690
> IP 1.10 1.37 1.21 0.58 128381
> OSPF 0.00 0.00 0.00 0.00 0
> RIP 0.00 0.00 0.00 0.00 20
> STP 0.00 0.00 0.00 0.00 0
> VRRP 0.00 0.00 0.00 0.00 0
> Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> IPv6 0.00 0.00 0.00 0.01 33559
> ICMP6 0.00 0.01 0.01 0.01 31180
> ND6 0.00 0.00 0.00 0.00 1461
> RIPng 0.00 0.00 0.00 0.00 41
> OSPFv3 0.00 0.00 0.00 0.00 0
> IPV6_RX 0.00 0.00 0.00 0.00 181
>
> IP is the only high CPU process, but it's only at 1.1-1.37%.
>
>
> This is the output of show cpu:
> show cpu
> 99 percent busy, from 289 sec ago
> 1 sec avg: 99 percent busy
> 5 sec avg: 99 percent busy
> 60 sec avg: 99 percent busy
> 300 sec avg: 99 percent busy
>
> It's 99% busy and the control plane is barely responding.
>
> Any thoughts on a hidden process that may be monopolizing the system?
>
>
> ?On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry-
> nsp-bounces@puck.nether.net on behalf of
> derek.maxwell@chosentechgroup.com> wrote:
>
> Yes and that's exactly what I was looking for. Thanks!
>
> Will continue to debug from here when it pops up again and see what I
> find.
>
> -----Original Message-----
> From: Jörg Kost <jk@ip-clear.de>
> Sent: Wednesday, April 3, 2019 11:26 AM
> To: Derek Maxwell <derek.maxwell@chosentechgroup.com>; foundry-nsp
> <foundry-nsp@puck.nether.net>
> Subject: Re: [f-nsp] FCX High CPU Diagnostics
>
> Does
> show processes cpu
> work on the FCX?
>
> On 3 Apr 2019, at 19:51, Derek Maxwell wrote:
>
> > Anyone have any thoughts on how to diagnose intermittent 99% CPU
> > utilization on a FCX624? We're running Layer 3 code and high CPU usage
> > seems to be causing BGP sessions to drop.
> >
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

hf0002 at uah

Apr 3, 2019, 12:30 PM

Post #6 of 11 (3983 views)

Permalink

I don't know FCX, but on the ICX we have the command "dm cpu" that
gives more details. Lists the 5 most busy tasks on the system. Run it
two times 1 minute apart, if the first time gave you bogus info.

SSH@LAB-7450STACK#dm cpu
appl-task pid 1322 tid 0xa5270460 cpu usage 84%
console-task pid 1369 tid 0xa4a42460 cpu usage 0%
flash-task pid 1281 tid 0xb5fd6460 cpu usage 0%
sys-tmr-task pid 1280 tid 0xb5ff6460 cpu usage 0%
itc-task pid 1285 tid 0xb5f76460 cpu usage 0%
loop cnt: 567307594 569243809 -> 1936215
read cnt: 136318039 136318293 -> 254
loop rate = 1936215/270 = 7171
rcode_old = 100 - (100*7171/3900) = -83
rcode_log2= 100-(100*log2(7171)/log2(3900))=-7
1/100 busy, from 27 sec ago

--
Hunter Fuller
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering

On Wed, Apr 3, 2019 at 2:28 PM Tim Warnock <timoid@timoid.org> wrote:
>
> Are you trying to put more routes than can go into TCAM?
>
> Does 'show ip bgp sum' show any of your BGP sessions doing FIB programming?
>
> I don't know the FCX but a command from mlx/ce[s-r] is 'rcon lp', then run 'show task'. That will give you an idea if the LP is busy, if the command exists.
>
> > -----Original Message-----
> > From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On
> > Behalf Of Derek Maxwell
> > Sent: Thursday, 4 April 2019 5:14 AM
> > To: Jörg Kost <jk@ip-clear.de>; foundry-nsp <foundry-
> > nsp@puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > So, something doesn't add up here:
> >
> > During a high CPU event, this is the output of show processes cpu:
> >
> > show processes cpu
> > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> > ARP 0.00 0.00 0.00 0.00 1989
> > BGP 0.00 0.00 0.00 0.00 2809
> > DOT1X 0.00 0.00 0.00 0.00 0
> > GVRP 0.00 0.00 0.00 0.00 0
> > ICMP 0.00 0.00 0.00 0.00 1690
> > IP 1.10 1.37 1.21 0.58 128381
> > OSPF 0.00 0.00 0.00 0.00 0
> > RIP 0.00 0.00 0.00 0.00 20
> > STP 0.00 0.00 0.00 0.00 0
> > VRRP 0.00 0.00 0.00 0.00 0
> > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> > IPv6 0.00 0.00 0.00 0.01 33559
> > ICMP6 0.00 0.01 0.01 0.01 31180
> > ND6 0.00 0.00 0.00 0.00 1461
> > RIPng 0.00 0.00 0.00 0.00 41
> > OSPFv3 0.00 0.00 0.00 0.00 0
> > IPV6_RX 0.00 0.00 0.00 0.00 181
> >
> > IP is the only high CPU process, but it's only at 1.1-1.37%.
> >
> >
> > This is the output of show cpu:
> > show cpu
> > 99 percent busy, from 289 sec ago
> > 1 sec avg: 99 percent busy
> > 5 sec avg: 99 percent busy
> > 60 sec avg: 99 percent busy
> > 300 sec avg: 99 percent busy
> >
> > It's 99% busy and the control plane is barely responding.
> >
> > Any thoughts on a hidden process that may be monopolizing the system?
> >
> >
> > ?On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry-
> > nsp-bounces@puck.nether.net on behalf of
> > derek.maxwell@chosentechgroup.com> wrote:
> >
> > Yes and that's exactly what I was looking for. Thanks!
> >
> > Will continue to debug from here when it pops up again and see what I
> > find.
> >
> > -----Original Message-----
> > From: Jörg Kost <jk@ip-clear.de>
> > Sent: Wednesday, April 3, 2019 11:26 AM
> > To: Derek Maxwell <derek.maxwell@chosentechgroup.com>; foundry-nsp
> > <foundry-nsp@puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > Does
> > show processes cpu
> > work on the FCX?
> >
> > On 3 Apr 2019, at 19:51, Derek Maxwell wrote:
> >
> > > Anyone have any thoughts on how to diagnose intermittent 99% CPU
> > > utilization on a FCX624? We're running Layer 3 code and high CPU usage
> > > seems to be causing BGP sessions to drop.
> > >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp@puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
> >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp@puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

derek.maxwell at chosentechgroup

Apr 3, 2019, 12:49 PM

Post #7 of 11 (3983 views)

Permalink

Output of dm cpu:

idle pid 0 id 536918812 cpu 0 usage 0
main pid 1 id 536918812 cpu 0 usage 0
flash pid 0 id 536918812 cpu 0 usage 0
arrays at 2238d894 and 2238dd6c
from entry 78 to entry 35
loop cnt: 3395 3626 -> 231
read cnt: 57670 61592 -> 3922
loop rate = 231/430 = 0
rcode = 100 - (100*0/2550) = 100
99/100 busy, from 43 sec ago

sho ip bgp sum does not show any FIB programming

We have fewer than 10 routes in so we're definitely not exceeding any limits.

We have icmp redirects disabled

?On 4/3/19, 12:35 PM, "foundry-nsp on behalf of Hunter Fuller" <foundry-nsp-bounces@puck.nether.net on behalf of hf0002@uah.edu> wrote:

I don't know FCX, but on the ICX we have the command "dm cpu" that
gives more details. Lists the 5 most busy tasks on the system. Run it
two times 1 minute apart, if the first time gave you bogus info.

SSH@LAB-7450STACK#dm cpu
appl-task pid 1322 tid 0xa5270460 cpu usage 84%
console-task pid 1369 tid 0xa4a42460 cpu usage 0%
flash-task pid 1281 tid 0xb5fd6460 cpu usage 0%
sys-tmr-task pid 1280 tid 0xb5ff6460 cpu usage 0%
itc-task pid 1285 tid 0xb5f76460 cpu usage 0%
loop cnt: 567307594 569243809 -> 1936215
read cnt: 136318039 136318293 -> 254
loop rate = 1936215/270 = 7171
rcode_old = 100 - (100*7171/3900) = -83
rcode_log2= 100-(100*log2(7171)/log2(3900))=-7
1/100 busy, from 27 sec ago

--
Hunter Fuller
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering

On Wed, Apr 3, 2019 at 2:28 PM Tim Warnock <timoid@timoid.org> wrote:
>
> Are you trying to put more routes than can go into TCAM?
>
> Does 'show ip bgp sum' show any of your BGP sessions doing FIB programming?
>
> I don't know the FCX but a command from mlx/ce[s-r] is 'rcon lp', then run 'show task'. That will give you an idea if the LP is busy, if the command exists.
>
> > -----Original Message-----
> > From: foundry-nsp [mailto:foundry-nsp-bounces@puck.nether.net] On
> > Behalf Of Derek Maxwell
> > Sent: Thursday, 4 April 2019 5:14 AM
> > To: Jörg Kost <jk@ip-clear.de>; foundry-nsp <foundry-
> > nsp@puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > So, something doesn't add up here:
> >
> > During a high CPU event, this is the output of show processes cpu:
> >
> > show processes cpu
> > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> > ARP 0.00 0.00 0.00 0.00 1989
> > BGP 0.00 0.00 0.00 0.00 2809
> > DOT1X 0.00 0.00 0.00 0.00 0
> > GVRP 0.00 0.00 0.00 0.00 0
> > ICMP 0.00 0.00 0.00 0.00 1690
> > IP 1.10 1.37 1.21 0.58 128381
> > OSPF 0.00 0.00 0.00 0.00 0
> > RIP 0.00 0.00 0.00 0.00 20
> > STP 0.00 0.00 0.00 0.00 0
> > VRRP 0.00 0.00 0.00 0.00 0
> > Process Name 5Sec(%) 1Min(%) 5Min(%) 15Min(%) Runtime(ms)
> > IPv6 0.00 0.00 0.00 0.01 33559
> > ICMP6 0.00 0.01 0.01 0.01 31180
> > ND6 0.00 0.00 0.00 0.00 1461
> > RIPng 0.00 0.00 0.00 0.00 41
> > OSPFv3 0.00 0.00 0.00 0.00 0
> > IPV6_RX 0.00 0.00 0.00 0.00 181
> >
> > IP is the only high CPU process, but it's only at 1.1-1.37%.
> >
> >
> > This is the output of show cpu:
> > show cpu
> > 99 percent busy, from 289 sec ago
> > 1 sec avg: 99 percent busy
> > 5 sec avg: 99 percent busy
> > 60 sec avg: 99 percent busy
> > 300 sec avg: 99 percent busy
> >
> > It's 99% busy and the control plane is barely responding.
> >
> > Any thoughts on a hidden process that may be monopolizing the system?
> >
> >
> > On 4/3/19, 11:36 AM, "foundry-nsp on behalf of Derek Maxwell" <foundry-
> > nsp-bounces@puck.nether.net on behalf of
> > derek.maxwell@chosentechgroup.com> wrote:
> >
> > Yes and that's exactly what I was looking for. Thanks!
> >
> > Will continue to debug from here when it pops up again and see what I
> > find.
> >
> > -----Original Message-----
> > From: Jörg Kost <jk@ip-clear.de>
> > Sent: Wednesday, April 3, 2019 11:26 AM
> > To: Derek Maxwell <derek.maxwell@chosentechgroup.com>; foundry-nsp
> > <foundry-nsp@puck.nether.net>
> > Subject: Re: [f-nsp] FCX High CPU Diagnostics
> >
> > Does
> > show processes cpu
> > work on the FCX?
> >
> > On 3 Apr 2019, at 19:51, Derek Maxwell wrote:
> >
> > > Anyone have any thoughts on how to diagnose intermittent 99% CPU
> > > utilization on a FCX624? We're running Layer 3 code and high CPU usage
> > > seems to be causing BGP sessions to drop.
> > >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp@puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> >
> >
> > _______________________________________________
> > foundry-nsp mailing list
> > foundry-nsp@puck.nether.net
> > http://puck.nether.net/mailman/listinfo/foundry-nsp
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

jk at ip-clear

Apr 3, 2019, 1:12 PM

Post #8 of 11 (3983 views)

Permalink

Can you eliminate any SNMP tool, that might spin the CPU around?

On 3 Apr 2019, at 21:49, Derek Maxwell wrote:

> Output of dm cpu:
>
> idle pid 0 id 536918812 cpu 0 usage 0
> main pid 1 id 536918812 cpu 0 usage 0
> flash pid 0 id 536918812 cpu 0 usage 0
> arrays at 2238d894 and 2238dd6c
> from entry 78 to entry 35
> loop cnt: 3395 3626 -> 231
> read cnt: 57670 61592 -> 3922
> loop rate = 231/430 = 0
> rcode = 100 - (100*0/2550) = 100
> 99/100 busy, from 43 sec ago
>
> sho ip bgp sum does not show any FIB programming
>
> We have fewer than 10 routes in so we're definitely not exceeding any
> limits.
>
> We have icmp redirects disabled
>
>
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

derek.maxwell at chosentechgroup

Apr 3, 2019, 1:55 PM

Post #9 of 11 (3983 views)

Permalink

That may have actually done the trick. We aren't using any snmp tools on this box, but I added no snmp-server to the config and that brought cpu back down to 1%. I'll keep watching and see where it lands.

If there are any other thoughts, feel free to send them over. I appreciate any input.

-----Original Message-----
From: J?rg Kost <jk@ip-clear.de>
Sent: Wednesday, April 3, 2019 1:12 PM
To: Derek Maxwell <derek.maxwell@chosentechgroup.com>
Cc: foundry-nsp <foundry-nsp@puck.nether.net>
Subject: Re: [f-nsp] FCX High CPU Diagnostics

Can you eliminate any SNMP tool, that might spin the CPU around?

On 3 Apr 2019, at 21:49, Derek Maxwell wrote:

> Output of dm cpu:
>
> idle pid 0 id 536918812 cpu 0 usage 0
> main pid 1 id 536918812 cpu 0 usage 0
> flash pid 0 id 536918812 cpu 0 usage 0
> arrays at 2238d894 and 2238dd6c
> from entry 78 to entry 35
> loop cnt: 3395 3626 -> 231
> read cnt: 57670 61592 -> 3922
> loop rate = 231/430 = 0
> rcode = 100 - (100*0/2550) = 100
> 99/100 busy, from 43 sec ago
>
> sho ip bgp sum does not show any FIB programming
>
> We have fewer than 10 routes in so we're definitely not exceeding any
> limits.
>
> We have icmp redirects disabled
>
>
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp

Re: FCX High CPU Diagnostics [ In reply to ]

mmonka at netzquadrat

Apr 4, 2019, 4:40 AM

Post #10 of 11 (3974 views)

Permalink

Hi,

we had the same problem with CER 2024. Querying BGP Stats via SNMP is quite
heavy for the box.

Best Regards
Markus

On Wed, Apr 3, 2019 at 10:55 PM Derek Maxwell <
derek.maxwell@chosentechgroup.com> wrote:

> That may have actually done the trick. We aren't using any snmp tools on
> this box, but I added no snmp-server to the config and that brought cpu
> back down to 1%. I'll keep watching and see where it lands.
>
> If there are any other thoughts, feel free to send them over. I appreciate
> any input.
>
> -----Original Message-----
> From: Jörg Kost <jk@ip-clear.de>
> Sent: Wednesday, April 3, 2019 1:12 PM
> To: Derek Maxwell <derek.maxwell@chosentechgroup.com>
> Cc: foundry-nsp <foundry-nsp@puck.nether.net>
> Subject: Re: [f-nsp] FCX High CPU Diagnostics
>
> Can you eliminate any SNMP tool, that might spin the CPU around?
>
> On 3 Apr 2019, at 21:49, Derek Maxwell wrote:
>
> > Output of dm cpu:
> >
> > idle pid 0 id 536918812 cpu 0 usage 0
> > main pid 1 id 536918812 cpu 0 usage 0
> > flash pid 0 id 536918812 cpu 0 usage 0
> > arrays at 2238d894 and 2238dd6c
> > from entry 78 to entry 35
> > loop cnt: 3395 3626 -> 231
> > read cnt: 57670 61592 -> 3922
> > loop rate = 231/430 = 0
> > rcode = 100 - (100*0/2550) = 100
> > 99/100 busy, from 43 sec ago
> >
> > sho ip bgp sum does not show any FIB programming
> >
> > We have fewer than 10 routes in so we're definitely not exceeding any
> > limits.
> >
> > We have icmp redirects disabled
> >
> >
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp@puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

Re: FCX High CPU Diagnostics [ In reply to ]

derek.maxwell at chosentechgroup

Apr 4, 2019, 11:09 AM

Post #11 of 11 (3974 views)

Permalink

What’s interesting is we weren’t using SNMP on this box at all. However, disabling it seemed to have an immediate impact.

We’ll see if it comes back and that was just a coincidence. That box is being replaced by a XMR either tonight or this weekend anyway. Hopefully the XMR is robust and problem free.

I appreciate everyone’s help and guidance. It’s great having some support from the community.

From: Markus Monka <mmonka@netzquadrat.de>
Date: Thursday, April 4, 2019 at 4:40 AM
To: Derek Maxwell <derek.maxwell@chosentechgroup.com>
Cc: Jörg Kost <jk@ip-clear.de>, foundry-nsp <foundry-nsp@puck.nether.net>
Subject: Re: [f-nsp] FCX High CPU Diagnostics

Hi,

we had the same problem with CER 2024. Querying BGP Stats via SNMP is quite heavy for the box.

Best Regards
Markus

On Wed, Apr 3, 2019 at 10:55 PM Derek Maxwell <derek.maxwell@chosentechgroup.com<mailto:derek.maxwell@chosentechgroup.com>> wrote:
That may have actually done the trick. We aren't using any snmp tools on this box, but I added no snmp-server to the config and that brought cpu back down to 1%. I'll keep watching and see where it lands.

If there are any other thoughts, feel free to send them over. I appreciate any input.

-----Original Message-----
From: Jörg Kost <jk@ip-clear.de<mailto:jk@ip-clear.de>>
Sent: Wednesday, April 3, 2019 1:12 PM
To: Derek Maxwell <derek.maxwell@chosentechgroup.com<mailto:derek.maxwell@chosentechgroup.com>>
Cc: foundry-nsp <foundry-nsp@puck.nether.net<mailto:foundry-nsp@puck.nether.net>>
Subject: Re: [f-nsp] FCX High CPU Diagnostics

Can you eliminate any SNMP tool, that might spin the CPU around?

On 3 Apr 2019, at 21:49, Derek Maxwell wrote:

> Output of dm cpu:
>
> idle pid 0 id 536918812 cpu 0 usage 0
> main pid 1 id 536918812 cpu 0 usage 0
> flash pid 0 id 536918812 cpu 0 usage 0
> arrays at 2238d894 and 2238dd6c
> from entry 78 to entry 35
> loop cnt: 3395 3626 -> 231
> read cnt: 57670 61592 -> 3922
> loop rate = 231/430 = 0
> rcode = 100 - (100*0/2550) = 100
> 99/100 busy, from 43 sec ago
>
> sho ip bgp sum does not show any FIB programming
>
> We have fewer than 10 routes in so we're definitely not exceeding any
> limits.
>
> We have icmp redirects disabled
>
>
_______________________________________________
foundry-nsp mailing list
foundry-nsp@puck.nether.net<mailto:foundry-nsp@puck.nether.net>
http://puck.nether.net/mailman/listinfo/foundry-nsp