Mailing List Archive: Re: High CPU utilization on Cisco 7206VXR seems to belimiting performance of higher-speed users

Frank Bulk <> wrote on Thursday, April 12, 2007 8:49 PM:

> We've had two complaints from 2 Mbps customers that they aren't
> getting their contracted bandwidth. I went to one of them and
> confirmed that it's mixed, getting only up to 1.5 Mbps at times.
> Half of our customers run at 128/128 kbps, another 40% at 1024/256
> kbps, and the remaining at 2048/384 kbps.
>
> We have a Cisco 7206VXR with an NPE400 with 491520K/32768K bytes of
> memory running c7200-is-mz.122-26.bin. The processor is running at
> about 60%, up from 40% a year ago. I believe that the CPU has
> something do with the performance.
>
[...]
>
> I followed the advice on Cisco's web pages on troubleshooting IP
> Input CPU load on Friday but nothing I tried seemed to make a
> difference.
>
> Our DSL customers come in on two OC3's and we have some FTTH customers
> coming in on Fa0/0. Our main Ethernet interface, Fa0/1, does have
> quite a few drops and flushes, but you can see the loads are low and
> cacti reports interface utilization of about 10 to 15 Mbps.
[...]
> A complete 'show interfaces switching' and 'show ip traffic', one
> after another, can be found zipped up in this file:
> http://www.mtcnet.net/~fbulk/show_interfaces_switching_ip_traffic.zip
>
> I also dropped the two ACLs we have on our Ethernet interface and it
> didn't make a difference.
>
> We have about 2013 active PPPoA connections and 35 PPPoE connections.

You are definitly switching too much traffic using process-switching.
There are several reasons for this:
- You are re-assembling on the router (not sure if any L2TP is
involved)?
- You are using per-user features which are not supported in CEF
- ...

We'd need to see the full config, more snapshots of "show int switching"
(best do a "clear counter", wait 5 mins and show the counters) and
multiple "show ip traffic" a few minutes apart to see which counter is
increasing. A "show ip interface virtual-access <x>" of a couple of
users could also provide hints, please also show the Radius profiles.

Feel free to send unicast..

oli

_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba

With Oli's most gracious assistance I was able to bring the Cisco 7206VXR
down from the 60-75% range to about 35%.

I discovered that when running a 'sh ip interface' that most of my
Virtual-Access interfaces were using TCP header compression Oli informed me
that 'ip tcp header compression' triggers process switching of the traffic.
Although it was not explicitly configured on my 7206VXR, our RADIUS server
was returning the "Van-Jacobson Header Compression" attribute (inherited
from our dialup config, and alluded to here:
http://puck.nether.net/pipermail/cisco-bba/2004-December/000348.html). I
had a colleague a lot smarter than me remove that attribute from the RADIUS
responses for requests coming from the 7206VXR. That brought things down to
45%, or a savings of 15 to 30%.

But Oli was still not satisfied and noticed in my config that I was using
'ip igmp join-group' rather than 'ip igmp static-group'. I changed those
and saved myself another 10%, bringing IP Input down to sub 5% levels.

Thanks, Oli!

Kind regards,

Frank

-----Original Message-----
From: cisco-bba-bounces@puck.nether.net
[mailto:cisco-bba-bounces@puck.nether.net] On Behalf Of Frank Bulk
Sent: Thursday, April 12, 2007 1:49 PM
To: cisco-bba@puck.nether.net
Subject: [cisco-bba] High CPU utilization on Cisco 7206VXR seems to
belimiting performance of higher-speed users

We've had two complaints from 2 Mbps customers that they aren't getting
their contracted bandwidth. I went to one of them and confirmed that it's
mixed, getting only up to 1.5 Mbps at times. Half of our customers run at
128/128 kbps, another 40% at 1024/256 kbps, and the remaining at 2048/384
kbps.

We have a Cisco 7206VXR with an NPE400 with 491520K/32768K bytes of memory
running c7200-is-mz.122-26.bin. The processor is running at about 60%, up
from 40% a year ago. I believe that the CPU has something do with the
performance.

========
Router#sh proc cpu | exc 0.00.*0.00
CPU utilization for five seconds: 61%/27%; one minute: 62%; five minutes:
60%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
3 15913764 57964253 274 0.16% 0.04% 0.04% 0 PPP auth
4 176627692 11860384 14892 0.00% 0.41% 0.43% 0 Check heaps
15 1228555796 497123198 2471 2.51% 3.14% 3.51% 0 ARP Input
16 67532280 9516840 7096 0.00% 0.14% 0.16% 0 HC Counter
Timer
22 417429972 77856792 5361 0.32% 0.75% 0.80% 0 Net
Background
40 14789027243609760351 0 23.24% 23.18% 23.30% 0 IP Input
41 12942320 4757230 2720 0.08% 0.05% 0.06% 0 CDP Protocol
49 53911324 50794855 1061 0.00% 0.07% 0.07% 0 IP Background
63 169456380 62895851 2694 0.16% 0.29% 0.31% 0 CEF process
94 309167136 7797068 39652 0.89% 0.79% 0.80% 0 Compute load
avg
103 676779760 45869622 14754 1.02% 3.72% 3.99% 0 PPPOE
discovery
113 210967168 765640753 275 0.16% 0.18% 0.19% 0 PPP manager
126 488852524 15943470 30661 4.37% 2.72% 2.79% 0 VTEMPLATE
Backgr
127 40 152 263 0.08% 0.03% 0.00% 2 Virtual Exec
Router#
========

I followed the advice on Cisco's web pages on troubleshooting IP Input CPU
load on Friday but nothing I tried seemed to make a difference.

Our DSL customers come in on two OC3's and we have some FTTH customers
coming in on Fa0/0. Our main Ethernet interface, Fa0/1, does have quite a
few drops and flushes, but you can see the loads are low and cacti reports
interface utilization of about 10 to 15 Mbps.
========
FastEthernet0/1 is up, line protocol is up
Hardware is i82543 (Livengood), address is 000d.6633.dc06 (bia
000d.6633.dc06)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 12/255, rxload 56/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 00:05:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/4135105/302954774 (size/max/drops/flushes); Total output
drops: 10256
^^^^^^^ ^^^^^^^^^ ^^^^^ ^^^^^^^
^^^^^
========
Router#show interfaces switching
<snipped out the FastEthernet0/0>
FastEthernet0/1
Throttle count 916697
Drops RP 4135723 SP 0
SPD Flushes Fast 303102861 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 58974374 Drops 0

Protocol Path Pkts In Chars In Pkts Out Chars Out
Other Process 18572355 1146224998 3926592 235595520
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
IP Process 3017811923 1959581225 2355715836 3994592527
Cache misses 0
Fast 4013202896 604980241 3965937813 3361756168
Auton/SSE 0 0 0 0
ARP Process 823012431 2140444865 38144362 2441239168
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
CDP Process 654814 270489385 655922 347750478
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
PPP over ATM Process 0 0 9 540
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
MSCP Process 0 0 31363715 2007277760
Cache misses 0
Fast 0 0 0 0
Auton/SSE 0 0 0 0
========
A complete 'show interfaces switching' and 'show ip traffic', one after
another, can be found zipped up in this file:
http://www.mtcnet.net/~fbulk/show_interfaces_switching_ip_traffic.zip

I also dropped the two ACLs we have on our Ethernet interface and it didn't
make a difference.

We have about 2013 active PPPoA connections and 35 PPPoE connections.

We haven't changed code or anything for 2+ years, before my time. We are
cutting some customers over from PPPoA coming in on an ATM interface to
PPPoE on Fa0/0 as we convert them to a FTTH installation, but that's perhaps
been 25 connections or so.

I would welcome any ideas anyone has. I would like to avoid upgrading to a
G1 or G2 if I could.

Kind regards,

Frank

_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba

_______________________________________________
cisco-bba mailing list
cisco-bba@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba