Mailing List Archive

Whats happens when TCAM is full on 7600/RSP720RSP-3CXL?
Hi,

I'm currently using a 7606 (RSP720RSP-3CXL) and taking in full BGP on v4
and v6. Obviously it the TCAM is almost full and the box needs to be
replaced.

But a have a couple of questions.

I have been hearing different scenario of what would happen when the
TCAM is full:
1. The whole thing goes into software routing mode for all routes which
causes 100% CPU and resulting and unusable box
2. New route entries will just get dropped, current entries just stay in
TCAM
3. New route entries will be software routed, but entries that are
already in TCAM will be hardware routed. You won't notice much impact in
the beginning.

What is true?

The only reason that our 7606 needs to be replaces it because of the
TCAM. It doesn't do much traffic, like 3Gbps upstream. Only BGP/OSPF.
And not many ports, 8 x 10Gb fiber + 30 x 1Gb copper (local servers).

We will probably go for the ASR9006. But I would like to use it like I'm
using the 7600 now, as a router/switch. I have been reading that you
need to make some uncommon config to create Ethernet VLAN/Trunk
interfaces and ports, as this is not commonly not done with this router.
But is this good practice? Will it be fine once I fingered it out?

Last question. Can I take a full BGP feed on both v4 and v6 with a
A9K-RSP440-TR? Or do I need the -SE?

Chiel



Bellow are some output of our current 7600:

#show mls cef maximum-route
 IPv4 + MPLS         - 832k (default)
 IPv6                - 90k
 IP Multicast        - 1k

#show mls cef su
Total routes:                     915422
    IPv4 unicast routes:          822144
    IPv4 Multicast routes:        8
    MPLS routes:                  2050
    IPv6 unicast routes:          91220
    IPv6 multicast routes:        3
    EoM routes:                   0

#show mls cef exception status
Current IPv4 FIB exception state = FALSE
Current IPv6 FIB exception state = FALSE
Current MPLS FIB exception state = FALSE

#show platform hardware capacity forwarding
L3 Forwarding Resources
 Module              FIB TCAM usage: Total        Used     %Used
   2                     72 bits (IPv4, MPLS, EoM)      851968
824115     97%
                        144 bits (IP mcast, IPv6)       98304 91198     93%

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
Hi,

at least for Sup720-3B(XL) and Sup-2T it results in number 1 for the
family that hit the limit.

So in most cases it will look that way:
#show mls cef exception status
Current IPv4 FIB exception state = TRUE
Current IPv6 FIB exception state = FALSE
Current MPLS FIB exception state = FALSE

And yes, the box will drop down to a few MBit of Traffic.

kind regards
Rolf

> Hi,
>
> I'm currently using a 7606 (RSP720RSP-3CXL) and taking in full BGP on v4
> and v6. Obviously it the TCAM is almost full and the box needs to be
> replaced.
>
> But a have a couple of questions.
>
> I have been hearing different scenario of what would happen when the
> TCAM is full:
> 1. The whole thing goes into software routing mode for all routes which
> causes 100% CPU and resulting and unusable box
> 2. New route entries will just get dropped, current entries just stay in
> TCAM
> 3. New route entries will be software routed, but entries that are
> already in TCAM will be hardware routed. You won't notice much impact in
> the beginning.
>
> What is true?
>
> The only reason that our 7606 needs to be replaces it because of the
> TCAM. It doesn't do much traffic, like 3Gbps upstream. Only BGP/OSPF.
> And not many ports, 8 x 10Gb fiber + 30 x 1Gb copper (local servers).
>
> We will probably go for the ASR9006. But I would like to use it like I'm
> using the 7600 now, as a router/switch. I have been reading that you
> need to make some uncommon config to create Ethernet VLAN/Trunk
> interfaces and ports, as this is not commonly not done with this router.
> But is this good practice? Will it be fine once I fingered it out?
>
> Last question. Can I take a full BGP feed on both v4 and v6 with a
> A9K-RSP440-TR? Or do I need the -SE?
>
> Chiel
>
>
>
> Bellow are some output of our current 7600:
>
> #show mls cef maximum-route
>  IPv4 + MPLS         - 832k (default)
>  IPv6                - 90k
>  IP Multicast        - 1k
>
> #show mls cef su
> Total routes:                     915422
>     IPv4 unicast routes:          822144
>     IPv4 Multicast routes:        8
>     MPLS routes:                  2050
>     IPv6 unicast routes:          91220
>     IPv6 multicast routes:        3
>     EoM routes:                   0
>
> #show mls cef exception status
> Current IPv4 FIB exception state = FALSE
> Current IPv6 FIB exception state = FALSE
> Current MPLS FIB exception state = FALSE
>
> #show platform hardware capacity forwarding
> L3 Forwarding Resources
>  Module              FIB TCAM usage: Total       
> Used     %Used
>    2                     72 bits (IPv4, MPLS,
> EoM)      851968
> 824115     97%
>                         144 bits (IP mcast,
> IPv6)       98304 91198     93%
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>


_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
Hi,

> On 18 Sep 2020, at 11:50, Rolf Hanßen <nsp@rhanssen.de> wrote:
>
> Hi,
>
> at least for Sup720-3B(XL) and Sup-2T it results in number 1 for the
> family that hit the limit.
>
> So in most cases it will look that way:
> #show mls cef exception status
> Current IPv4 FIB exception state = TRUE
> Current IPv6 FIB exception state = FALSE
> Current MPLS FIB exception state = FALSE
>
> And yes, the box will drop down to a few MBit of Traffic.

Performance will depend on where the traffic is going. If it’s
going to unaffected prefixes, it will be still hardware forwarded.

For traffic going to prefixes that failed to be installed in HW,
resulting performance will be similar to what you can get on those
pretty small, non-x86 CPUs.

There’s no easy way to clean up HW after failure to program it and
then somehow “split” prefixes between hardware only and software
only. With Sup2T and newer we have an option to check if ACL/QoS
and other policies will fit before even trying to program them
(so called ‘atomic’ commit), but that’s not the case for FIB.
The only way going forward is to reload the box after making sure
it won’t get into the same time after re-establishing routing
protocol adjacencies and getting prefixes again.

Set up 'maximum-prefixes' on your eBGP sessions to sensible value
depending on PFC/DFC models and you should be fine. Also, there’s
whole guide how to adjust MLS CEF scale numbers on 6500/7600:
https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-switches/117712-problemsolution-cat6500-00.html

--
?ukasz Bromirski
CCIE R&S/SP #15929, CCDE #2012::17, PGP Key ID: 0xFD077F6A

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
Hi,

On Fri, Sep 18, 2020 at 10:54:53AM +0200, chiel wrote:
> 3. New route entries will be software routed, but entries that are
> already in TCAM will be hardware routed. You won't notice much impact in
> the beginning.

"Route entries that have churn will end up in software" *and* "the
software path is heavily rate-limited".

So "for some targets you'll see massive packet loss".

Only a reload will get this fixed -> avoid this situation.

> What is true?
>
> The only reason that our 7606 needs to be replaces it because of the
> TCAM. It doesn't do much traffic, like 3Gbps upstream. Only BGP/OSPF.
> And not many ports, 8 x 10Gb fiber + 30 x 1Gb copper (local servers).
>
> We will probably go for the ASR9006. But I would like to use it like I'm
> using the 7600 now, as a router/switch. I have been reading that you
> need to make some uncommon config to create Ethernet VLAN/Trunk
> interfaces and ports, as this is not commonly not done with this router.
> But is this good practice? Will it be fine once I fingered it out?

If spanning tree is involved, the ASR9k gets clunky.

Besides that, it does routing/switching on VLANs and trunks nicely
(though the config is quite different to "more switchy" platforms
like the 6500/7600).

> Last question. Can I take a full BGP feed on both v4 and v6 with a
> A9K-RSP440-TR? Or do I need the -SE?

The RSP440 will cope nicely. It is end-of-life, though.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
Hi,

On Fri, Sep 18, 2020 at 12:44:55PM +0200, ??ukasz Bromirski wrote:
> For traffic going to prefixes that failed to be installed in HW,
> resulting performance will be similar to what you can get on those
> pretty small, non-x86 CPUs.

Much worse, actually, as the control-plane limiters hit hard - so you
do not get "full software-switched performance, with 100% CPU load"
but "CPU load is mostly normal, everything looks normal, some targets
have 80% packet loss".

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
Hey,

> So in most cases it will look that way:
> #show mls cef exception status
> Current IPv4 FIB exception state = TRUE
> Current IPv6 FIB exception state = FALSE
> Current MPLS FIB exception state = FALSE
>
> And yes, the box will drop down to a few MBit of Traffic.

Not only that, but there are three possible configurable actions for
exception state, freeze (default), reset and recover. CTAC didn't know
what recovery does. Freeze means no updates are going to HW, so
understanding that it just affects prefixes not fitting HW is
incorrect, if label gets reprogrammed in software, HW retains old
information and you break your VPN security promise.

The correct configuration has 'reset' manually configured and box will
reload in loop until recovered. I.e. don't let it happen.

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
On 18-09-2020 12:12, Mikael Abrahamsson wrote:
> My advice is "don't let that happen".

Thanks for the advice and feedback all!

I will set "maximum-prefix" on my sessions, and stop receiving full v6
table and just make a default route for v6. Then I can give more space
to v4 (will require reboot) until I get new routers in.

Is it ok to set "maximum-prefix 0" on my v6 session? Or is it better to
make a prefix list like " ipv6 prefix-list IPV6-IN seq 10 deny ::/0 le
128" top stop receiving routes from my upstream?

Chiel
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
18.09.2020 15:54, chiel wrote:

> I'm currently using a 7606 (RSP720RSP-3CXL) and taking in full BGP on v4 and v6.
> Obviously it the TCAM is almost full and the box needs to be replaced.

I use same router with two copies of full BGP on v4 and v6.
There is a way to mitigate the problem without replacing the box.

The following configuration places a filter on routes being installed
from BGP RIB to TCAM (FIB) limiting its usage by length of AS-PATH,
so traffic to very distant networks goes via default route.

This way, "show platform hardware capacity" shows:

Module FIB TCAM usage: Total Used %Used
5 72 bits (IPv4, MPLS, EoM) 884736 422247 48%
144 bits (IP mcast, IPv6) 81920 8412 10%
Works fine for me.

mls cef maximum-routes ip 850
mls cef maximum-routes mpls 1
router bgp NNNN
address-family ipv4
table-map BGP2FIB filter
address-family ipv6
table-map BGP6FIB filter
!
route-map BGP2FIB permit 10
match as-path 400
!
route-map BGP2FIB permit 20
match ip address prefix-list SELECTED-SET
!
route-map BGP6FIB permit 10
match as-path 500
!
ip as-path access-list 400 deny ^.+_.+_.+_.+
ip as-path access-list 400 deny ^.+_65535$
ip as-path access-list 400 permit ^.*
ip as-path access-list 500 deny ^.+_.+_.+
ip as-path access-list 500 deny ^.+_65535$
ip as-path access-list 500 permit ^.*

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
> Whats happens when TCAM is full on 7600/RSP720RSP-3CXL?

2016.


> Last question. Can I take a full BGP feed on both v4 and v6 with a
> A9K-RSP440-TR? Or do I need the -SE?

More seriously - the 'TR' scale and 'SE' scale don't differ in TCAM,
only in their installed RAM. This may matter to you, but for most
'border edge devices' it is unlikely to impact your DFZ survival rate.

The only issue now is (as gert alludes to) RSP-440 are somewhat old, as
is the 9006 chassis. You can shim RSP-880-RL-TR/SE ('RL' for "rate
limited", to exactly RSP-440 perf) into a 9006 and it was available at
the same list price, as well as capable of running 64-bit XR.

The 9906 is a chassis with longer legs, if you can get one. I would
definitely find the newest 'BRKARC-2003.pdf' that you can. There's
usually at least one presented each year for each of Cisco Live! US &
Europe.

Here's one from this year:

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKARC-2003.pdf


Good luck!

--
Tom
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
Hi,

On Mon, Sep 21, 2020 at 10:14:40PM +0100, Tom Hill wrote:
> The only issue now is (as gert alludes to) RSP-440 are somewhat old, as
> is the 9006 chassis.

It's not "somewhat old", Cisco has explicitly declared the RSP-440
end-of-life. The ASR-9006 chassis is still supported.

https://www.cisco.com/c/en/us/products/collateral/routers/asr-9000-series-aggregation-services-routers/eos-eol-notice-c51-737819.html

the RSP-440 stopped selling in 2017, and it's formally no longer supported
beyond IOS XR 6.4.

6.5.3 works nicely (*ahem*), but is officially "never tested on RSP440,
never bugfixed, we do not care".


So - if you know exactly what you are getting yourself into, getting a
used ASR-9006 + RSP-440 + Typhoon LCs will be a nice bargain, but you
won't get a service contract for it, and you *will* run into "ah, no,
*that* feature is not implemented on this linecard..." issues.

OTOH, for the original requirements "up to 3 Gbit/s", getting a box that
uses 1000+ Watts *and* needs so much space *and* is end-of-life might
not be a good choice... in Cisco land, there's the ASR9001 (nice box,
though the MPAs all carry the "we do not want to sell them" price tag)
or the ASR1000 as alternative (though I'd never buy one), or you look
into Juniper land for a MX204. The MX204 is really like "the box".

Given the IOS XR is sufficiently different from IOS that you need to
invest in training anyway, have a close look at the MX204.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de
Re: Whats happens when TCAM is full on 7600/RSP720RSP-3CXL? [ In reply to ]
On 22/09/2020 07:10, Gert Doering wrote:
> It's not "somewhat old", Cisco has explicitly declared the RSP-440
> end-of-life.

In colloquial British English use, these things mean the same thing :p


> So - if you know exactly what you are getting yourself into, getting a
> used ASR-9006 + RSP-440 + Typhoon LCs will be a nice bargain, but you
> won't get a service contract for it, and you *will* run into "ah, no,
> *that* feature is not implemented on this linecard..." issues.
>
> OTOH, for the original requirements "up to 3 Gbit/s", getting a box that
> uses 1000+ Watts *and* needs so much space *and* is end-of-life might
> not be a good choice... in Cisco land, there's the ASR9001 (nice box,
> though the MPAs all carry the "we do not want to sell them" price tag)
> or the ASR1000 as alternative (though I'd never buy one), or you look
> into Juniper land for a MX204. The MX204 is really like "the box".
>
> Given the IOS XR is sufficiently different from IOS that you need to
> invest in training anyway, have a close look at the MX204.


Good advice, to be honest. I made no determination on the number of
ports required, but if you don't need port density (or even if you do!)
there may well be better options than a huge, power-hungry chassis, and
those should be explored - Gert's advice here is good.

I suspect the OP isn't entirely aware that the 9000 is a 'fully
distributed' platform, unlikely the Cat6500/7600 platforms, where the
DFCs were always optional, and my oh my does it cost.

--
Tom
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/