In a message written on Mon, Oct 12, 2009 at 05:09:41PM -0700, Owen DeLong wrote:
> With IPv6, it probably won't be the ideal 1:1 ratio, but, it will come
> much closer. Even if the average drops to 1/2, you're
> talking about a 70,000 route table today, and, likely growth in the
> 250-300,000 route range over the next 5-10 years.
> CAM will probably scale faster than that.
Here's a presentation from 2007.
http://www.vaf.net/~vaf/apricot-plenary.pdf On page 13, you'll find a table. It starts with numbers in November
of 2006, and makes projections. The 5 year projections (Nov 2011)
have already been exceeded, in both IPv4 Internet Routes and Active
ASN's.
The problem isn't that we have 300,000 "global routes" on the
Internet (
http://www.cidr-report.org/as2.0/#General_Status), but
that there are other things that compete for TCAM space. It's that TCAM
must hold not only the global routes, but also:
- Internal routes. Your IGP routes, no-exported customer
deagregations, blackhole routes, etc.
- MPLS Labels, including:
- MPLS Traffic Engineering
- MPLS VPN Identifiers
- Virtual Routing Instances for Layer 3 VPN's.
- ARP Entries
- Multicast Routes
Unfortunately details are hard to come by as most of the folks who
see this in any significant way (e.g. global "tier 1" full service
ISP's) tend not to release too many specific numbers for competitive
reasons.
That said, even using some basic assumptions (some of which are in
the preso) those 300,000 global routes might have added to them:
300,000 global routes
150,000 internal routes
20,000 MPLS labels
200,000 VPN/VRF Routes
5,000 ARP Entries
20,000 Multicast Routes
--------
695,000 TCAM Entries Consumed
That's today, right now, in major ISP's routers. All the sudden
those "1 million route" core routers don't seem so large.
Keep in mind we've passed the 2006 projection in this report in 3
years, not 5. So we're growing faster than we expected. Add in
your 70,000 route IPv6 table, plus growth, and the 1 million route
routers are probably failing sometime in 2011.
Someone will likely pipe up, but Cisco has a 3 million route processor
now! (I believe that is the spec of the just announced PRP3, but
can't find a reference on Cisco's web site). Yes, that's a route
processor that can do the job, but in these high end boxes the TCAM
is distributed on the linecards. So upgrading from the 1 million
route TCAM core routers to the 3 Million route TCAM means upgrading
every linecard in each router you upgrade.
Ouch.
The picture I painted above is actually the rosy part of the picture.
Many of these backbones have older equipment in the core which can't
even do 1M routes. They use careful design and other techniques
to limit the number of entries particular boxes have to see.
> The problematic time scale is that time where we have to support dual
> stack for a majority of the network. That's what will
> really stress the CAM as the IPv6 table becomes meaningfully large
> (but not huge) and the IPv4 table cannot yet be
> retired.
While I think Verizon's move is somewhat premature, I can see why
they might be afraid of routing table growth. I think there is an
extremely high probability that given the growth of the table due
to primarily to IPv6 and the growth of MPLS VPN offerings, combined
with the current economic climate which has reduced the capital
available for upgrades that we will see several providers "hit the
wall" of various popular bits of equipment. I think some of the
engineering staff at various major providers has already realized
this as well.
We don't seem to have a technological solution. LISP has scaling
issues of its own, and would require swapping out a huge amount of
equipment. TCAM scaling is at best cost prohibitive, at worst not
possible due to the physical ram speed, and both are being improved at a
modest rate (the preso suggests 10% per year).
Worse, the problem is being made worse at an alarming rate. MPLS
VPN's are quicky replacing frame relay, ATM, and leased line circuits
adding MPLS lables and VPN/VRF routes to edge routers. Various
RIR's are pushing "PI for all" in IPv6 based on addressing availbility.
Some networks are actually finally using multicast for IPTV services,
generating much larger number of entries than the global multicast table
would otherwise indicate.
The next 5 years may bring internet instability problems and route
filtering on a scale we haven't seen since the early 90's.
--
Leo Bicknell - bicknell@ufp.org - CCIE 3440
PGP keys at
http://www.ufp.org/~bicknell/