Mailing List Archive

BBN Planet appears to have helped make this happen ;-(.

Top 20 Added Classful Routes from 23Dec96 to 30Dec96
591 AS1 BBN Planet backbone
248 AS200 BBN Planet Western Region
239 AS560 BBN Planet, New England Region (N
[...]
Top 20 Added Classles Routes from 23Dec96 to 30Dec96
167 AS1 BBN Planet backbone
156 AS200 BBN Planet Western Region
46 AS279 SURAnet Southern AS
[...]

Config error ? or something else ?

--Tony

Paul Ferguson <pferguso@cisco.com> writes:
* Looks like the 45k mark was reached:
*
* Table History
*
*
* Date Prefixes
* 231296 43352
* 241296 44089
* 251296 43735
* 261296 43391
* 271296 43093
* 281296 43912
* 291296 45352
* 301296 45532
*
* Happy New Year on the Internet,
*
* - paul
*
* ref: http://www.employees.org:80/~tbates/cidr-report.html
*
*
- - - - - - - - - - - - - - - - -

>
> Looks like the 45k mark was reached:
>
> Table History
>
>
> Date Prefixes
> 231296 43352
> 241296 44089
> 251296 43735
> 261296 43391
> 271296 43093
> 281296 43912
> 291296 45352
> 301296 45532
>
> Happy New Year on the Internet,
>
> - paul
>
> ref: http://www.employees.org:80/~tbates/cidr-report.html
>
>

Been there for a while...

sandbox>sh ip bgp sum
BGP table version is 1305923, main routing table version 1305923
46037 network entries (231928/267168 paths) using 13185156 bytes of memory


--
--bill
- - - - - - - - - - - - - - - - -

Looks like the 45k mark was reached:

Folks with 7000's and SSE's should start monitoring their memory
utilization via "show sse summary".

Tony
- - - - - - - - - - - - - - - - -

In message <3.0.32.19961230154408.00691d90@lint.cisco.com>, Paul Ferguson write
s:
>Looks like the 45k mark was reached:
>
>Table History
>
>
>Date Prefixes
>231296 43352
>241296 44089
>251296 43735
>261296 43391
>271296 43093
>281296 43912
>291296 45352
>301296 45532

I would like to note, that at 45,000+ prefixes even with
prefix length filters dropping that down to 37,000 prefixes
a 16Meg cisco router can no longer hold a reasonable table.
It might be ok if people ran flap dampening at their boarders,
but with several flaps per second per provider, and the extra
memory and cpu used by dampening and filter-lists,
those old 68030s can't cut it.

For those people that still have "full routes" going into
2500 series routers, i.e. smaller dual attached providers, you'd
better keep an eye out for your BGP sessions to start randomly
dropping as you run out of memory. (Oh and if this happens enough
Sprint and Digex will dampen your prefixes, and possible others...)

---
Jeremy Porter, Freeside Communications, Inc. jerry@fc.net
PO BOX 80315 Austin, Tx 78708 | 1-800-968-8750 | 512-458-9810
http://www.fc.net
- - - - - - - - - - - - - - - - -

Hmm, it's not news for us. 45K can hold core routing only as
inter-back-bone router, not more.

But why, why this crasy CISCO could not predict future when
they designed 45K routers? It was not difficult for them
develop this box to cary 64 or 128MB RAM.

>
> Looks like the 45k mark was reached:
>
> Folks with 7000's and SSE's should start monitoring their memory
> utilization via "show sse summary".
>
> Tony
>

---
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
(+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager)
(+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
- - - - - - - - - - - - - - - - -

Hmm, it's not news for us. 45K can hold core routing only as
inter-back-bone router, not more.

But why, why this crasy CISCO could not predict future when
they designed 45K routers? It was not difficult for them
develop this box to cary 64 or 128MB RAM.

> Looks like the 45k mark was reached:
>
> Folks with 7000's and SSE's should start monitoring their memory
> utilization via "show sse summary".

There's a couple of comments here:

First, 45k is not the limit. More like 60k. You'll pardon me for being
cautious.

The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses
for its high speed forwarding table. You don't want to pay for 64Mbytes of
SRAM. ;-)

When cisco's engineers designed the SSE, we knew very well what was
happening. We expected to be given the opportunity to produce subsequent
hardware which implemented the SSE in an ASIC. If, by that time, CIDR
hadn't killed off the exponential growth, we would have expanded the
address space. Unfortunately, cisco management decided that the SSE ASIC
should not be implemented (a mistake which, to my knowledge, cisco has not
corrected). Thus, the 7500 exists without an SSE.

Tony
- - - - - - - - - - - - - - - - -

not to make this too cisco specific, but...

the number of entries in the forwarding cache on the
sse is generally more than the number of routes in
the bgp rib (because of the way the cache handles
more-specifics of over-lapping aggregates). so in
addition to the raw number of routes in the rib, the
efficiency (and scope) of aggregation are also
important data points

now a question. what does an sse do when its cache
fills? it used to(*) bring down the whole sse, which
doesn't really make much sense given that it's a
cache and therefore it's normal operation for it to
be incomplete. anyone know an answer to that one?

/jws

(*) -- "used to" implies that it happened before,
which it did. but that was largely due to the
ineffeciency of the data structures, and the problem
was solved such that 2 years (and counting) was added
to its life

>
> Hmm, it's not news for us. 45K can hold core routing only as
> inter-back-bone router, not more.
>
> But why, why this crasy CISCO could not predict future when
> they designed 45K routers? It was not difficult for them
> develop this box to cary 64 or 128MB RAM.
>
> > Looks like the 45k mark was reached:
> >
> > Folks with 7000's and SSE's should start monitoring their memory
> > utilization via "show sse summary".
>
> There's a couple of comments here:
>
> First, 45k is not the limit. More like 60k. You'll pardon me for being
> cautious.
>
> The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses
> for its high speed forwarding table. You don't want to pay for 64Mbytes of
> SRAM. ;-)
>
> When cisco's engineers designed the SSE, we knew very well what was
> happening. We expected to be given the opportunity to produce subsequent
> hardware which implemented the SSE in an ASIC. If, by that time, CIDR
> hadn't killed off the exponential growth, we would have expanded the
> address space. Unfortunately, cisco management decided that the SSE ASIC
> should not be implemented (a mistake which, to my knowledge, cisco has not
> corrected). Thus, the 7500 exists without an SSE.
>
> Tony
- - - - - - - - - - - - - - - - -

: BBN Planet appears to have helped make this happen ;-(.

Greetings and Happy New Year.

As you all noticed, BBN Planet had a bit of routing announcement
incontinence last weekend and this week. This was due to
massive configuration error [.on my part- so please to toss
the rotten tomatoes my way if you feel so inclined.]

The changes were massive enough that it was hard to return
to the previous state. [.Which, as we all know, wasn't very good
in the first place.] In any case, the route table is back to
approximately what it was before I touched the routers, and over
the next two weeks or so I will be clearing out more leakage-
which was my orginal intent. :/

Hopefully, BBN will be off the top of the criminals list by the
end of the month, at least. Any more changes to our outbound
policy will be excruciatingly examined before applying. I
apologize for the painful errors in the previously applied
policy.

Best wishes for a great year,

-Jennifer "Kobi" Roberts
BBN Planet Network Engineering


- - - - - - - - - - - - - - - - -

First, Happy New Year.

2'th, sorry for my mistake - I just meant 45K as Cisco 4500... Through
it have not changed issue seriously -:)

3'nd.. Do you mean VIP2 idea is worst then SSE idea?

>
> Hmm, it's not news for us. 45K can hold core routing only as
> inter-back-bone router, not more.
>
> But why, why this crasy CISCO could not predict future when
> they designed 45K routers? It was not difficult for them
> develop this box to cary 64 or 128MB RAM.
>
> > Looks like the 45k mark was reached:
> >
> > Folks with 7000's and SSE's should start monitoring their memory
> > utilization via "show sse summary".
>
> There's a couple of comments here:
>
> First, 45k is not the limit. More like 60k. You'll pardon me for being
> cautious.
>
> The limitation is not DRAM. It's the 64k words of SRAM that the SSE uses
> for its high speed forwarding table. You don't want to pay for 64Mbytes of
> SRAM. ;-)
>
> When cisco's engineers designed the SSE, we knew very well what was
> happening. We expected to be given the opportunity to produce subsequent
> hardware which implemented the SSE in an ASIC. If, by that time, CIDR
> hadn't killed off the exponential growth, we would have expanded the
> address space. Unfortunately, cisco management decided that the SSE ASIC
> should not be implemented (a mistake which, to my knowledge, cisco has not
> corrected). Thus, the 7500 exists without an SSE.
>
> Tony
>

---
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
(+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager)
(+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
- - - - - - - - - - - - - - - - -

BTW, really all data incorporated from BGP protocl for
3 - 8 core routings do not need more than 32MB RAM; the fact CISCO
need more than 32MB ram for core routing is due to inefficient data
structures... Exactly it's because they (I think) have solved some
other task (not minimize memory, but CPU and development time).

What is really 100K routes?

100K * (
8 bytes - address + maska
4 - pointer to interface
4 - pointer to ASPATH (ASPATHES are really shared by many
different routes; You have not 1000,000 different AS
PATHES at all)
8 - accounting
8 - some other,

) - it's about 32 bytes in this example.

Of cource, it's not true - really we can expect about 64 bytes/route.

64 bytes * 100K = 6 Mb RAM. I can propose it's nessesary 128bytes - it'll be
12MB RAM.

But anyway, if somebody designes specially INTERNET CORE router,
he can hold any core routing tables in 16 MB RAM, not more. There is
a lot of different ways to minimise memory usage.

We are far from the theoretical memory leak; the more serious problem
may be efficiensy etc..., but there is CISCO FUTIONS and other ideas
to hold CPU usage in acceptable range.

I am shure we'll be the witnesses of the such events as it have been last
year when CISCO's engeneers have fixed serious BGP problem in 1 or 2
nights. Through this way (to solve hardware problems by software)
have it's limits.



---
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
(+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager)
(+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
- - - - - - - - - - - - - - - - -

tli or pferguso might be able to shed some light on the actual numbers,
assuming he has the time and interest :)

cisco might consider those numbers a trade secret, so I'll decline. I'll
simply point out that yours truly worked quite hard at conserving memory.
As you correctly point out, there is indeed the inevitable time space
tradeoff, and so some memory inefficiency is inevitable as there's only a
finite amount of time (always insufficient, natch ;-).

You might also recall that there's forwarding table, the cache, and then
the SSE data structures. BGP is very efficient as a whole. The problem is
that it's just a big messy problem.

Tony



- - - - - - - - - - - - - - - - -

3'nd.. Do you mean VIP2 idea is worst then SSE idea?

The VIP2 is a fine idea. However, it's again using conventional
microprocessor technology for forwarding. And unfortunately, it needs to
be fairly high end processor technology to sustain the rates. The problem
is that the traffic demand far exceeds the growth rate of processor
forwarding.

So the problem is that as we approach higher speed (post-7500) interfaces,
you're forced into more specialized hardware. At this point, not having an
SSE ASIC becomes somewhat silly. The rest of the board is ASICs, which in
volume end up cheaper than processors...

Tony
- - - - - - - - - - - - - - - - -

Tony Li wrote:

>So the problem is that as we approach higher speed (post-7500) interfaces,
>you're forced into more specialized hardware. At this point, not having an
>SSE ASIC becomes somewhat silly. The rest of the board is ASICs, which in
>volume end up cheaper than processors...

There's not enough volume in high-end box market to get to the break-even
point on the price-performance. Providing that there's a way to get
a bunch of cheapo CPUs to do the job. And don't forget the DRAM vs SRAM
issue. I did the arithmetic.

--vadim
- - - - - - - - - - - - - - - - -

There's not enough volume in high-end box market to get to the break-even
point on the price-performance.

That's not clear.

Providing that there's a way to get a bunch of cheapo CPUs to do the
job.

True, but there's a complexity hit (which translates into a reliability hit
and a development lag) from this approach.

And don't forget the DRAM vs SRAM issue. I did the arithmetic.

Fortunately, there are new memory technologies today that are more
interesting than those when we started work on the SSE. And ASIC
technology helps you further here...

Tony


- - - - - - - - - - - - - - - - -

On Mon, 30 Dec 1996 16:26:23 -0800 (PST)
Tony Li <tli@jnx.com> alleged:

>
> Looks like the 45k mark was reached:
>
> Folks with 7000's and SSE's should start monitoring their memory
> utilization via "show sse summary".
>
Nah, they should be trading them in for Pentium-Pro's and running BSD-Unix
with gated :-)

Mass Cisco Car boot sale -> Film at 23:00 :-)

Regards,
Neil.
--
Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C
neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor)
Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>

- - - - - - - - - - - - - - - - -

At 10:35 AM 1/3/97 -0500, Avi Freedman wrote:

>
>Well, it's also somewhat more complicated to distribute the tasks in a router
>to multiple CPUs.
>
>There are plenty of ways to break Bays by accidentally doing that distribution
>wrong.
>
>
>Avi
>

Hmmm. There are some who would contend that there are plenty of ways to break
Bays by accidentally doing something right. ;)

mike v

- - - - - - - - - - - - - - - - -

Tony wrote:

> Providing that there's a way to get a bunch of cheapo CPUs to do the
> job.

>True, but there's a complexity hit (which translates into a reliability hit
>and a development lag) from this approach.

Well, i do not think that using off-the-shelf cheapo CPUs debugged by
somebody else and tested in millions of applications, instead of
single-purpose ASICs qualifies as a reliability hit or a development lag :)

--vadim
- - - - - - - - - - - - - - - - -

Well, i do not think that using off-the-shelf cheapo CPUs debugged by
somebody else and tested in millions of applications, instead of
single-purpose ASICs qualifies as a reliability hit or a development lag :)

Agreed that the part itself is correct. However, unlike you, the rest of
us tend to take perfect parts and make mistakes putting them together. ;-)

Tony


- - - - - - - - - - - - - - - - -

> Well, i do not think that using off-the-shelf cheapo CPUs debugged by
> somebody else and tested in millions of applications, instead of
> single-purpose ASICs qualifies as a reliability hit or a development lag :)

>Agreed that the part itself is correct. However, unlike you, the rest of
>us tend to take perfect parts and make mistakes putting them together. ;-)

Ah, i do a lot of mistakes. Anyway, it's much easier to put together
debugged parts, than put together new silicon. I've had more than
enough of that at nCUBE.

--vadim
- - - - - - - - - - - - - - - - -

On Fri, 3 Jan 1997 alex@relcom.EU.net wrote:

|} Yes, no doubt. But I'd like to see router's vendors predicted future
|} better than in the past. We all know few serious mistakes CISCO did -
|} with the memory size in CS4500, with CS7000 and CS7010 routers; and I
|} am afraid to fail into new trap in future...

Alex - I'm sure there were the visionaries at Cisco with their heads in
the clouds who were telling the various marketing and engineering bodies
to build a box like the 7500 or BFR some time ago. However, proving the
market would be there and that there would actually be a customer base for
such a product is a little more difficult.

I believe, that only in the past 1-2yrs has ISP revenue actually accounted
for a good portion of Cisco's overall revenue. (Perhaps someone from
Cisco or someone with their financial statement can speak to this)
Building boxes that require a huge amount of R&D and won't sell more than
< 5,000 of isn't a very profitable business. Cisco should be thankful
that the overall majority of their ISP user base is rather understanding
when it comes to performance limitations, discovering and working around
software issues, etc. etc.


-jh-


- - - - - - - - - - - - - - - - -

Really, there is one important question (not the blame about IOS's
memory or so on) - does hardware vendors (just as CISCO etc) really predicts
the future?

There's a complicated answer. Yes, certain people at router vendors do
understand and can predict what is necessary. However, this does not imply
that the right thing happens. Please recall that most (all?) router
manufacturers are driven by the enterprise market, not the ISP market.

Memory constraints in the enterprise market are not an issue. Memory costs
(and even the cost of the extra SIMM socket) _are_ an issue, as they affect
the price that everyone on the street pays. This either cuts into
manufacturers profit margins or into their sales.

Thus, router manufacturers can indeed predict what's needed for the ISP
market. But history shows that they consistently make a business decision
to design hardware for the enterprise market.

It's easy to install 2 or 4 CPU into SUN ULTRA-2 or SGI SERVER
computers, but it's impossible to do it with routers. And so on.

In fact, it's not impossible. That's _exactly_ what you're doing when you
install a VIP.

But what we'll do with the routers next 2 years? Just now existing
CS7500 and CS4700 are 30% - 50% loaded by ROUTING (not switching but
routing) and there is not ways to scale them easily.

Again, getting the switching off of the CPU is key. Once that happens
consistently, then there is much more CPU for routing computation. 30-50%
for routing is fine once that happens. It just implies that we need to
scale up the processor to match the growth in _routing_ traffic.
Fortunately, that's _FAR_ lower than transit traffic, so I believe it's a
tractable problem. Just gotta stay on the processor curve.

Tony


- - - - - - - - - - - - - - - - -

>
> 3'nd.. Do you mean VIP2 idea is worst then SSE idea?
>
> The VIP2 is a fine idea. However, it's again using conventional
> microprocessor technology for forwarding. And unfortunately, it needs to
> be fairly high end processor technology to sustain the rates. The problem
> is that the traffic demand far exceeds the growth rate of processor
> forwarding.
>
> So the problem is that as we approach higher speed (post-7500) interfaces,
> you're forced into more specialized hardware. At this point, not having an
> SSE ASIC becomes somewhat silly. The rest of the board is ASICs, which in
> volume end up cheaper than processors...
>
> Tony
! Sorry, my crazy Win95 (I HATE MS!) drop me from TELNET 3 times when
! I did attempted to answer this from the home -:).
Really, there is one important question (not the blame about IOS's
memory or so on) - does hardware vendors (just as CISCO etc) really predicts
the future? When CISCO developed CS4500, they have decided 32MB RAM would be
enougph just as MIPS processor they established. Just now they (seems)
think it'll be enougph new MIPS and 128MB RAM. It's amazing but I can
easy upgrade my 1,000$ IBM PC up to 256MB RAM, but I can't do it for
my CISCO routers (about 10 - 30,000$) at all. It's easy to install
2 or 4 CPU into SUN ULTRA-2 or SGI SERVER computers, but it's impossible to do
it with routers. And so on.

What's about switching itself - I am not shure they (vendors) are on the wrong way.
Seems CISCO have well defined plan - they are moving toward from
1 CPU doing everything (CS2500 and CS4700) to (1 CPU for routing, few CPU for the
switching - VIP2) and to (1 CPU for the Routing, and many switches for the
switching - MPOATM, TAG Switching, etc) - they are trieing to build
mixed _routers + ATM_NETWORK_ backbone when routers would decide how to
forward some virtual streams (TCP connections etc) and switches would
forward the packets toward their destinations.

And this mean we'll have more troubles with the routers, not switches.
There is a lot of ways to increase switching capability (look on Stratocom,
for example); but the routers itself are far beyong. We can easyly increase
the CPU and Memory for the UNIX Servers; it's not difficult to build
server with 512MB RAM, 4x300 MHZ CPU, 10 PCI or SBUS cards; but
no one router can be expanded easyly. I wonder why it's so many discussion
there about DRAM/SRAM/etc..., there exist fast and nonblocking switches just
now; and moreover, you can install them in parallel because no one customer
need 200Mbit data links itself /this means you can use 4x100 mbit links
instead of 1x400Mbit withouth the lost of quality/. But what we'll do with
the routers next 2 years? Just now existing CS7500 and CS4700 are 30% - 50% loaded
by ROUTING (not switching but routing) and there is not ways to
scale them easily.


Or we'll go back to the UNIX boxes + GATED? Sometimes I guess this is true _:)



---
Aleksei Roudnev, Network Operations Center, Relcom, Moscow
(+7 095) 194-19-95 (Network Operations Center Hot Line),(+7 095) 239-10-10, N 13729 (pager)
(+7 095) 196-72-12 (Support), (+7 095) 194-33-28 (Fax)
- - - - - - - - - - - - - - - - -