Mailing List Archive

The jury is still very much out on whether layering IP on
top of frame relay with frame relay switches providing the
principal long-haul switching fabric and routers making
decisions about which PVCs to direct IP datagrams into is
a reasonable, scalable approach.

There are several advantages to the approach, vis.
greater port density, especially in terms of
ports-per-dollar on most FR devices than on high-end
Ciscos; the (perhaps safe) bet that a medium-size user and
channel of routers might have more of an influence over
the development and engineering of an FR vendor's
Internet-oriented products than over Cisco; that an
organization with expenditure limits on bandwidth and
relatively long minimum term committment lengths on
cross-country circuits could do better traffic management,
particularly in the presence of multiple parallel DS1s,
than a similar setup using either inverse multiplexors or
load balancing; and the fact that using a mesh along these
lines is something that readily can interoperate with a
migration to clear-channel DS3s or other technologies, or
from a melting-down cross-country ethernet-over-ATM or
FR-over-ATM backbone.

As well, there is the probability that the users of the
model you're asking about have lots and lots of
points-of-presence used in an effort to reduce backhaul
costs, which get quite substantial as the numbers of
customers grow. Also, having lots of POPs interconnected
at varying bandwidths can make designing a topology that
is friendly to routers a bit more difficult in some ways.

The model generally also very neatly hides the underlying
topology from the IP layer, rendering traceroute
essentially useless as a means of diagnosing weird time
asymmetries and the like. There may be 13 physical hops
between A and B on a network using this model, but
traceroute might reveal only two. This has some obvious
advantages to the provider keen on hiding the design of
its network from "outsiders", or even from its own
customers.

Key to the decision is the gamble that the switching
ability of FR switches will compare with the switching
ability of high-end routers in the same environment. This
is not a gamble I am comfortable with, particularly as I
have very strong ideas of how well an SSE-equipped 7000
can switch packets in a backbone environment, and so far
have only early impressions about one variety of popular
FR switch in a similar environment.

Also key to the decision is a bet on how well the
FR-switch-based networks will scale beyond multiple DS1s
and DS3s, especially in comparison to routers.

There are substantial disadvantages, too.

In order to take advantage of the greater
port-density(-per-dollar) on FR switches right now, the
end user has to use FR, which is not always practical
or desirable.

The monitoring and management systems of all FR switching
products with which I am familiar are not
router-geek-friendly. PVCs can do strange things, as can
underlying circuits, which one really wants the
IP-switching layer to recognize and adapt to; at present
there is effectively no practical means to do this in
anything but a failure of a PVC.

There is a serious management nightmare in maintaining
anything approaching a full mesh in a large network, and
it worsens if one pushes the routing decisions right out
to the edges of a given network. Maintaining lots of PVCs
has many of the well-known disadvantages of maintaining
lots of standard iBGP neighbours or any other system in
which involves configuration length, complexity or both
approaching or equivalent to N**2.

There are a number of other deeply technical questions
about the model which remain the subject of occasionally
heated speculation, but the bottom line is that nobody
really knows how well it will work as this type of network
grows really big, or even as heavily trafficked as
SprintLink and InternetMCI.

Finally, pushing the envelope of technology is hard
work at all times; I believe a pragmatic approach is
to push only n things at any given time where n is
ideally the smallest possible number. Adding frame
relay switching technology into the mix of things that
backbone NSPs have to deal with already strikes
me as asking for trouble.

| PSI and UUNET have apparently been following this strategy with their
| backbones for some time. Netcom has also come on board.

Your information may be better than mine, but I gathered
that UUNET has just recently migrated over to this model
from the previous MFS-supplied technologies they had been using.

Frankly, neither netcom nor PSI does sufficient traffic
to learn anything interesting about how the model behaves
under stressful conditions.

Alternet's experience over the next few months will be enlightening.

Sean.

>
> In order to take advantage of the greater
> port-density(-per-dollar) on FR switches right now, the
> end user has to use FR, which is not always practical
> or desirable.
>

This is actually not true, at least one popular brand
of FR switch has a builtin PPP FRAD (I must say that
debugging the connections is still a medium nightmare).

I agree with most of the other points you make, you do
miss one other potential reason for using FR/ATM technology:
competing in the VPN market with the old players there
(Sprint et al).

Simon

At 05:47 PM 11/17/95 -0500, Sean Doran wrote:

>
>There are substantial disadvantages, too.
>
>In order to take advantage of the greater
>port-density(-per-dollar) on FR switches right now, the
>end user has to use FR, which is not always practical
>or desirable.
>

One of the reasons why end users find frame-relay undesireable is
that they cannot be assured that their provider is not grossly
oversubscribed on PVC-per-port density. When you buy a T1 private
line, you can be assured that you're not sharing it with 120 other
end-users. :-)

- paul

>When you buy a T1 private line, you can be assured that you're not
>sharing it with 120 other end-users. :-)

A T-1 private line to what? Do you think Sprint has sold
more than 30 T-1s, filling up their DS-3 backbone?

Do you think MCI has sold more than 30 T-1s, filling up
their DS-3 backbone?

You _are_ sharing bandwidth, unless 100% of your traffic
is to your provider.

>- paul


Ehud

--
Ehud Gavron (EG76)
gavron@Hearts.ACES.COM

> >There are substantial disadvantages, too.
> >
> >In order to take advantage of the greater
> >port-density(-per-dollar) on FR switches right now, the
> >end user has to use FR, which is not always practical
> >or desirable.
> >
>
> One of the reasons why end users find frame-relay undesireable is
> that they cannot be assured that their provider is not grossly
> oversubscribed on PVC-per-port density. When you buy a T1 private
> line, you can be assured that you're not sharing it with 120 other
> end-users. :-)

Nor can you know that with a Dedicated line, that the other side doesn't feed
into a 2501 into a collision-ridden Boca 10baseT hub, into a Welfleet bridging
over Frame-Relay-over-ATM at 1.5Mbps simplex speeds into an over-committed
4Mbps port speed on the other side.

i.e., the fact that your buying a leased line to connect and running HDLC
over it doesn't count for anything. When it comes right down to it, customers
can't really be reassured of anything.

But then, nobody expects the customer to realize this, right?

Dave

--
Dave Siegel President, RTD Systems & Networking, Inc.
(520)623-9663 Systems Consultant -- Unix, LANs, WANs, Cisco
dsiegel@rtd.com User Tracking & Acctg -- "Written by an ISP,
http://www.rtd.com/ for an ISP."

At 05:28 PM 11/17/95 -0700, Ehud Gavron wrote:

>>When you buy a T1 private line, you can be assured that you're not
>>sharing it with 120 other end-users. :-)
>
> A T-1 private line to what? Do you think Sprint has sold
> more than 30 T-1s, filling up their DS-3 backbone?
>
> Do you think MCI has sold more than 30 T-1s, filling up
> their DS-3 backbone?
>
> You _are_ sharing bandwidth, unless 100% of your traffic
> is to your provider.
>


I think you missed my point.

This is not exactly an issue of how your traffic, once routed
through your providers router, traverses their backbone. Rather,
its an issue of the ability (or inability, as it were) to sloppily
engineer capacity due to the nature of defining excessively
high ratios of virtual circuits-to-aggregate ingress bandwidth.

- paul

At 05:45 PM 11/17/95 -0700, Dave Siegel wrote:

>> One of the reasons why end users find frame-relay undesireable is
>> that they cannot be assured that their provider is not grossly
>> oversubscribed on PVC-per-port density. When you buy a T1 private
>> line, you can be assured that you're not sharing it with 120 other
>> end-users. :-)
>
>Nor can you know that with a Dedicated line, that the other side doesn't feed
>into a 2501 into a collision-ridden Boca 10baseT hub, into a Welfleet bridging
>over Frame-Relay-over-ATM at 1.5Mbps simplex speeds into an over-committed
>4Mbps port speed on the other side.
>
>i.e., the fact that your buying a leased line to connect and running HDLC
>over it doesn't count for anything. When it comes right down to it, customers
>can't really be reassured of anything.
>

Quite true. Appearances can be deceiving. :-)

I was just reiterating (what I thought to be) the obvious.

>But then, nobody expects the customer to realize this, right?
>
>Dave
>


Um, I won't touch this one.

- paul

In message <Pine.SUN.3.91.951117162036.28454M-100000@tigger.jvnc.net>, Gordon C
ook writes:
> People are starting to tell me that the way out of smashing into the
> brick wall at 60 miles and hour that the Links on the Blink discussion
> was worrying about a week ago, is to take cascade or other frame relay
> switches for the national service providers backbone topology and push
> routers out to spoke and wheel hubs at the perimiters of the back bone
> where traffic sent on the backbone at the data link layer (2) can pop out
> and into the appropriate router for layer 3 routing to the final destination.
>
> PSI and UUNET have apparently been following this strategy with their
> backbones for some time. Netcom has also come on board. Its advocates
> say that this combination can scale and avoid the 60 mile per hour brick
> wall crash. Are they right? And if so why aren't sprint and MCI running
> in this direction and away from purely router based networks as fast as
> they can?
>
> If they are why are they wrong? Where does this architecture fail? If
> it fails.
>
>
> ********************************************************************
> Gordon Cook, Editor & Publisher Subscript.: Individ-ascii $85
> The COOK Report on Internet Non Profit. $150
> 431 Greenway Ave, Ewing, NJ 08618 Small Corp & Gov't $200
> (609) 882-2572 Corporate $350
> Internet: cook@cookreport.com Corporate. Site Lic $650
> Newly expanded COOK Report Web Pages http://pobox.com/cook/
> ********************************************************************


Gordon,

I don't want to bash any equipment vendors or other service providers.
I suggest you take into consideration who is unstable and what
component or network element of their network is going unstable. The
brick wall is that a particular piece of equipment from a particular
vendor that a lot of service providers have made a large investment in
doesn't really perform all that well in the real world.

That doesn't mean that piece of equipment is terrible, it just has
some limits. It is actually a very nice piece of technology if you
can keep within its limits and if keeping within those limits meets
your needs. It also means some service providers may not have been as
aware of its limits as they should have been and got caught stepping
over the limits on their backbone. I can think of one service
provider using the same equipment that has managed to keep their
network within its limits and has kept a very stable network. The
technology we use has some very serious limits too.

wrt your question - There is no real disadvantage to going into a
level 2 cloud if either you are overprovisioned or congestion control
is properly taken into account. That means neither too much or too
little buffering if using tail drop. Doing something a little better
than tail drop helps. Shredding packets into cells and dropping cells
hurts big time. As long as you are not exceeding the capacity of your
router technology, going with a level 2 cloud has no real advantage
either. You can screw up in either case if you don't know what you
are doing.

We think there is better chance of working with IP router vendors to
get IP requirements met, particularly in the area of traffic
management since the technologies closer to the telcos tend to have
different ideas about traffic management than the IP world. Vendors
of the technologies closer to the telcos tend to be averse to
accommodating the dynamics of TCP traffic, to the point of being
hostile toward the idea (suggestions that the solution is to get rid
of TCP, etc).

In terms of capacity, router based backbone nodes are supporting
multiple DS3s. This is pushing FR already. If you need to go higher,
you need to go to OC-3 and then you are probably into ATM. The
problem here is that ATM traffic management available today is an
incredible botch. Some feel ABR will fix this. I remain skeptical of
that, but it may work. For now, you must still build a router based
network using ATM VBR without the "V", by setting PCR==SCR and traffic
shaping not to exceed that. Happily DS3 still has some life left in it.

One aspect of being a good service provider is understanding the
limits of the technology you are working with and understanding the
upcoming technology that you will be faced with. You must work with
vendors to make sure you don't exceed the capability of you existing
technology and make sure your vendors understand your needs and take
them into consideration in their next generation technology. One
aspect of being a good vendor is understanding the needs of your
customer and the limits of your own equipment and responding by
removing the limits before your customers have operational problems.

As far as FR, ATM, router networks or layer 2 clouds goes, different
service providers will choose different paths. I suggest you take a
look at who has a reputation for understanding the limitation of the
technology and for keeping their backbone stable and take what
everyone says with an appropriately sized grain of salt. Time will
tell who was right. I think reporting that one technology is clearly
better would be irresponsible unless you say better at what (equipment
supporting the technology is more mature, more cost effective,
theoretically scales to a higher limit, available products perform
well, under what conditions the technology or available equipment
performs well or poorly, etc).

Curtis

ps - I tried to be fair and hope to avoid any religious wars.

In message <199511180045.RAA12135@seagull.rtd.com>, Dave Siegel writes:
>
> Nor can you know that with a Dedicated line, that the other side doesn't feed
> into a 2501 into a collision-ridden Boca 10baseT hub, into a Welfleet bridgin
> g
> over Frame-Relay-over-ATM at 1.5Mbps simplex speeds into an over-committed
> 4Mbps port speed on the other side.
>
> i.e., the fact that your buying a leased line to connect and running HDLC
> over it doesn't count for anything. When it comes right down to it, customer
> s
> can't really be reassured of anything.


What are you talking about. You can go back and tell your provider
that you are losing one packet per 10^-5 and expect them to actually
do something to fix it with a leased line. You can't do that with FR
unless your PVC CIR is wire speed or you traffic shape to your CIR.

Who do you buy leased lines from? Nobody we buy from. If you are
talking about a tiny provider buying a T1 from a larger IP provider,
you might want to find another provider if that is your impression. I
think the discussion was building a backbone though.

Curtis

Curtis -

| The brick wall is that a particular piece of equipment from a particular
| vendor that a lot of service providers have made a large investment in
| doesn't really perform all that well in the real world.

Please allow me to mitigate your politics with a dose
of reality.

sl-dc-8.sprintlink.net, a now-fairly-old Cisco 7000 with one
of the first four 2MB SSP boards ever shipped outside
Cisco's doors has been observed to switch 125kpps through
several interfaces over a 15 minute period several times in
the past three weeks.

The bottleneck is not in terms of switching capacity nor
is it in terms of throughput across its backplane at present.

The latter issue is looming, but we're simply not there yet.

There have been substantial problems with respect to
convergence times. Many of these have been ameliorated with
experimental code now deployed throughout SprintLink and
ICM, which does selective packet dropping to assist
convergence rather than having the box keel over dead
process-switching packets when the SSE cache is being
completely repopulated.

We are no longer hovering close to the practical limits
of the current limitation, and are not very near the
reasonable maximum for the current platform.

This is not to say that we have all that much breathing-room,
but this and other developments in the works does and will
buy us much more time than we would gain by moving towards
a system of the kind other providers appear to favour.

The immediate danger is still in terms of BGP routing on
defaultless routers, and we are all now keenly aware of that
and I believe that even you have accepted that despite
available alternatives like dedicated route servers,
we must CIDRize or die.

I am shocked and surprised to see you insinuating that my
colleagues here and elsewhere and I simply don't know what
the hell we're doing or what we're installing in our
backbones.

We are all too keenly aware that the growth of much of the
Internet is outpacing the ability to build correctly-sized
equipment and is forcing most of us to design plans for the
deployment of large, scalable and for now over-engineered
routers to cope with the current trends in traffic increases
which do not look like they're going to shrink before the
end of 1997.

Not all of us are in the enviable position of being able
to sit back and enjoy the air of superiority that seems
to accompany being the chief commentator for a once-important
and formerly leading-edge backbone.

I think that you will find that at least two organizations
who aren't in that position have been pushing Cisco very
hard on the rather ingenious idea of developing an ISP test
lab wherein real problems can be reproduced and studied,
and where the characteristics of the current and
in-development router technologies with respect to heavy-duty
ISP environments can better be understood.

You'll probably say that this is all familiar and old-hat,
but kindly put your sneer aside and remember that the NSFNET
backbone service ended a doubling and a quarter ago with
respect to the size of the Internet then and now.

These problems are difficult, nobody has complete answers,
and "gee, just turn off a whole bunch of customers and
you'll be fine" doesn't strike me as a plausible solution
for Sprint, MCI, or AlterNet.

Sean.

P.S.: You might want to consider some "Sprint-did-it-firsts" which
developed both within ICM and SprintLink vis a vis
Cisco and general router technology deployment:
7000s, 64Mb RPs, BGP4, SSPs, 2MB SSPs, 7500s,
reprioritization of forwarding vs other tasks,
selective packet drop, and so on and so forth.
Vadim Antonov and Peter Lothberg were and are never idle,
and I fully intend to carry on the tradition of
pushing useful new technology into the field as fast as
it is available, because quite frankly, we need it all.
DS3? OC3? Hah. You ain't seen nothing yet, baby.

don't confuse the link encoding with back-haul design

if the network is deeply over-subscribed, you will drop packets.
the only question is "where?"

whether the link uses F/R-1490 framing or cisco HDLC doesn't change
that.

-mo

At 07:30 AM 11/18/95 -0500, Mike O'Dell wrote:

>
>don't confuse the link encoding with back-haul design
>

Don't confuse backhaul design with excessively high concentrations
of PVCs, grossly oversubscribed in ratio of aggragate ingress bandwidth.

If you don't drop the bits on ingress, you at least stand a fighting
chance of getting them (the bits) onto the backbone in the first place.

:-)

>if the network is deeply over-subscribed, you will drop packets.
>the only question is "where?"
>

Where indeed.

>whether the link uses F/R-1490 framing or cisco HDLC doesn't change
>that.
>


This has nothing at all to do with it. Regardless of the frame-relay
encapsulation, the fact that one can oversubscribe at ingress exists
and lends itself to what Vadim calls 'too may points of indirection'.
A private line only has two end-points (let's not discuss imuxes).

The possibility of sloppy or careless engineering is just a tad
higher when building frame-relay networks. This does not mean that
sloppiness can't be avaoided; it certainly can.

Just a thought,

- paul

Look up the Nov 6th issue of Communciations Week International.
Article entitled "Internet providers move to frame relay" by Richard
Karpinski and Kenneth Hart. Discusses how PSI, UUnet and Netcom have moved to
B-STDX 9000 frame relay switches by Cascade.

Hank

Don't believe everything you read in the funny papers......

as i said, comparing network architectures is not the same as
being mentioned in press releases.

-mo

On Fri, 17 Nov 1995, Ehud Gavron wrote:

> >When you buy a T1 private line, you can be assured that you're not
> >sharing it with 120 other end-users. :-)
>
> A T-1 private line to what? Do you think Sprint has sold
> more than 30 T-1s, filling up their DS-3 backbone?
>
> Do you think MCI has sold more than 30 T-1s, filling up
> their DS-3 backbone?
>
> You _are_ sharing bandwidth, unless 100% of your traffic
> is to your provider.

Yes, that is true but most ISP's in my area are overselling 100 X or
more. Sure I mux 4 T1 FR to 1 T1 port and oversell my bandwidth, but all
of my links have at least 20% free on them during our peek.

Sure oversell, but I think it is wrong to keep overselling when your
links are at 90% - 100% most of the time and you sell more t1's to get
the money.

Nathan Stratton CEO, NetRail, Inc. Your Gateway to the World!
---------------------------------------------------------------------------
Phone (703)524-4800 NetRail, Inc.
Fax (703)534-5033 2007 N. 15 St. Suite 5
Email sales@netrail.net Arlington, Va. 22201
WWW http://www.netrail.net/ Access: (703) 524-4802 guest
---------------------------------------------------------------------------

] Yes, that is true but most ISP's in my area are overselling 100 X or
] more. Sure I mux 4 T1 FR to 1 T1 port and oversell my bandwidth, but all
] of my links have at least 20% free on them during our peek.

What is your sample interval? Cisco's 5 minute average?

80% utilization at 5 minute samples means it is very likely the
lines are choked a good portion of that time.

-alan

At 06:56 PM 11/18/95 -0600, Alan Hannan wrote:

>
>] Yes, that is true but most ISP's in my area are overselling 100 X or
>] more. Sure I mux 4 T1 FR to 1 T1 port and oversell my bandwidth, but all
>] of my links have at least 20% free on them during our peek.
>
> What is your sample interval? Cisco's 5 minute average?
>
> 80% utilization at 5 minute samples means it is very likely the
> lines are choked a good portion of that time.
>
>-alan
>
>

Peek. A Freudian slip. :-)

- paul

On Sat, 18 Nov 1995, Alan Hannan wrote:

>
> ] Yes, that is true but most ISP's in my area are overselling 100 X or
> ] more. Sure I mux 4 T1 FR to 1 T1 port and oversell my bandwidth, but all
> ] of my links have at least 20% free on them during our peek.
>
> What is your sample interval? Cisco's 5 minute average?
>
> 80% utilization at 5 minute samples means it is very likely the
> lines are choked a good portion of that time.

No, 1 min averages, the liks is normally under 40% used. If it get a lot
of 80% 1 min averages we will upgrade the link.

Nathan Stratton CEO, NetRail, Inc. Your Gateway to the World!
---------------------------------------------------------------------------
Phone (703)524-4800 NetRail, Inc.
Fax (703)534-5033 2007 N. 15 St. Suite 5
Email sales@netrail.net Arlington, Va. 22201
WWW http://www.netrail.net/ Access: (703) 524-4802 guest
---------------------------------------------------------------------------

Paul;

You state:

At 05:47 PM 11/17/95 -0500, Sean Doran wrote:

>
>There are substantial disadvantages, too.
>
>In order to take advantage of the greater
>port-density(-per-dollar) on FR switches right now, the
>end user has to use FR, which is not always practical
>or desirable.
>

One of the reasons why end users find frame-relay undesireable is
that they cannot be assured that their provider is not grossly
oversubscribed on PVC-per-port density. When you buy a T1 private
line, you can be assured that you're not sharing it with 120 other
end-users. :-)

DS3/DS1 Backbone/Trunk capacity planning principles, whether across a Frame Relay Backbone or Cisco 7000 hdlc trunk network are still the same. It's just as easy to over configure DS3/DS1 Cisco HDLC trunks as Frame Relay trunks.

Potentially at issue here is not Frame Relay networks as a transport but that a Cisco 7000 can not scale properly to support 120+ end-users. :-)

Modern Frame Relay switches:

1) have sub-msec latency
2) can support multiple trunks at DS3+ (to include ATM)
3) are not burdened with processing any of the IP layer 3 nor routing overhead
4) because of 3 have a cost per port that is 300 to 400% less than a Cisco 7000
5) can have it's backbone shared across multiple services thereby reducing both capitalization and bandwidth expense
6) allow ISP to pass the cost savings on to customers


-----------------------------------------------------------------------------
Jeff Oliveto | Phone: +1.703.760.1764
Sr.Mgr Opns Technical Services | Fax: +1.703.760.3321
Cable & Wireless, Inc | Email: joliveto@cwi.net
1919 Gallows Road | URL: http://www.cwi.net/
Vienna, VA 22182 | NOC: +1.800.486.9999
-----------------------------------------------------------------------------

At 09:17 AM 11/19/95 -0500, Jeffrey P. Oliveto wrote:

>
>DS3/DS1 Backbone/Trunk capacity planning principles, whether across a Frame
>Relay Backbone or Cisco 7000 hdlc trunk network are still the same. It's
>just as easy to over configure DS3/DS1 Cisco HDLC trunks as Frame Relay
>trunks.
>

I would have a tendency to disagree with you here, but que sera, sera.


>Potentially at issue here is not Frame Relay networks as a transport but that
>a Cisco 7000 can not scale properly to support 120+ end-users. :-)
>
>Modern Frame Relay switches:
>
>1) have sub-msec latency
>2) can support multiple trunks at DS3+ (to include ATM)
>3) are not burdened with processing any of the IP layer 3 nor routing overhead
>4) because of 3 have a cost per port that is 300 to 400% less than a Cisco 7000
>5) can have it's backbone shared across multiple services thereby reducing
both
>capitalization and bandwidth expense
>6) allow ISP to pass the cost savings on to customers
>
>

I partially agree with your points above, but still maintain that it is much
easier to sloppily engineer a frame-relay network than one consisting of
point-to-point links.

My $.02.

- paul

Sean,

I tried to make some blanket statements without pointing fingers in
ant dirrection. I was pointing out that everyone has equipment
limitations to deal with. For example, we have some very severe PPS
limitations, but can still build an extremely stable and low loss
backbone if we can arrange our topology to keep within those limits.
That won't last forever and we know that. Everyone has to understand
and deal with the limits of the equipment they use.

In message <95Nov17.215523-0000_est.20701+37@chops.icp.net>, Sean Doran writes:
>
> Curtis -
>
> | The brick wall is that a particular piece of equipment from a particular
> | vendor that a lot of service providers have made a large investment in
> | doesn't really perform all that well in the real world.
>
> Please allow me to mitigate your politics with a dose
> of reality.
>
> sl-dc-8.sprintlink.net, a now-fairly-old Cisco 7000 with one
> of the first four 2MB SSP boards ever shipped outside
> Cisco's doors has been observed to switch 125kpps through
> several interfaces over a 15 minute period several times in
> the past three weeks.
>
> The bottleneck is not in terms of switching capacity nor
> is it in terms of throughput across its backplane at present.

I never said that the problem had anything to do with running out of
PPS. I said they did not perform well. Switching major amounts of
traffic for a 15 minute period and then falling over and dieing now
and then is not my idea of "performing well". Particularly if routers
can take others down in the process and create a sustained state of
routing instability. I feel justified in calling that "poor
performance". It's a subtle semantic difference. ;-)

> The latter issue is looming, but we're simply not there yet.
>
> There have been substantial problems with respect to
> convergence times. Many of these have been ameliorated with
> experimental code now deployed throughout SprintLink and
> ICM, which does selective packet dropping to assist
> convergence rather than having the box keel over dead
> process-switching packets when the SSE cache is being
> completely repopulated.
>
> We are no longer hovering close to the practical limits
> of the current limitation, and are not very near the
> reasonable maximum for the current platform.

Fine. The routers are being improved to make them more stable. They
certainly need it. Getting rid of the current caching design that
indicates that it is time to refresh by bombarding the RP with packets
would be a welcome change.

> This is not to say that we have all that much breathing-room,
> but this and other developments in the works does and will
> buy us much more time than we would gain by moving towards
> a system of the kind other providers appear to favour.
>
> The immediate danger is still in terms of BGP routing on
> defaultless routers, and we are all now keenly aware of that
> and I believe that even you have accepted that despite
> available alternatives like dedicated route servers,
> we must CIDRize or die.

At the last NANOG I gave a talk about scaling up the Internet and
described CIDR as the single most promising thing that providers to
cooperate on to improve scaling. We are putting a lot of effort into
upgrading configuration based on the IRR to allow accurate aggregation
and aggregation across provider boundaries with cooperating providers.
Some providers are supportive of these goals. Some want to undermine it.

[. .. defensive posturing and personal insults from Sean deleted for
brevity .. ]

> P.S.: You might want to consider some "Sprint-did-it-firsts" which
> developed both within ICM and SprintLink vis a vis
> Cisco and general router technology deployment:
> 7000s, 64Mb RPs, BGP4, SSPs, 2MB SSPs, 7500s,
> reprioritization of forwarding vs other tasks,
> selective packet drop, and so on and so forth.
> Vadim Antonov and Peter Lothberg were and are never idle,
> and I fully intend to carry on the tradition of
> pushing useful new technology into the field as fast as
> it is available, because quite frankly, we need it all.
> DS3? OC3? Hah. You ain't seen nothing yet, baby.

Please keep in mind that Sprint was also the first in other things.
First to sustain high backbone packet loss due to Cisco full cache
flush problems. First to experience Cisco cache overlap bugs on a
running network. And now first to experience sustained instability
due to sending traffic from the SP to the RP after major route change.

[. aside: <g> reprioritization of forwarding vs other tasks, selective
packet drop. Neat ideas. Did you think of that? ;-) ]

The difference may be one of approach. ANS has tried and has been
very successful at anticipating problems and convincing our vendors to
fix them before they become operational problems. We warned Cisco in
1993 when the 7000 first came out and we tested them of the full cache
flush, difficulty of doing overlap in a cache right, and potential for
instabilility if sending traffic to the RP to signal a need for cache
refresh. Cisco did not fix these to our satisfaction and so we
limited deployment of Cisco 7000s. At this point we will be skipping
the Cisco 7000 entirely as a backbone router, opting for a later
generation, possibly a Baynet router. You've chosen to deploy the
Cisco 7000 in your backbone and stepped on many of the bugs that were
the reason we refused to deploy Ciscos in our backbone.

Back to the point of my original message:

In summary in response to Gordan: There is a tommorrow for the
Internet. Some people have been very aware of the wall. The wall
only exists with respect to a particular generation of routers.

In response to Sean: ANS has choosen to skip the Cisco 7000. There
are better routers on the immediate horizon that will allow ANS to
build a next generation backbone and allow ANS to continue to expand.
Other Internet providers can expand too, in whatever way they deem
appropriate or wise, whether that is Frame Relay, ATM or whatever.

Curtis

PS- Cisco has been very responsive to ISP needs, particularly in terms
of protocol and feature support. Their next generation looks
extremely promising. So does Baynet's current generation with some
(fairly major) software change that we expect alpha on soon. Baynet
has been trying to become very responsive to ISP needs and may be
ready to step to the plate for real.

Sean writes:

* Key to the decision is the gamble that the switching
* ability of FR switches will compare with the switching
* ability of high-end routers in the same environment. This
* is not a gamble I am comfortable with, particularly as I
* have very strong ideas of how well an SSE-equipped 7000
* can switch packets in a backbone environment, and so far
* have only early impressions about one variety of popular
* FR switch in a similar environment.

Perhaps you have not looked at the right FR switch.
I know of at least one FR switch that will switch at
a higher rate than most currently used routers in a
backbone environment.

-Marten