Mailing List Archive

Get a 100Mb FDDI or Ethernet connection between the two colocated
routers.

Then bring in multiple Fast packet services into your router which then
aggregates out the LAN port.



On Fri, 19 Jan 1996, George H. Clapp wrote:

> I'm also curious about the value of co-location. Using a fast packet
> service (Frame Relay, SMDS, or ATM) allows your on-site router to
> communicate directly with a router of another ISP. There's no need to
> purchase another router to place at the co-location site. Why incur the
> additional cost?
>
> George Clapp
> voice: 201-829-4610
> fax: 201-829-2504
> page: 800-980-1298
> email: clapp@bellcore.com
>

---
Stephen Balbach "Driving the Internet to Work"
VP, ClarkNet due to the high volume of mail I receive please quote
info@clark.net the full original message in your reply.

From: "George H. Clapp" <clapp@bellcore.com>

> I'm also curious about the value of co-location. Using a fast packet
> service (Frame Relay, SMDS, or ATM) allows your on-site router to
> communicate directly with a router of another ISP. There's no need to
> purchase another router to place at the co-location site. Why incur the
> additional cost?

To avoid having to use any of the fast packet services you mentioned?
Or to allow you to use routers, which you know about, instead of being
dependent on the random characteristics of switches the telephone
company bought?

Not using either of the last two fast packet services, in particular, will
also yield 30% more useful bandwidth from a T3 circuit. This all by itself
may make up for the co-location costs and the cost of a router.

Dennis Ferguson

> > I'm also curious about the value of co-location. Using a fast packet
> > service (Frame Relay, SMDS, or ATM) allows your on-site router to
> > communicate directly with a router of another ISP. There's no need to
> > purchase another router to place at the co-location site. Why incur the
> > additional cost?
>
> To avoid having to use any of the fast packet services you mentioned?
> Or to allow you to use routers, which you know about, instead of being
> dependent on the random characteristics of switches the telephone
> company bought?
>
> Not using either of the last two fast packet services, in particular, will
> also yield 30% more useful bandwidth from a T3 circuit. This all by itself
> may make up for the co-location costs and the cost of a router.

Actually, the overhead for IP over SMDS is more like 40%, if I remember
correctly, because of all the headers (8 byte SNAP, 32 byte SMDS) on top of
the ATM cell tax (5 out of 53 on cell headers, average 1/2 a cell wasted at
the end of the packet which occurs about ever 5 or 6 cells), figured on the
observed Internet average packet size of ~220 bytes a couple years ago.
What's the average packet size now? I think the overhead on DS-3 ATM/SMDS
is worse than on OC-3, if I recall correctly, because the DS-3 PLCP is so
wasteful as well. Someone should redo these calculations more accurately.

So, if the fast packet based access to NAPs is priced at ~40% less, *and*
the ATM switches perform as well as the FDDI switches, then it is a don't
care.


-- Jim

On Fri, 19 Jan 1996, Jim Forster wrote:

> > Not using either of the last two fast packet services, in particular, will
> > also yield 30% more useful bandwidth from a T3 circuit. This all by itself
> > may make up for the co-location costs and the cost of a router.
>
> What's the average packet size now? I think the overhead on DS-3 ATM/SMDS
> is worse than on OC-3, if I recall correctly, because the DS-3 PLCP is so
> wasteful as well. Someone should redo these calculations more accurately.

With PLCP, maximum bit rate available for ATM cells =

53 cells * 12 rows * 8bits / 125us = 40.704 mpbs.

So, 40.704 * 48/53 = 36.864 mpbs = data rate.

Now if you use AAL5, then it's another 2 bytes for SAR, and 2 bytes for
CS, so, 40.704 * (48-4)/53 = 33.792 mpbs.

Now, that's the theorectical maximum, and you go down from there.

-dorian
______________________________________________________________________________
Dorian Kim Email: dorian@cic.net 2901 Hubbard Drive
Network Engineer Phone: (313)998-6976 Ann Arbor MI 48105
CICNet Network Systems Fax: (313)998-6105 http://www.cic.net/~dorian

> > What's the average packet size now? I think the overhead on DS-3 ATM/SMDS
> > is worse than on OC-3, if I recall correctly, because the DS-3 PLCP is so
> > wasteful as well. Someone should redo these calculations more accurately.
>
> With PLCP, maximum bit rate available for ATM cells =
>
> 53 cells * 12 rows * 8bits / 125us = 40.704 mpbs.
>
> So, 40.704 * 48/53 = 36.864 mpbs = data rate.
>
> Now if you use AAL5, then it's another 2 bytes for SAR, and 2 bytes for
> CS, so, 40.704 * (48-4)/53 = 33.792 mpbs.

You are correct up til you get to the efficiency at the AAL level. AAL-5 does
not have any SAR overhead but it does have 8 bytes of CS overhead. Now, this
overhead is on a per PDU basis and NOT a per cell basis. Thus, you maximum
efficiency will be

36.864 * [(x-8) / x] where x = PDU size in bytes.

Thus, if you just send 64-byte packets, your min throughput will
be 36.8 * (56 / 64) = 32.3 Mbps.

Of course, larger PDUs are more efficient.



Shikhar

On Fri, 19 Jan 1996, Shikhar Bajaj wrote:

> You are correct up til you get to the efficiency at the AAL level. AAL-5 does
> not have any SAR overhead but it does have 8 bytes of CS overhead. Now, this
> overhead is on a per PDU basis and NOT a per cell basis. Thus, you maximum
> efficiency will be
>
> 36.864 * [(x-8) / x] where x = PDU size in bytes.
>
> Thus, if you just send 64-byte packets, your min throughput will
> be 36.8 * (56 / 64) = 32.3 Mbps.
>
> Of course, larger PDUs are more efficient.

Thanks for the correction. I knew that number didn't quite look right.

-dorian
______________________________________________________________________________
Dorian Kim Email: dorian@cic.net 2901 Hubbard Drive
Network Engineer Phone: (313)998-6976 Ann Arbor MI 48105
CICNet Network Systems Fax: (313)998-6105 http://www.cic.net/~dorian

At 12:24 PM 1/19/96 -0500, Stephen Balbach wrote:
>
>Get a 100Mb FDDI or Ethernet connection between the two colocated
>routers.
>
>Then bring in multiple Fast packet services into your router which then
>aggregates out the LAN port.
>
>
This breaks down when the connection speed goes from DS-3 to OC-3.

--Kent

> At 12:24 PM 1/19/96 -0500, Stephen Balbach wrote:
> >Get a 100Mb FDDI or Ethernet connection between the two colocated
> >routers.
> >Then bring in multiple Fast packet services into your router which then
> >aggregates out the LAN port.

> This breaks down when the connection speed goes from DS-3 to OC-3.

LAN speeds can keep up with WAN speeds at OC-3, just use ATM.

---
Stephen Balbach "Driving the Internet to Work"
VP, ClarkNet due to the high volume of mail I receive please quote
info@clark.net the full original message in your reply.

Hi.

My workstation is connected to a 640 megabit per second lan right now.

If you want one check out MyriCom at http://www.myri.com/

--jon.

Re: value of colo. Sometimes it's helpful to use a colo as a mini-POP
and drag lines out to local customers or to other colo members who want
to buy transit. A NAP with enough space to rent whole cages seems like
something a lot of people would want -- down at WilTel they really
frown on private wires between cages and they don't own a GIGAswitch.

Re: fast packet data services. The telcos have really screwed the pooch
with ISDN, SMDS, and ATM. I'm not sure why F-R has worked well without
years of delay, but it's a real anomoly. I have little if any confidence
that I will be able to buy OC3 speed ATM and use it to contact 200
(or even 20) metro-area NAP partners any time this century.

ISP's who practice conservative engineering (i.e., safe sex) will continue
to buy the fastest telco pipes they can get _among_those_which_actually_work_
and this is probably going to be raw fibre before it starts to be 155Mb ATM.
I'm not in love with the idea of a barn full of routers all hooked up to
a GIGAswitch milking machine, but I have to prefer the config that works.

I know that after years of trauma and chaos and delay, we now have a couple
of NAPs implemented on an ATM mesh. If I ever see an ATM NAP come online
with dozens of 155Mb/s members and no chaos, trauma or delay, I promise to
post a full apologia here.

On Fri, 19 Jan 1996, Kent W. England wrote:
> At 12:24 PM 1/19/96 -0500, Stephen Balbach wrote:
> >Get a 100Mb FDDI or Ethernet connection between the two colocated
> >routers.
> >
> >Then bring in multiple Fast packet services into your router which then
> >aggregates out the LAN port.
> >
> >
> This breaks down when the connection speed goes from DS-3 to OC-3.

I understand this (I think :), but I've a theoretical question.

Assuming the following:

T1 T1 T1 T1 ...
| | | | |
R R R R R [R = router]
| | | | |
+-------+-------+-------+-------+
| Etherswitch |
+---------------+---------------+
|(A) [A = 10Mb Ethernet]
+---------------+---------------+
| Cisco 4500 |
+---------------+---------------+
|
T3

Just how many T1->Router->Etherswitch connections can be run through
point A (a single 10Mb ethernet) before things become unbearable (real
world here, I think I can do the math :-)? Should I look at a 100Mb
ethernet port on the 4500 instead of the dual 10M ethernet option?

I'm working on a project for a small NAP, and this concerns me. I'd
rather not do something stupid with the money ;).

rus

Russ Pagenkopf (406) 542-0838 Internet Services Montana (ism.net)
Hardware and Business Manager Connecting the World to Montana
All questions can be answered thus. NO. YES. MAYBE. EH?

The reasons for co-location that have been mentioned so far include...

- zero-mile circuits
- a hardened location for equipment
- to establish a local presence rather than building out one's own space.
- skepticism about the value of the fast packet services; the skepticism
has several flavors:
- reduced bandwidth due to protocol overhead
- less reliability than a shared FDDI
- performance of ATM switches compared to FDDI switches

'Zero-mile circuit' isn't clear to me because, regardless of the technology
used for the NAP, it's still necessary to purchase a circuit from your site
to the NAP. The circuit is either included in the price of the fast packet
service or purchased separately as a leased line to the co-location site.

The next two points, 'hardened location' and 'establish a local presence,'
make sense. If you do not have a presence in the city where the NAP is
located, co-location serves that purpose.

The various statements made concerning the value of the fast packet services
are familiar arguments. I'd like to point out that CERFnet uses SMDS, the
Northwest NAP (NIX) uses Frame Relay, and the San Francisco and Chicago ATM
NAPs are carrying significant amounts of traffic. The Chicago NAP is
experiencing peak traffic of nearly 80 Mbps. What is needed to move the
debate forward are objective performance criteria and measurements, and I
look forward to the work of the Benchmarking Methodology Working Group to
provide that criteria.

George Clapp
voice: 201-829-4610
fax: 201-829-2504
page: 800-980-1298
email: clapp@bellcore.com

Paul Vixie wrote:
>I know that after years of trauma and chaos and delay, we now have a couple
>of NAPs implemented on an ATM mesh. If I ever see an ATM NAP come online
>with dozens of 155Mb/s members and no chaos, trauma or delay, I promise to
>post a full apologia here.

I'll be looking forward to that post :-)

Seriously though, Paul, the new platform (the Stratacom BPX) going in place
here at Pacific Bell will be a test to the concerns you raise, if indeed we
can bring online the PacBell NAP "with dozens of 155Mb/s members and no
chaos, trauma or delay". I believe this will be the test of the hypothesis
that the combination of properly engineered access line buffering, backplane
speeds and congestion management capabilities can alleviate the "good put"
problems previously experienced by IP over ATM implementations. We have
quite a bit of date for the DS3 level that we will be presenting at the
upcoming NANOG meeting, and we will certainly be compiling data on the OC3
testing as well (which, BTW, we will make public as soon as the data
compilation is complete.)
-------------------------------------------------------------------
Warren K. Williams, Director - NAP Project Management
Pacific Bell Business Communications Services
Email: wkwilli@pacbell.com Phone: 510.867.9065
-------------------------------------------------------------------

At 06:36 PM 1/19/96 -0500, Stephen Balbach wrote:

>> At 12:24 PM 1/19/96 -0500, Stephen Balbach wrote:
>> >Get a 100Mb FDDI or Ethernet connection between the two colocated
>> >routers.
>> >Then bring in multiple Fast packet services into your router which then
>> >aggregates out the LAN port.
>
>> This breaks down when the connection speed goes from DS-3 to OC-3.
>
>LAN speeds can keep up with WAN speeds at OC-3, just use ATM.
>
>---
>Stephen Balbach "Driving the Internet to Work"
>VP, ClarkNet due to the high volume of mail I receive please quote
>info@clark.net the full original message in your reply.
>
>

You are, obviously, kidding.

- paul [in singapore at the moment]

--
Paul Ferguson || ||
Consulting Engineering || ||
Reston, Virginia USA |||| ||||
tel: +1.703.716.9538 ..:||||||:..:||||||:..
e-mail: pferguso@cisco.com c i s c o S y s t e m s

> >> This breaks down when the connection speed goes from DS-3 to OC-3.
> >
> >LAN speeds can keep up with WAN speeds at OC-3, just use ATM.
> >
> >---
> >Stephen Balbach "Driving the Internet to Work"
> >VP, ClarkNet due to the high volume of mail I receive please quote
>
> You are, obviously, kidding.
> --
> Paul Ferguson || ||

He's obviously talking about some kind of LAN quite unlike Ethernet. ;-)

Dave

--
Dave Siegel President, RTD Systems & Networking, Inc.
(520)623-9663 Network Engineer -- Regional/National NSPs (Cisco)
dsiegel@rtd.com User Tracking & Acctg -- "Written by an ISP,
http://www.rtd.com/~dsiegel/ for an ISP."

somehow i missed the annoucement by the major router manufacturers that
they are supporting the technology you mentioned. (grin)

coding packets a 640 megabits isn't particularly hard - off-the-shelf
Fiberchannel parts can do 1.02 gigabits (each way) with no trouble (and
are used inside at least one "gigabit router" in concert with a GaAs
cross-bar switch to make gigabit-capable switching fabric.) [yes, there
are other issues in building fast switches.]

the general challenge is to make the technology into *products* which
work, are scalable, and manage to generate enough critical mass that
the technology suppliers will survive. this last point is as critical
as the first.

Cynicism aside, however, I have been following the MOSAIC/ATOMIC stuff
for several years and it is attractive for some purposes. I am hoping
for their every success. But I don't think that I'd bet the success of
a major internet interchange point on it right now.

That is, of course, if I knew it was in there... (big grin)

-mo

ps - thanks for mentioning the url. makes it easier to track what's
happening there.

At 04:57 PM 1/19/96 -0800, Paul A Vixie wrote:
>... The telcos have really screwed the pooch
>with ISDN, SMDS, and ATM. I'm not sure why F-R has worked well without
>years of delay, but it's a real anomoly. I have little if any confidence
>that I will be able to buy OC3 speed ATM and use it to contact 200
>(or even 20) metro-area NAP partners any time this century.
>
>ISP's who practice conservative engineering (i.e., safe sex) will continue
>to buy the fastest telco pipes they can get _among_those_which_actually_work_
>and this is probably going to be raw fibre before it starts to be 155Mb ATM.
>I'm not in love with the idea of a barn full of routers all hooked up to
>a GIGAswitch milking machine, but I have to prefer the config that works.
>
Religious arguments aside, the obstacle for TCP/IP performance over ATM has
been due to two factors, small buffers and no explicit flow control.

The new switches have large buffers. TCP works well with these switches.
Explicit flow control will allow more flexible bandwidth management. That
applies to TCP as well as ATM.

Check out the price of an ATM circuit (considering overhead, etc) and if you
perceive value for the BW then buy it. No apology needed, just purchase
orders. :-) If you find that you prefer raw bandwidth over optical fiber
and lots of cisco interfaces, fine, but if you like network level switching,
shared network bandwidth and the price for that, buy switched services.

--Kent

*some* new switches have adequate buffering for some potentially
useful values of delay-bandwidth product.

and other new switches barely have enough to replace a one-room
ethernet.

-mo

George H. Clapp <clapp@bellcore.com> wrote:

>The reasons for co-location that have been mentioned so far include...

>- zero-mile circuits

That means expansion of that "circuit" is free.

>- a hardened location for equipment

That is hardly relevant, as all large players have facilities just
as good.

>- to establish a local presence rather than building out one's own space.

Same.

> - skepticism about the value of the fast packet services; the skepticism
> has several flavors:
> - reduced bandwidth due to protocol overhead

That is not "scepticism" but a sad truth. Getting 40% less bang
(on a typical IP traffic mix which includes *lots* of packets barely
exceeding one cell) is rather silly.

> - less reliability than a shared FDDI

Shared FDDI is simply a way to offer lower performance for lower
price. ATM port costs are not variable.

> - performance of ATM switches compared to FDDI switches

FDDI just works. It worked five years ago.

>'Zero-mile circuit' isn't clear to me because, regardless of the technology
>used for the NAP, it's still necessary to purchase a circuit from your site
>to the NAP.

The difference is between the clearline circuit and circuit with 40%
protocol overhead. Some big players are carriers themselves and
want to use _own_ circuits whereever possible.

The reason for collocated NAPs you completely missed is called
"scalability". Having all equipment in one room provides a lot
of flexibility in regard to interconnections. For example, if
A and B found that they have lots of mutual traffic they may want
to plug one more FDDI board in their boxes and drag a private FDDI
wire, to offload the traffic from shared infrastructure. When
new whiz-bang LAN technology appears you can easily upgrade connection
by plugging a new board into the collocated router, leaving the
old interface as a backup.

Which also leads to the reason #2 which you also missed -- redundancy.
When ATM switch is fried or went banana you're dead. FIXes and their
progeny had backup connections from the day one.

--vadim

On Sat, 20 Jan 1996, Paul Ferguson wrote:

> >LAN speeds can keep up with WAN speeds at OC-3, just use ATM.
> >
> >---
> >Stephen Balbach "Driving the Internet to Work"
> >VP, ClarkNet due to the high volume of mail I receive please quote
> >info@clark.net the full original message in your reply.
>
> You are, obviously, kidding.

Yes, what are you talking about?

Nathan Stratton CEO, NetRail, Inc. Your Gateway to the World!
---------------------------------------------------------------------------
Phone (703)524-4800 NetRail, Inc.
Fax (703)534-5033 2007 N. 15 St. Suite 5
Email sales@netrail.net Arlington, Va. 22201
WWW http://www.netrail.net/ Access: (703) 524-4802 guest
---------------------------------------------------------------------------

> other new switches barely have enough to replace a one-room ethernet.

speakng from testing experience - more seem to fall into this state

than into the 'ok for data' list

Scott

Kent,

> Religious arguments aside, the obstacle for TCP/IP performance over ATM has
> been due to two factors, small buffers and no explicit flow control.

I think you forgot one. The requirement to do VC-based reassembly also
makes router interfaces hideously complex (relative to interfaces to
frame-based networks, certainly), and hence prone to all the difficulties,
limitations and unhappy corner cases that complexity breeds. And an
additional requirement that the interfaces do something with flow control
indications is not going to make them simpler.

You know it is very hard not to wax poetic about ATM, despite one's
best efforts to be objective. I am quite sure that ATM will support
IP just fine in the end, just throw a few more bizillion dollars of
R&D at it and fix everything. With all that fancy silicon laying
around being sold at bargain-basement prices it'll be impossible to
ignore. And all the while the segments of the industry which might
be building routers are instead sitting on their hands waiting for
the ever-delayed release of the next latest-and-greatest, all-seeing
all-knowing SAR chipsets with fantasy visions of big honking ATM switches
surrounded by teeny tiny routers dancing in their heads. And so I'm
sure we'll get the big honking ATM switches surrounded by teeny tiny
routers, and all will be happy and working well, just five years too
late and five times too complex as the ATM industry figures out what
many people already knew and funds yet more research, and produces yet
more Forum standards, to work around the conceived-in defects with the
technology when supporting this type of service. So you are indeed
right, ATM will work very well for IP in the end and we'll never need
to think about how irrelevant it might have been if we'd spent just a
tenth of the effort advancing router technology.

So, in any case, I agree with you, if you pick your equipment right
and shake out the bugs ATM will carry IP just fine. And if the
phone company prices the service so that it looks attractive compared
to that provided by (cheaper and more reliable) TDMs, it'll sell like
hotcakes. But I don't think it is being "religious" to be unwilling
to let the phone company use one's own data to get its primary-school
education about high-end data networking, and to shake the bugs out of
a buggy technology, nor is it being "religious" to have regrets about
what might have been. ATM can be its own self-fulfilling prophesy all
by itself, just phone (not using ATM, no doubt, even the phone company
is smart enough to avoid this for its bread and butter) when you get it
working.

Dennis Ferguson
Speaking only for myself.