Mailing List Archive

Jeff, would you or anyone give an example or two of when you get
PARTIALLY filled ATM cells? I've interviewed dave sincoskie, steve
tabaska, and Stephen von rump so far. they all rather like ATM. Don't
believe I heard about this problem from them. You aren't talking about
the switch not being able to tell whether a cell is mangled before it
transmits by any chance are you?????

*********************************************************************
Gordon Cook, Editor & Publisher Subscriptions: Individ-ascii $85
The COOK Report on Internet Individ. hard copy $150
431 Greenway Ave, Ewing, NJ 08618 USA Small Corp & Gov't $200
(609) 882-2572 Corporate $350
Internet: cook@cookreport.com Corporate Site Lic. $650
http://pobox.com/cook/ for new COOK Report Glossary of Internet terms
*********************************************************************


On Tue, 19 Mar 1996, Jeff Ogden wrote:

> If we are going to talk about ATM overhead when doing TCP/IP don't we need
> to talk about the overhead of partially filled ATM cells? Won't that cost
> you about 1/3 of your available bandwidth?
>
> -Jeff Ogden
> Merit
>
>
>

Gordon Cook writes:
> Jeff, would you or anyone give an example or two of when you get
> PARTIALLY filled ATM cells? I've interviewed dave sincoskie, steve
> tabaska, and Stephen von rump so far. they all rather like ATM. Don't
> believe I heard about this problem from them. You aren't talking about
> the switch not being able to tell whether a cell is mangled before it
> transmits by any chance are you?????

Since TCP/IP packets are of variable length, the last ATM cell
for the packet will not be full. The remaining bytes in the cell
are wasted bandwidth.

Gary Wright --------------------------------- gwright@connix.com
TCP/IP Illustrated Volume 2 ----- http://www.aw.com/cp/Vol2.html

> > If we are going to talk about ATM overhead when doing TCP/IP don't we need
> > to talk about the overhead of partially filled ATM cells? Won't that cost
> > you about 1/3 of your available bandwidth?
> >
> > -Jeff Ogden
> > Merit
>
> Jeff, would you or anyone give an example or two of when you get
> PARTIALLY filled ATM cells? I've interviewed dave sincoskie, steve
> tabaska, and Stephen von rump so far. they all rather like ATM. Don't
> believe I heard about this problem from them. You aren't talking about
> the switch not being able to tell whether a cell is mangled before it
> transmits by any chance are you?????

Gordon,

He's talking about the overhead due to carrying variable length IP packets
in fixed length ATM cells. Consequently the last cell of an AAL5 frame
will contain 0 - 39(?) bytes of padding, which is wasted bandwidth.
Assuming random length distributions (which they're not), the average waste
is about 20 bytes per packet.

My rough estimate, based on an average packet size of 200 bytes (used to be
correct, not sure anymore), is that the waste due to cell padding is about
10%. Due to the highly skewed packet size distribution the actual overhead
might vary substantially. Note that this 10% is on top of the ~10%
overhead due to the 5 byte ATM cell headers (5/53 ~= 10%), and various
other overheads (some of which are also present in frame over Sonet
schemes).

There's beginning to be some expectation that there will be a transmission
capacity crunch in the carrier's Sonet nets, and this ~25% ATM cell tax may
be looked at carefully as packet over Sonet solutions emerge.

-- Jim

Jim Forster <forster@cisco.com> wrote:

>He's talking about the overhead due to carrying variable length IP packets
>in fixed length ATM cells.
...
>There's beginning to be some expectation that there will be a transmission
>capacity crunch in the carrier's Sonet nets, and this ~25% ATM cell tax may
>be looked at carefully as packet over Sonet solutions emerge.

Given the bimodailty of IP traffic size distribution (about 40% of packets are
small, like TCP ACKs or telnet/rlogin keystrokes) the ATM "cell tax" is
closer to 32%.

I.e. a dual clearline DS-3 actually carries as much user data as OC-3c ATM.
Which, incidentally, was why SprintLink backbone design is easily expandable
to dual links (that includes carefully considering implications for routing).
Sean presented that design on NANOG a year ago, BTW. Funny thing, the design
is expandable beyond that, too, so OC-3 ATM is already obsolete.

--vadim

At 07:51 PM 3/19/96 -0800, Jim Forster wrote:

>
>There's beginning to be some expectation that there will be a transmission
>capacity crunch in the carrier's Sonet nets, and this ~25% ATM cell tax may
>be looked at carefully as packet over Sonet solutions emerge.
>

I think it will be critically important, especially in long-haul
interconnect scenarios.

Historically speaking, as providers have replaced congested links
with faster ones, they have also become quickly congested. And since
these high-speed long-haul links are not cheap, you can bet that these
organizations want to get the most bang-for-the-buck. I can't imagine
that ~25% bandwidth overhead would be acceptable, especially when
more efficient alternatives exist.

Nice synopsis, Jim.

- paul

Jim Forster <forster@cisco.com> writes:
He's talking about the overhead due to carrying variable length IP packets
in fixed length ATM cells. Consequently the last cell of an AAL5 frame
will contain 0 - 39(?) bytes of padding, which is wasted bandwidth.
Assuming random length distributions (which they're not), the average waste
is about 20 bytes per packet.

My rough estimate, based on an average packet size of 200 bytes (used to be
correct, not sure anymore), is that the waste due to cell padding is about
10%. Due to the highly skewed packet size distribution the actual overhead
might vary substantially. Note that this 10% is on top of the ~10%
overhead due to the 5 byte ATM cell headers (5/53 ~= 10%), and various
other overheads (some of which are also present in frame over Sonet
schemes).

There's beginning to be some expectation that there will be a transmission
capacity crunch in the carrier's Sonet nets, and this ~25% ATM cell tax may
be looked at carefully as packet over Sonet solutions emerge.


Great. Let's estimate the ATM bandwidth available on an OC-3. With
STS-3c SONET at 155.520 Mbps it gets reduced to 149.760 Mbps due to
section, line and path SONET overhead.

Next we reduce 10% due to 20 bytes lost on average 200 byte long
packets due to incomplete ATM cell fills resulting in 134.784 Mbps.

Then we need to subtract 9.43% due to ATM headers of 5 bytes in 53 byte
packets.

Thus we end up with a 122.069 Mbps true information rate which is about
78.5% of the nominal OC-3 capacity.

And we still have not raised the issues of wasted reserved bandwidth
and routing ease which were recently addressed very eloquently by
Vadim Antonov when comparing connection based with packet based
transport.


Wolfgang


Wolfgang Henke wolfgang@whnet.com . http://www.whnet.com/wolfgang/
WH Networks ... ftp.whnet.com /pub/wolfgang
2672 Bayshore Parkway Suite 503 ....... +1 415 390 9316
Mountain View CA 94043 ................... fax +1 415 390 9317

> Let's estimate the ATM bandwidth available on an OC-3. ...

You can find a paper that addresses this question at

http://www.msci.magic.net/docs/magic/overhead.ps or
ftp:ftp.magic.net/pub/magic/overhead.ps

It covers OC-3c, OC-12c, 100 and 140 Mbps TAXI, and DS3 framing, and
gives the effective bandwidth at each layer.
--
John Cavanaugh EMail: johnc@msc.edu
Minnesota Supercomputer Center, Inc. johnc@cray.com
1200 Washington Ave. S Phone: +1 612 337-3556
Minneapolis, MN 55415 FAX: +1 612 337-3400

johnc@msc.edu (John Cavanaugh) writes:
> You can find a paper that addresses this question at
>
> http://www.msci.magic.net/docs/magic/overhead.ps or
> fte:ftp.magic.net/pub/magic/overhead.ps
>
> It covers OC-3c, OC-12c, 100 and 140 Mbps TAXI, and DS3 framing, and
> gives the effective bandwidth at each layer.

Thank you for the reference. I will certainly check it out. Here is a
little table which I hope will help ISPs planning to upgrade bandwidth:


SONET (Synchronous Optical Network) speeds given in Mbps

nominal w/o Sonet ATM TCP/IP
overhead

OC-3 STS-3c 155.520 149 122 137 future net backbone

OC-12 STS-12c 622.080 599 488 549

OC-48 STS-48c 2488.32 2396 1953 2197 current telco backbone

OC-192 STS-192c 9953.28 9585 7813 8788

on single mode fiber
or multi mode fiber
or CATV 75 ohm coax (the lower speeds)

Additions and corrections are welcome.
Wed Mar 20 07:56:41 PST 1996
http://www.whnet.com/wolfgang/sonet.html

> John Cavanaugh EMail: johnc@msc.edu
> Minnesota Supercomputer Center, Inc. johnc@cray.com
> 1200 Washington Ave. S Phone: +1 612 337-3556
> Minneapolis, MN 55415 FAX: +1 612 337-3400


Wolfgang


Wolfgang Henke wolfgang@whnet.com . http://www.whnet.com/wolfgang/
WH Networks ... ftp.whnet.com /pub/wolfgang
2672 Bayshore Parkway Suite 503 ....... +1 415 390 9316
Mountain View CA 94043 ................... fax +1 415 390 9317

At 08:38 AM 3/20/96 -0800, Wolfgang Henke wrote:

>
>Thank you for the reference. I will certainly check it out. Here is a
>little table which I hope will help ISPs planning to upgrade bandwidth:
>
>
>SONET (Synchronous Optical Network) speeds given in Mbps
>
> nominal w/o Sonet ATM TCP/IP
> overhead
>
>OC-3 STS-3c 155.520 149 122 137 future net backbone
>
>OC-12 STS-12c 622.080 599 488 549
>
>OC-48 STS-48c 2488.32 2396 1953 2197 current telco backbone
>
>OC-192 STS-192c 9953.28 9585 7813 8788
>
>on single mode fiber
>or multi mode fiber
>or CATV 75 ohm coax (the lower speeds)
>

And here's one that [fills in the gaps] shows the relationship between STS
and SDH:

SONET/SDH Heirarchies

SONET Bit Rate SDH

STS-1/OC-1 51.84 Mb -
STS-3/OC-3 155.52 Mb STM-1
STS-12/OC-12 622.08 Mb STM-4
STS-24/OC-24 1244.15 Mb -
STS-48/OC-48 2488.32 Mb STM-16
STS-192/OC-192 9953.28 Mb STM-64

- paul

>> There's beginning to be some expectation that there will be a transmission
>> capacity crunch in the carrier's Sonet nets, and this ~25% ATM cell tax may
>> be looked at carefully as packet over Sonet solutions emerge.

> I.e. a dual clearline DS-3 actually carries as much user data as OC-3c ATM.
> Which, incidentally, was why SprintLink backbone design is easily expandable
> to dual links (that includes carefully considering implications for routing).
> Sean presented that design on NANOG a year ago, BTW. Funny thing, the design
> is expandable beyond that, too, so OC-3 ATM is already obsolete.
>
> --vadim


Other than the SprintLink design Vadim mentions above, what other
alternatives are emerging for high bandwidth IP transport? Is anyone
implementing PPP over SONET (RFC 1619)?


Shikhar

> From: Wolfgang Henke <wolfgang@whnet.com>
> Subject: Re: MCI [ATM overhead]
> To: johnc@msc.edu
> Date: Wed, 20 Mar 1996 08:38:08 -0800 (PST)
> Cc: nanog@merit.edu
> a [...]
> SONET (Synchronous Optical Network) speeds given in Mbps
>
> nominal w/o Sonet ATM TCP/IP
> overhead
>
> OC-3 STS-3c 155.520 149 122 137 future net backbone
> [...]

I think your 122 Mbps "ATM" number could be a bit confusing, even knowing
the assumptions you described in earlier mail. (Also, more bandwidth seems
to be available to "TCP/IP" than appears to be available from ATM...)

If it helps, the following numbers are from John Cavanaugh's paper:

Line Rate 155.520 Mbps

Available to ATM 149.760
(SONET payload)
Available to AAL 135.632
(ATM payload)

John then computes the overhead for three MTUs, and yields rates
available to IP and TCP:

MTU
576 9180 65527

Available to IP 125.198 135.102 135.547

Available to TCP 116.504 134.513 135.464

These are the maximum available rates, namely they assume MTU-sized
packets.

The reader can apply their favorite packet size distributions to these
numbers.


Having said all that, I am not sure where that leaves us.

One could theoretically remove the SONET overhead, but then one looses
the ability to manage the SONET link.

One could remove the ATM overhead, but then one has a point-to-point
link, rather than a link over which data from many sources can be
multiplexed.

-tjs

Jeff Ogden wrote:

>I don't think you can look at bandwidth without also looking at cost. If
>the cost for an OC-3 is less than the cost for two DS3s, ...

Then you can just put sync muxes at ends and get three clearline
DS-3s. Still beats ATM over OC-3. (Of course, RFC-1916 is better).

The pricing on ATM transport is merely an artefact of "pilot"
status of ATM networks. Carriers lose money on that. When
market will be established the prices are bound to rise to
that of native IP transport, or, likely, more (as ATM does not handle
levels of overcommitment found in IP backbones now).

IP transport also tends to be overpriced, as revenues are used
to sustain constant build-up (as opposed to capital investments
used to build FR and ATM networks). That silly kind of accounting
(when growth costs are counted as operational expenses in IP
service) is what i was rallying against in Sprint for quite a
time (that's being fixed, or so i told, now).

--vadim

> Line Rate 155.520 Mbps
> Available to ATM 149.760
> Available to AAL 135.632

> The reader can apply their favorite packet size distributions to these
> numbers.
> Having said all that, I am not sure where that leaves us.

That leaves us with the throughput of two DS3 links. It doesn't seem worth
the effort given how short term a solution it is.

> I.e. a dual clearline DS-3 actually carries as much user data as OC-3c ATM.
> Which, incidentally, was why SprintLink backbone design is easily expandable
> to dual links (that includes carefully considering implications for routing).
> Sean presented that design on NANOG a year ago, BTW. Funny thing, the design
> is expandable beyond that, too, so OC-3 ATM is already obsolete.
>
> --vadim
>

If OC-3 is obsolete, what does that make DS-1s and DS-3s?


Shikhar

> From: salo@msc.edu (Tim Salo)
> > From: Wolfgang Henke <wolfgang@whnet.com>
> > SONET (Synchronous Optical Network) speeds given in Mbps
> >
> > nominal w/o Sonet ATM TCP/IP
> > overhead
> >
> > OC-3 STS-3c 155.520 149 122 137 future net backbone
> > [...]
>
> I think your 122 Mbps "ATM" number could be a bit confusing, even knowing
> the assumptions you described in earlier mail. (Also, more bandwidth seems
> to be available to "TCP/IP" than appears to be available from ATM...)
>
I believe that the number is for TCP/IP carrying capacity _without_ ATM.


> One could remove the ATM overhead, but then one has a point-to-point
> link, rather than a link over which data from many sources can be
> multiplexed.
>
Rather, that leaves us with the excellent (very desirable) option of a
link where data from many sources are multiplexed by TCP/IP....

I do not see what ATM buys in this situation.

WSimpson@UMich.edu
Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
BSimpson@MorningStar.com
Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

>
> > One could remove the ATM overhead, but then one has a point-to-point
> > link, rather than a link over which data from many sources can be
> > multiplexed.
> >
> Rather, that leaves us with the excellent (very desirable) option of a
> link where data from many sources are multiplexed by TCP/IP....
>
> I do not see what ATM buys in this situation.

Like any technology, ATM will not be all things to all people. If all you are
looking for is a "big, fat pipe" below IP, you will look at all your available
options in addition to ATM. (You will be looking at all options regardless).
In the end, you will choose whatever makes sense technologically AND ECONOMICALLY.


Shikhar

> Date: Thu, 21 Mar 96 08:48:40 GMT
> From: "William Allen Simpson" <wsimpson@greendragon.com>
> To: nanog@merit.edu
> Subject: Re: MCI [ATM overhead]
>
> > From: salo@msc.edu (Tim Salo)
> > > From: Wolfgang Henke <wolfgang@whnet.com>
> > > SONET (Synchronous Optical Network) speeds given in Mbps
> > >
> > > nominal w/o Sonet ATM TCP/IP
> > > overhead
> > >
> > > OC-3 STS-3c 155.520 149 122 137 future net backbone
> > > [...]
> >
> > I think your 122 Mbps "ATM" number could be a bit confusing, even knowing
> > the assumptions you described in earlier mail. (Also, more bandwidth seems
> > to be available to "TCP/IP" than appears to be available from ATM...)
> >
> I believe that the number is for TCP/IP carrying capacity _without_ ATM.

I don't know. It doesn't look right.

> > One could remove the ATM overhead, but then one has a point-to-point
> > link, rather than a link over which data from many sources can be
> > multiplexed.
> >
> Rather, that leaves us with the excellent (very desirable) option of a
> link where data from many sources are multiplexed by TCP/IP....

You are correct in observing that IP traffic can be multiplexed across
a point-to-point link. As shown below, ATM provides link-layer multiplexing
of data from multiple [link-layer] geographic locations. Of course,
IP traffic can be multiplexed over ATM virtual channels, just as it can
across point-to-point links.

> I do not see what ATM buys in this situation.

I believe that we have at least one mid-level using ATM to connect to
multiple NAPs in roughly the following configuration:


NAP NAP NAP
| | |
| | |
...........................
. .
. .
. ATM Wide-Area Service .
. .
. .
...........................
|
|
Mid-Level

There are also several production wide-area IP networks which are using
a similar configuration, including parts of ESNet and NASA NREN.

While I have not been privy to the economic analysis which justified
these networks, I suspect that ATM wide-area networks provided a useful
price/performance point.

I also believe that a number NSPs are using ATM as a multiplexing technology,
or are using carrier services which use ATM as a multiplexing technology.

-tjs

[.These arguments sound a bit like the Cray [the supercomputer person/company]
approach to memory: "real computers have real memory." I guess those
who couldn't afford or didn't need gigabytes of real memory made do with
virtual memory.]

I don't think you can look at bandwidth without also looking at cost. If
the cost for an OC-3 is less than the cost for two DS3s, ...

-Jeff Ogden
Merit


At 9:26 AM 3/21/96, Shikhar Bajaj wrote:
>> I.e. a dual clearline DS-3 actually carries as much user data as OC-3c ATM.
>> Which, incidentally, was why SprintLink backbone design is easily expandable
>> to dual links (that includes carefully considering implications for
>>routing).
>> Sean presented that design on NANOG a year ago, BTW. Funny thing, the design
>> is expandable beyond that, too, so OC-3 ATM is already obsolete.
>>
>> --vadim
>>
>
>If OC-3 is obsolete, what does that make DS-1s and DS-3s?
>
>
>Shikhar

salo@msc.edu (Tim Salo) wrote:

> I don't know. It doesn't look right.

Tim, please note that I keep the page updated to whatever I perceive
the current consensus is. http://www.whnet.com/wolfgang/sonet.html

Thank you for posting!

Wolfgang

At 03:09 PM 3/21/96 +0800, Vadim Antonov wrote:
> ... (as ATM does not handle
>levels of overcommitment found in IP backbones now).
>
>
>--vadim
>
>
Vadim;

I think that statement, as I interpret it, could only apply
if you are running IP over a rate-controlled ATM backbone (either
CBR or VBR QoS). Nobody in their right mind does that if they
have an alternative.

ATM responds to "overcommitment", if you mean "excess offered
load", by shedding that load either in the middle of the backbone
at the bottleneck if you are running IP over UBR or at the edge of
the network if you are running a proprietary ABR.

IP routers respond to excess offered load by shedding that load in
the middle of the backbone, since IP router QoS is essentially the
same as ATM UBR.

In this sense, with ATM you have two choices for handling overcommit-
ment (if your ATM switch vendor has a proprietary ABR) but with
routers you have only one.

--Kent

Thank you for your interest in our announcement, "Connections to the
Internet". We will send you a copy as soon as possible. For your
information, please note that this information is also available at the
web site http://www.cise.nsf.gov/cise/ncri/connect96.html

Best regards,

Mark Luker





_________________________________________________
Mark Luker, Program Director NSFNet
National Science Foundation
4201 Wilson Blvd, room 1175
Arlington, VA 22230
mluker@nsf.gov (703)306-1949 Fax: (703)306-0621


On Wed, 20 Mar 1996, Paul Ferguson wrote:

> At 08:38 AM 3/20/96 -0800, Wolfgang Henke wrote:
>
> >
> >Thank you for the reference. I will certainly check it out. Here is a
> >little table which I hope will help ISPs planning to upgrade bandwidth:
> >
> >
> >SONET (Synchronous Optical Network) speeds given in Mbps
> >
> > nominal w/o Sonet ATM TCP/IP
> > overhead
> >
> >OC-3 STS-3c 155.520 149 122 137 future net backbone
> >
> >OC-12 STS-12c 622.080 599 488 549
> >
> >OC-48 STS-48c 2488.32 2396 1953 2197 current telco backbone
> >
> >OC-192 STS-192c 9953.28 9585 7813 8788
> >
> >on single mode fiber
> >or multi mode fiber
> >or CATV 75 ohm coax (the lower speeds)
> >
>
> And here's one that [fills in the gaps] shows the relationship between STS
> and SDH:
>
> SONET/SDH Heirarchies
>
> SONET Bit Rate SDH
>
> STS-1/OC-1 51.84 Mb -
> STS-3/OC-3 155.52 Mb STM-1
> STS-12/OC-12 622.08 Mb STM-4
> STS-24/OC-24 1244.15 Mb -
> STS-48/OC-48 2488.32 Mb STM-16
> STS-192/OC-192 9953.28 Mb STM-64
>
> - paul
>
>

This is betting on ATM prices being low for a long time --
long enough for investments to ATM equipment to pay off.

From the point of view of ISPs which get lines at cost this
is a no-brainer choice.

--vadim

> From: jon@branch.com (Jon Zeeff)
> Subject: Re: MCI [ATM overhead]
> To: salo@msc.edu (Tim Salo)
> Date: Thu, 21 Mar 1996 09:04:44 -0500 (EST)
> Cc: wolfgang@whnet.com, nanog@merit.edu
>
> > Line Rate 155.520 Mbps
> > Available to ATM 149.760
> > Available to AAL 135.632
>
> > The reader can apply their favorite packet size distributions to these
> > numbers.
> > Having said all that, I am not sure where that leaves us.
>
> That leaves us with the throughput of two DS3 links. It doesn't seem worth
> the effort given how short term a solution it is.

I am not sure I follow your arithmetic. The line rate on a DS-3 is
44.736 Mbps. You need to subtract several overheads from that: at
least some DS-3 and some PPP overhead.

How does that add up to half of the throughput of an OC-3c IP/ATM link?

As long as I took this opportunity to inquire, what to you mean by
"given how short term a solution it is?"

-tjs

> Date: Thu, 21 Mar 1996 15:09:51 +0800
> From: avg@postman.ncube.com (Vadim Antonov)
> To: jogden@merit.edu, nanog@merit.edu
> Subject: Re: MCI [ATM overhead]
> [...]
> The pricing on ATM transport is merely an artefact of "pilot"
> status of ATM networks. Carriers lose money on that. When
> market will be established the prices are bound to rise to
> that of native IP transport, or, likely, more (as ATM does not handle
> levels of overcommitment found in IP backbones now).
> [...]

Hmmm... Does that imply that the NSP that can take advantage of
underpriced services, (perhaps including ATM, if you are correct),
will have a competitive advantage?

-tjs