Mailing List Archive

>one other solution (being implemented, I believe), is a DNS server
>that listens to the BGP traffic, so it knows how far away things are,
>and when you ask it, it can chose from multiple responses to pick
>one "close".
>
> -mo

The only problem that it does not work at all.

Which WWW server is closer to me - one with BGP path

1239 3491 690 1333 (cnn.com)

or one at

1239 1792 (www.cnc.ac.cn)?

The second is in China, for God's sake!

There ain't no such thing as global metrics. The only
useful kind of metrics is administrative, and therefore
they cannot reflect any real characteristics of paths.

Even if there are uniform metrics, how do you tell
which link is overloaded and which isn't?

Cacheing appears to be the only sane way to distribute
load. You always know where the closest cacheing server is.
Now, there's a problem with coherency, but at least it
can be done w/o magic.

--vadim
- - - - - - - - - - - - - - - - -

"Public cacheing" has little merit, as efficiency of
cacheing decreases when covered population grows
beyond some threshold (i.e when diversity of requests
overwhelms the cacheing capacity -- process better
known as "thrashing").

On the other side, small populations do not produce
aggregatable demand patterns.

I.e. it looks like that ISP-provided cache servers would
be optimal. Conviniently, that allows to sidestep the problem
of finding cache servers. When you never talk to other provider's
cacheing servers you don't need to locate them.

There are two ways to do distribute load: "supply side"
caches (like secondary DNS servers) and "demand side"
caches (like cacheing proxy servers).

The supply-side cacheing has a fundamental problem is that
you cannot tell which supply-side server is best. So much
is clear from the DNS saga.

--vadim
- - - - - - - - - - - - - - - - -

>If I could _enforce_ caching by traffic interception then
>I could construct vaguely logical cache structures as an ISP.

One way to do that is to spoof TCP sessions in the routers :)
That is a horrible kludge, though...

>But relying on users to configure their end systems with the
>identity of the closest cache server using some ill defined
>"network metric" is always going to be an erratic affair at best.

How about allocating fixed IP address for cacheing servers?
Routing will take care of locating the "closest" one, and
browser vendors would be able to hardwire the default address.

Although one can tell that they have little clue and probably won't
do anything about it, given such impressive items on their track
record as HTTP 1.0 or Netscape HTML extensions.

--vadim
- - - - - - - - - - - - - - - - -

Well, i simply pointed out that solution based on
*hosts* choosing the best paths in global networks
is not going to work. It is in the same category as
source-based routing.

The network is simply too large for end-hosts to be
able to make any useful decisions. I though that was
understood long time ago, so nowadays resolvers do not
even attempt to guess which address is best in case
of multihomed hosts (beyond obvious directly-connected
network trick).

Pinging all addresses may be worse than just talking to
a random server. To make any meaningful measurement
you need to send many dozens of probe packets. Incidentally,
that is about as much as the average WWW exchange takes.

So the real solution is not to bother, and concentrate
on demand-side cacheing.

--vadim

PS I would also be sceptical about attempts to
"try" such things as perpetuum mobile or
palm reading just on the chance that they may
work.

PPS The funny thing is that there's a hope that a
good heuristic can be found -- since the network
is rather hierarchal a good choice of topologically
significant addressing is likely to produce situation
in which bit distance between addresses is a good
approximation of some "goodput" metric!

This is very much the same effect as that of
well-placed aggregation which produces routes close
to optimal with a lot less information.



Date: Mon, 13 May 1996 20:30:38 -0400
From: "Mike O'Dell" <mo@uunet.uu.net>

Vadim,

Yes, caching is a good idea (technically).

And who said "AS path length" was a wonderful global metric???
just because the "usual suspects" BGP implementation generally does
route selection based on that attribute doesn't mean that's the
ONLY thing one can do.

Depending on how much additional information
one wished to supply about ASes, their general level of connectivity,
and geographic location, one can conceivably produce hybrid metrics,
probably heuristic, which reflect some sense of "performance".

for example, the server could do round-trip measurements to "sufficiently
interesting" ASes so that it could base its behavior on observations.
the goal wouldn't be to do fine-grained decision-making, but if the
"Bruce Springsteen Ticket Server" caused some reasonably long-lived
congestion, it might be worthwhile redirecting some responses.

The assumption is that this special DNS server might be concentrated
on fielding responses for a special set of servers, like special Web
servers (could be caches), and not general connectivity.

So while few things are perfectly-universal solutions, the prospect
of implementing heuristics that we all use today in getting a sense of how
things are going, and what to try when the first guess is hosed,
seems like a worthwhile attempt.

some will work better than others.

that is neither news, nor a reason to not try it.

-mo

- - - - - - - - - - - - - - - - -

Should i drop into Noel-speak mode and start explaining difference
between O(n) and O(exp(n))?

If something works in several dozens or several hundreds places
does not mean it'll work in million places. And even if it can
work in million places but can't work with two million places
it is only half year worth of growth.

--vadim

check out sonar, draft-moore-sonar-01.txt.

randy
- - - - - - - - - - - - - - - - -

At 6:15 PM 13/5/96, Vadim Antonov wrote:
>Cacheing appears to be the only sane way to distribute
>load. You always know where the closest cacheing server is.

not even this is the case - you can guess but that may not always
work!

If I could _enforce_ caching by traffic interception then
I could construct vaguely logical cache structures as an ISP.

But relying on users to configure their end systems with the
identity of the closest cache server using some ill defined
"network metric" is always going to be an erratic affair at best.

geoff


- - - - - - - - - - - - - - - - -

Vadim Antonov wrote:
>Cacheing appears to be the only sane way to distribute
>load. You always know where the closest cacheing server is.
>Now, there's a problem with coherency, but at least it
>can be done w/o magic.

Discovering the topology of a group of CACHE servers also needs some
clarification. Typically, you only know your internal cache servers. There
is a strong international focus on this topic. However, I cannot see how
these can be of any "general public" value since a provider would construct
a cache to benefit internal customers.

My question remains, is this value to our customers and consequent reduction
in extrnal traffic sufficient to justify the effort? The research says yes.
What about actual experience in a real network?

..mike..
Mike Trest, ATMNET Voice: 619 643-1805
5440 Morehouse Drive Fax: 619 643-1901
San Diego, CA 92121 Pager: 619 960-9070

- - - - - - - - - - - - - - - - -

Vadim,

Yes, caching is a good idea (technically).

And who said "AS path length" was a wonderful global metric???
just because the "usual suspects" BGP implementation generally does
route selection based on that attribute doesn't mean that's the
ONLY thing one can do.

Depending on how much additional information
one wished to supply about ASes, their general level of connectivity,
and geographic location, one can conceivably produce hybrid metrics,
probably heuristic, which reflect some sense of "performance".

for example, the server could do round-trip measurements to "sufficiently
interesting" ASes so that it could base its behavior on observations.
the goal wouldn't be to do fine-grained decision-making, but if the
"Bruce Springsteen Ticket Server" caused some reasonably long-lived
congestion, it might be worthwhile redirecting some responses.

The assumption is that this special DNS server might be concentrated
on fielding responses for a special set of servers, like special Web
servers (could be caches), and not general connectivity.

So while few things are perfectly-universal solutions, the prospect
of implementing heuristics that we all use today in getting a sense of how
things are going, and what to try when the first guess is hosed,
seems like a worthwhile attempt.

some will work better than others.

that is neither news, nor a reason to not try it.

-mo
- - - - - - - - - - - - - - - - -

>The only problem that it does not work at all.
>
>Which WWW server is closer to me - one with BGP path
>
> 1239 3491 690 1333 (cnn.com)
>
>or one at
>
> 1239 1792 (www.cnc.ac.cn)?
>
>The second is in China, for God's sake!
>
>There ain't no such thing as global metrics. The only
>useful kind of metrics is administrative, and therefore
>they cannot reflect any real characteristics of paths.

On a global level, how useful would it be to base the decision
on whether the client address belongs to the the InterNIC block,
APNIC or the european registry ? I can live with exceptions
such as an MCI customer in India gets redirected to the
north american mirror server instead of the one in Singapore
because their address belongs to an InterNIC allocated MCI
block. How often do these exceptions happen ? Where can I
find the association of address blocks and registries ?

We are currently working on mirroring a copule of sites on all
major continents. I understand this approach does not optimize
routing to the web servers within north america.

Sanjay.
- - - - - - - - - - - - - - - - -

Even if there were workable global routing metrics, this problem _cannot_
be solved inside the confines of DNS, which specifies that there is no
meaning to the order of RR's in a response. So even if a server could put
them in the right order for a given client, that "client" might actually be
a recursive server whose connectivity was different from the end-TCP
client's. The recursive ("caching") server(s) can reorder the RR's and
frequently do (either LIFO or random). The client can reorder the RR's.
It is impermissable to send back a single RR when multiple RRs exist in
the RRset. Setting TTL to 0 to prevent caching is not good enough.

Doing this inside DNS is an idea utterly without merit. To find the "right"
way, start with Keith Moore's SONAR and then make it better in minor ways
and then implement it inside all exit gateways from this day forward.

Picking the closest server is an end-host issue.
- - - - - - - - - - - - - - - - -

Having a dynamic determination of which cache to use (say, by using
the one which responds fastest to a ping) wouldn't be so bad though.
Have each site keep a list of cache servers it considers close enough,
and pick the one with the instantaneous best performance whenever
fetching something.

-george william herbert
gherbert@crl.com
- - - - - - - - - - - - - - - - -

> Well, i simply pointed out that solution based on
> *hosts* choosing the best paths in global networks
> is not going to work. It is in the same category as
> source-based routing.

Actually, people do both and it seems to work well enough.

Some web servers track throughput to sites and redirect users to
mirror sites based on that data.

Some routers route based on source address (for various policy reasons,
including load).

- - - - - - - - - - - - - - - - -

> Well, i simply pointed out that solution based on
> *hosts* choosing the best paths in global networks
> is not going to work. It is in the same category as
> source-based routing.

Actually, people do both and it seems to work well enough.

Some web servers track throughput to sites and redirect users to
mirror sites based on that data.

Some routers route based on source address (for various policy reasons,
including load).


- - - - - - - - - - - - - - - - -

On Tue, 14 May 1996, Geoff Huston wrote:

> But relying on users to configure their end systems with the
> identity of the closest cache server using some ill defined
> "network metric" is always going to be an erratic affair at best.

I was wondering if someone had gotten around to intercepting the port 80
traffic and redirecting it to a local caching proxy. Doesn't seem much
more difficult to do than NAT stuff and there are enough providers running
FreeBSD boxes and the like as gateway routers that it may be worth trying
as an experiment.

Now that Cisco is adding NAT capabilities to the standard IOS it appears
that these "non-routing" functions may not be too hard to get standardised
across a large part of the net.

Michael Dillon Voice: +1-604-546-8022
Memra Software Inc. Fax: +1-604-546-3049
http://www.memra.com E-mail: michael@memra.com

- - - - - - - - - - - - - - - - -

On Mon, 13 May 1996, Vadim Antonov wrote:

> "Public cacheing" has little merit, as efficiency of
> cacheing decreases when covered population grows
> beyond some threshold (i.e when diversity of requests
> overwhelms the cacheing capacity -- process better
> known as "thrashing").
>
> On the other side, small populations do not produce
> aggregatable demand patterns.
>
> I.e. it looks like that ISP-provided cache servers would
> be optimal.

Especially so since ISP's have the opportunity to do social engineering on
their users by maintaining a "What's HOT" page on their server with daily
updates. If you can get people to check in on your WWW reviews first then
you have a much greater chance of getting cache hits.


Michael Dillon Voice: +1-604-546-8022
Memra Software Inc. Fax: +1-604-546-3049
http://www.memra.com E-mail: michael@memra.com

- - - - - - - - - - - - - - - - -

On Mon, 13 May 1996, Vadim Antonov wrote:

> Pinging all addresses may be worse than just talking to
> a random server.

Heuristics again. You don't really want to pick the *BEST* server, you
want to avoid the *WORST* server and if you track response times and
choose the server randomly (sort of) then you already know which ones are
the worst. Just add some sort of weighting that makes a poorly responding
server less and less likely to be chosen. Then when you get a good
response from it, slowly bring the weighting back to normal. Maybe have
some sort of TTL on the weighting as well.

> To make any meaningful measurement
> you need to send many dozens of probe packets. Incidentally,
> that is about as much as the average WWW exchange takes.

Yep. And the WWW exchange provides all the measurements you should need.

> PS I would also be sceptical about attempts to
> "try" such things as perpetuum mobile or
> palm reading just on the chance that they may
> work.

Palm reading is just a hash function that nature uses to make sure we
don't all rush to the same side of the boat :-)


Michael Dillon Voice: +1-604-546-8022
Memra Software Inc. Fax: +1-604-546-3049
http://www.memra.com E-mail: michael@memra.com

- - - - - - - - - - - - - - - - -

On Mon, 13 May 1996, Sanjay Dani wrote:

> On a global level, how useful would it be to base the decision
> on whether the client address belongs to the the InterNIC block,
> APNIC or the european registry ? I can live with exceptions
> such as an MCI customer in India gets redirected to the
> north american mirror server instead of the one in Singapore
> because their address belongs to an InterNIC allocated MCI
> block. How often do these exceptions happen ? Where can I
> find the association of address blocks and registries ?

I don't have any statistics on geographical exceptions, but the allocation
of IPv4 space by the IANA is available at
http://www.isi.edu/in-notes/iana/assignments/ipv4-address-space.

// Matt Zimmerman Chief of System Management NetRail, Inc.
// mdz@netrail.net sales@netrail.net
// (703) 524-4800 [voice] (703) 524-4802 [data] (703) 534-5033 [fax]

- - - - - - - - - - - - - - - - -

On Tue, 14 May 1996 10:09:44 +1000 Geoff Huston wrote:
>At 6:15 PM 13/5/96, Vadim Antonov wrote:
>>Cacheing appears to be the only sane way to distribute
>>load. You always know where the closest cacheing server is.
>
>not even this is the case - you can guess but that may not always
>work!
>
>If I could _enforce_ caching by traffic interception then
>I could construct vaguely logical cache structures as an ISP.

AS378 does exactly that. It blocks port 80 to abroad and forces everyone
to the Netscape proxy servers it runs (2 of them in a load balance fashion
with DNS set to point to only one if the other fails). AS378 is the
Israeli University academic network of 7 universities.

>
>But relying on users to configure their end systems with the
>identity of the closest cache server using some ill defined
>"network metric" is always going to be an erratic affair at best.

Not unless you don't have a choice :-)

Go to: http://www.tau.ac.il/Shiber/May96/13May96

There you will find their numbers for yesterday. Port 80 now consumes
54% of all traffic.

>
>geoff
>
>

Hank

- - - - - - - - - - - - - - - - -

Hi,

>On a global level, how useful would it be to base the decision
>on whether the client address belongs to the the InterNIC block,
>APNIC or the european registry ?

Little to none.

>I can live with exceptions
>such as an MCI customer in India gets redirected to the
>north american mirror server instead of the one in Singapore
>because their address belongs to an InterNIC allocated MCI
>block.

North America is topologically closer to India than Singapore (you go
through North America to get to Singapore from India).

>How often do these exceptions happen?

Most of the time. The delegation of regional blocks is (arguably) an
administrative convenience. It has nothing to do with the network
topology.

>Where can I
>find the association of address blocks and registries?

Somebody else answered this.

>We are currently working on mirroring a copule of sites on all
>major continents.

In the Asia Pacific Rim region, nearly all the bandwidth goes from AP
region countries to the US directly. This is true due to the
tariffing situation, although it is now beginning to change (some
intra-Asia networks have already been established).

Putting mirrors in countries usually makes sense (particularly when a
country has an Internet exchange or two), but putting them on a
continental basis generally doesn't.

Regards,
-drc

- - - - - - - - - - - - - - - - -

At 07:37 PM 5/15/96 +0900, David R. Conrad wrote:

>
>In the Asia Pacific Rim region, nearly all the bandwidth goes from AP
>region countries to the US directly. This is true due to the
>tariffing situation, although it is now beginning to change (some
>intra-Asia networks have already been established).
>

I think many of us have been *very* vocal supporters of encouraging
creationm of multiple AP NAPs to avoid this form of dementia. :-)

When traffic transits back to the US West Coast to reach another AP
location, this clearly contributes to the overall problem.


>Putting mirrors in countries usually makes sense (particularly when a
>country has an Internet exchange or two), but putting them on a
>continental basis generally doesn't.
>

Agreed. Creating exchange points would also help immensely.

- paul

- - - - - - - - - - - - - - - - -

Paul,

You can be a vocal as you desire, but ultimately from this part of the
globe the dominant factor in any ISP business is the cost of the
International Private Lease. This lease cost is approximately 10 times
the cost of domestic infrastructure.

Now when you construct an IPL in a competitive environment where do
you terminate it? Generally you are loking for an optimal mix of price
and functionality. The observation for the AP region today is that the
cheapest IPL half circuits for the AP region terminate in the
US. Hence Randy's observation. The internal infrastructure within the
AP region happens in a second pass, once the primary objective of
major connectivity is achieved internal infrastructure can be cost
effective if there is internal traffic flow to match.

About the only thing that could hasten regional infrastructure is a
drastic revision of the trading practices and expectation of return on
investment by the undersea cable investors. Exchange points have
little impact per se as they are, in economic terms, a minor aspect of
the entire equation.

Thanks,


Geoff

>
> At 07:37 PM 5/15/96 +0900, David R. Conrad wrote:
>
> >
> >In the Asia Pacific Rim region, nearly all the bandwidth goes from AP
> >region countries to the US directly. This is true due to the
> >tariffing situation, although it is now beginning to change (some
> >intra-Asia networks have already been established).
> >
>
> I think many of us have been *very* vocal supporters of encouraging
> creationm of multiple AP NAPs to avoid this form of dementia. :-)
>
> When traffic transits back to the US West Coast to reach another AP
> location, this clearly contributes to the overall problem.
>
> >Putting mirrors in countries usually makes sense (particularly when a
> >country has an Internet exchange or two), but putting them on a
> >continental basis generally doesn't.
> >
>
> Agreed. Creating exchange points would also help immensely.
>
> - paul
>
>
>

- - - - - - - - - - - - - - - - -

At 06:57 AM 5/16/96 +1000, Geoff Huston wrote:

>Paul,
>
>You can be a vocal as you desire, but ultimately from this part of the
>globe the dominant factor in any ISP business is the cost of the
>International Private Lease. This lease cost is approximately 10 times
>the cost of domestic infrastructure.
>

Geoff, I'm stating what would be ideal, not necessarily practical.

And pointing out what should already be obvious. :-)

- paul


- - - - - - - - - - - - - - - - -

On Thu, 16 May 1996, Geoff Huston wrote:

> Paul,
>
> You can be a vocal as you desire, but ultimately from this part of the
> globe the dominant factor in any ISP business is the cost of the
> International Private Lease. This lease cost is approximately 10 times
> the cost of domestic infrastructure.
>
> Now when you construct an IPL in a competitive environment where do
> you terminate it? Generally you are loking for an optimal mix of price
> and functionality. The observation for the AP region today is that the
> cheapest IPL half circuits for the AP region terminate in the
> US. Hence Randy's observation. The internal infrastructure within the
> AP region happens in a second pass, once the primary objective of
> major connectivity is achieved internal infrastructure can be cost
> effective if there is internal traffic flow to match.
>
> About the only thing that could hasten regional infrastructure is a
> drastic revision of the trading practices and expectation of return on
> investment by the undersea cable investors. Exchange points have
> little impact per se as they are, in economic terms, a minor aspect of
> the entire equation.

Exactly. And all the regional governments should realize that the best way
of shifting traffic from North America to their region is de-regulating
the international telecommunications market, scrapping monopolies and
increasing competition among carriers. That will result much more
effective than policy-making and verbal "declarations of independence".

Enzo
- - - - - - - - - - - - - - - - -