Mailing List Archive

On Wed, 26 Apr 1995, Rob Hartill wrote:
> > 2. Discussion group for large web sites (NCSA C&C would like
> > to talk with administrators of sites which get more
> > than 1 million hits a day)
>
> I can only think of Netscape, NCSA and GNN who'd probably fit
> into this category. Are there any others ?

HotWired got close to 600K the other day - throw in a few extraneous
redirects and I'm sure we could go over the top :)

But yeah, I'd lower it to 500K or something....or not have a markoff at
all necessarily.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

> 2. Discussion group for large web sites (NCSA C&C would like
> to talk with administrators of sites which get more
> than 1 million hits a day)

I can only think of Netscape, NCSA and GNN who'd probably fit
into this category. Are there any others ?

> GNN could be in the million+ range
> anyway if you count all the mirror sites, but by that standard, I suppose
> it's possible that the movie database counts as well.

I don't even think that with all the mirror sites combined that it
gets even 1/2 million a day.

That might change soon though. AOL are making enquiries to mirror the
database, no strings attached, as part of a project they working on
called "Giving something back to the internet". It's all unofficial
right now, so don't make it public.

> Another possible million+ site is sunsite-UNC,

Hmm, that'd explain their slow response times.

Anyway, there doesn't seem to be many sites in the the 1 million/day
club, so the proposed NCSA meeting would be a quiet one :-)


robh

On Apr 26, 2:44pm, Rob Hartill wrote:
} Subject: Re: restructuring the server (was Re: virtual host again)
>
> Anyway, there doesn't seem to be many sites in the the 1 million/day
> club, so the proposed NCSA meeting would be a quiet one :-)
>
> robh
}-- End of excerpt from Rob Hartill

Well that was just the million+ group was just one of the
suggested BOF type meetings requested. The administration
and CGI tutorials are another set of meetings suggested. I
figured those in this group would be most interested in
the third set of meetings where we listen to gripes, discuss
requested features and present our grand plans for a
Dynamic Object Repository (DOR) as a bull's eye for y'all
to shoot at. In particular we would like help developing
an Application Program Interface (API) to replace/enhance
CGI. I'd like at least one discussion about log formats
(1 vs mult. log files, limited configurability in format,
etc.).

-Beth Frank
efrank@ncsa.uiuc.edu

Yahoo was getting over a million hits a day when they stopped advertising
their aggregate traffic, and the graph of total hits over the previous
hundred days still looked like a perfect exponential curve (absent the
late-December dip due to the holidays).

Re: GNN, I seem to remember that at the time when they (and everyone else)
encountered the listen-queue-jam problem for the first time, and started
to discuss it on www-talk, the webmaster there mentioned in passing his
daily hit rate, and at that point, it was only something like 200-300K.
(Of course, my memory could be wrong). GNN could be in the million+ range
anyway if you count all the mirror sites, but by that standard, I suppose
it's possible that the movie database counts as well.

(Another possible million+ site is sunsite-UNC, and I suppose it's possible
that IUMA makes the cut as well, particularly if mirrors are added in).

rst

On Apr 26, 2:25pm, Rob Hartill wrote:
} Subject: Re: restructuring the server (was Re: virtual host again)
>
> > 2. Discussion group for large web sites (NCSA C&C would like
> > to talk with administrators of sites which get more
> > than 1 million hits a day)
>
> I can only think of Netscape, NCSA and GNN who'd probably fit
> into this category. Are there any others ?
>
>
>
}-- End of excerpt from Rob Hartill

I'm told the Whitehouse gets somewhere in that range, and
that Yahoo is over a million. As someone earlier pointed
out an absolute cut off point is probably not useful as long
as it is made clear that the discussion will focus on the
problems related to very large, very busy sites.

-Beth Frank
efrank@ncsa.uiuc.edu

From: efrank@ncsa.uiuc.edu (Elizabeth Frank)
Date: Wed, 26 Apr 1995 18:38:38 -0500

I'm told the Whitehouse gets somewhere in that range, and
that Yahoo is over a million.

Yahoo was over a million the last time they were giving out the
numbers, and I don't think they're getting any less now. The White
House is another story. I just checked with someone who sees their
log summaries, and while they're understandably sensitive about the
actual numbers, he would allow me to say that a million hits a day is
well in excess of their actual load.

(The White House setup was indeed designed to handle very high levels
of traffic, but it hasn't yet been pushed close to the limit).

rst

> > 2. Discussion group for large web sites (NCSA C&C would like
> > to talk with administrators of sites which get more
> > than 1 million hits a day)
>
> I can only think of Netscape, NCSA and GNN who'd probably fit
> into this category. Are there any others ?

Probably Penthouse, but who gives a damn. ;)

Ay.

> At NCSA we are in the architure stage of redesigning the server.
> If you would like make requests about structure, interfaces
> internal or external, etc. now is the time to do it.
As I already mentioned once I would love to see multithreaded server
(POSIX Threads based for example). The very first requirement is that
all procedures should be reentrant. And that's something I would
suggest if you want to redesign whole thing. Once everything is
reentrant, multithreading comes at "no time". Once it's multithreaded,
it's whole new world with cacheing possibilities starting from access
evaluations and finishing at pages themselves.

BTW, guys! Has anybody ever considered to cache ~user aliases
translations and access evaluations? Latter should do really nice job
for latex2html documents and other <IMG> junkies.

Andy.

Date: Thu, 27 Apr 1995 09:36:24 +0200
From: Andy Polyakov <appro@fy.chalmers.se>

BTW, guys! Has anybody ever considered to cache ~user aliases
translations and access evaluations? Latter should do really nice job
for latex2html documents and other <IMG> junkies.

Andy.

In the non-forking server, we actually do that already, on a
per-process basis (unless Rob's changed it). Surprise!

How did this happen? Well, every /~foo is entered into the aliases
table as it's encountered, for the sake of unmunge_name. When the
same server process sees /~foo again (with the same user foo), the
alias is used and the passwd-file lookup is bypassed.

For NCSA 1.4, this actually caused a very nasty bug, because the
aliases were stored in a fixed-size table which overflowed after
seeing enough /~foos. As a result, NCSA has to ditch these new
aliases after every transaction to keep them from piling up.

Apache, however, stores them in a malloc'ed linked list, so overflow
of a fixed-length table is not a problem. And so, when I was first
merging Brandon's non-forking code with our own stuff, I decided not
to keep the /~foo aliases around, precisely for this caching effect.

(Incidentally, this cache is flushed along with the rest of the alias
database when the rereads the config files, as it properly should be).

Serendipity is fun when it happens...

rst

> BTW, guys! Has anybody ever considered to cache ~user aliases
> translations and access evaluations? Latter should do really nice job
> for latex2html documents and other <IMG> junkies.
>
> In the non-forking server, we actually do that already, on a
> per-process basis (unless Rob's changed it). Surprise!
No, I'm not suprised!
>
> How did this happen? Well, every /~foo is entered into the aliases
> table as it's encountered, for the sake of unmunge_name. When the
> same server process sees /~foo again (with the same user foo), the
> alias is used and the passwd-file lookup is bypassed.
Right! Shouldn't they be obsoleted in some time? Just in case you move
user's home directory over? And what about access evaluation cacheing?
Could be usefull as well, I beleive...

Andy.

| From: rst@ai.mit.edu (Robert S. Thau)
|
| (Another possible million+ site is sunsite-UNC, and I suppose it's possible
| that IUMA makes the cut as well, particularly if mirrors are added in).

just for the record, we're just now hitting the 250k/day mark on our
primary server. the mirrors are waiting to be updated with 2.0 but are
rather low accesses right now as a result of being a bit out of date. we
are pegging our T1, so i expect we will need to update that before we
move to the million mark. anyone have a spare T3? ;}

jon r. luini, iuma kaiser - falcon@iuma.com IUMA Info: info@iuma.com
.___ ____ ___ _____ _____
| | | \ \ / _ \ the net's first hi-fi music archive
| | | / Y \/ /_\ \ .:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:
| | | / | \ | \ The Internet Underground Music Archive
|___|______/____|__ /___|__ / bands/music/labels/images/lawn darts
===================\/=======\/============================================
URL - http://www.iuma.com/ Phone: (408) 426-4862

I have a fairly clean T1 here (just got another from UUNET) and our
access is quite good. Would be happy to mirror anything if necessary.

Roderick Murchison, Jr. Newbridge Networks, Inc.
----------------------- Network Engineering
murchiso@newbridge.com 460 Herndon Parkway
http://www.vivid.newbridge.com/~murchiso Herndon, VA 22070-5241
(703) 318-5759 [office]

On Fri, 28 Apr 1995, jon r. luini wrote:

> | From: rst@ai.mit.edu (Robert S. Thau)
> |
> | (Another possible million+ site is sunsite-UNC, and I suppose it's possible
> | that IUMA makes the cut as well, particularly if mirrors are added in).
>
> just for the record, we're just now hitting the 250k/day mark on our
> primary server. the mirrors are waiting to be updated with 2.0 but are
> rather low accesses right now as a result of being a bit out of date. we
> are pegging our T1, so i expect we will need to update that before we
> move to the million mark. anyone have a spare T3? ;}
>
> jon r. luini, iuma kaiser - falcon@iuma.com IUMA Info: info@iuma.com
> .___ ____ ___ _____ _____
> | | | \ \ / _ \ the net's first hi-fi music archive
> | | | / Y \/ /_\ \ .:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:.:
> | | | / | \ | \ The Internet Underground Music Archive
> |___|______/____|__ /___|__ / bands/music/labels/images/lawn darts
> ===================\/=======\/============================================
> URL - http://www.iuma.com/ Phone: (408) 426-4862
>
>