Mailing List Archive

> Has anyone else in the group taken a look at RST's Shambhala?
> I'd be interested in reading about other experiences.

Yes, I've had it running here for a day. Theres a bug in the DBM auth
code RST knows about. Its taken 10k hits without a problem and I'll
make it go live for the telescope Web next week (300k a week).

Mark
Mark J Cox, mark@telescope.org -- URL:http://www.telescope.org/mark.html
University of Bradford, England ---------- tel +44.1274.384070/fax 391333

BTW, I've incorporated Mark's fix for the dbm_auth stuff (thanks, Mark),
and a few other patches into a minor respin of Shambhala; anyone who's
interested in playing with it might as well have the bug fixes. That's
at

ftp://ftp.ai.mit.edu/pub/users/rst/shambhala4.4.tar.Z

(for version 0.4.4... the naming convention here obviously needs a bit
of work).

rst

so what's the situation regarding Shambhala and Apache, are those of
you who have switched to it giving up on Apache and this project ?
If so, do you need a separate list to discuss Shambhala ?

Rob Thau, do you plan to release and support Shambhala by yourself ?,
it looks like that's the plan, if so I wish you luck. If not, how
is this going to relate to the Apache project ?

-=-=-=-=

Feature development on Apache 0.7 has been static. I don't remember
anyone reporting that they were working on any ideas/code. There's
a few suggestions in the bugs mailbox that the group should look at.
Did anyone write an accept serializer for Solaris ? it's a trivial
thing to do with filelocking, but we need someone who uses Solaris
to write and test it (that should limit the number of compatibility
problems that could arise if someone tried to write it for Solaris
but not on a Solaris machine).

0.7.3k looks very stable on the platforms that are running it. I've
had it running at Cardiff in various versions for 3 or four weeks,
with well over a million (cgi-biassed) requests served, there haven't
been any reports of problems, it only needs a nightly SIGHUP to rotate
logfiles. Brian's got it running on apache.org (and elsewhere ?). My
quiet HPUX system is running it, so that's another OS we know works.

Any news if 0.7.3 works with Linux ?, if not are there any debugging
clues to point us in the right direction ?

So who in this list is testing 0.7.3 ?, do we need to bring in testers
from outside of the list ?


rob

Apologies for using short names, (robh = Rob Hartill, rst = Rob Thau).

On Fri, 30 Jun 1995, Rob Hartill wrote:
> Is Shambhala a drop in replacement binary for apache ?

I think that's rst's design, yes. I would even be a fan of ditching the
current configuration file setup for something more logical, but that's a
huge political problem. It looks like all that Shambala lacks right now
is xbithack - and we've discussed better ways of accomplishing that
functionality (if on a request for file.html, file.shtml exists instead,
serve that up server-side-include-parsed).

> What are the feelings of all you people listening to this list ?

I think the effort that has been spent on making 0.7 stable and reentrant
has resulted in very valuable clues and wisdom as to how to properly build a
non-forking server. I also think that the future of web servers (and
apache in particular) deeply involves modular design, something which the
original NCSA base lacks.

I think the best course of action would be for you, robh, to get
shambala and apply any and all wisdom gained from working on 0.7 to
making it a solid product, and something which we could name Apache
0.8. After a few more rounds of feature-adding and debugging (and
maybe? maybe? HTTP-NG?) we can officially release an Apache 1.0. Do *not*
regard this as a vote of no confidence in your work on 0.7, robh - if
anything it's been fun watching you shove an elephant down the eye of a
needle :)

Some questions rst should answer before going further I guess: is making
Shambala the core of the next rev of Apache alright? Is it ready to be
put under the patch database system we have in place?

I'm presuming all the answers would be "yes".

The nice thing about having a new source base would be that we could
divorce the NCSA copyright question entirely (unless there was a good
chunk of snarfed code from 1.3). I don't know if anyone wants to change
it - I certainly have no qualms about leaving it in the public domain.

Those are my thoughts, in any rate. I hope we can find common ground.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

> Whoa! I cant' speak for Rob Thau on this, but since he
> seems to be out of Email range today, I'll add these comments...
>
> If you've taken a look at the Shambhala code, you'll notice that
> RST has been releasing it to the group from day 1 with the Apache
> copyright included. I don't think that he has ever condidered
> *not* offering it back to the group for further development.

I haven't looked at the code. My question was "how does this
relate to apache". If the assumption is that we're all going
to switch to Shambhala, then we all need to decide to abandon the
NCSA 1.3 base, and not continue to try to have two completely
separate servers.

My feelings are that the NCSA base has been tried and tested, and moreover
lots of people like the way it works and is configured.
I'm happy with the NCSA code as a base.

If both apache and Shambhala continue to be developed via this group,
it'll inevitably end in a split.. that's already happening. The question
is, should we all go in one direction, continue as things stand or
Shambahla goes off on its own.

If everyone decides to abandon the current code base, then that's
fine, but I just can't see it happening. We've seen how reluctant some
people can be to switch from NCSA to Apache.

Is Shambhala a drop in replacement binary for apache ?

What are the feelings of all you people listening to this list ?

rob
--
http://nqcd.lanl.gov/~hartill/

On Fri, 30 Jun 1995, Rob Hartill wrote:
> I remember saying words to the effect of "this is what I plan to
> do, stop me if you think this isn't a good idea". Why the hell didn't
> anyone say something ?

I wasn't sure what rst's time frame was, and it seemed reasonable that a
lot of knowlege gained by pushing ahead with the non-forking model could
be applied to shambala anyways. I was surprised at how quickly shambala
came to fruition! Having 0.7x around has definitely helped the
situations I'm involved with, too.

> That said, and after looking at the code and trying (for a few minutes)
> it looks great, and I've no objection to switching to it.
>
> If we'd switched earlier, we'd be in much better shape now. The code
> would have had a much broader testing.. e.g. one of the first random
> URLs I clicked on exposed a minor problem that'll need fixing.

I'm actually planning on putting shambala on links.net today, which will
give it a 250K/day test site (though it doesn't use most of Apache's
advanced features so it'll more useful for basic performance testing than
feature-based bug fixing). As soon as xbithack's in I can set it up on
hyperreal.

Glad we have reached consensus.

Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/

> Apologies for using short names, (robh = Rob Hartill, rst = Rob Thau).
>
> On Fri, 30 Jun 1995, Rob Hartill wrote:
> > Is Shambhala a drop in replacement binary for apache ?
>
> I think that's rst's design, yes.

Hmm, I think were going to have to put this down to lack of
communication on the list. Had I known a month ago what I know now,
I certainly wouldn't have done all that work on 0.7.x.

Maybe it was rst's choice of phrases, such as "garage project" and
it having a different name, maybe I didn't read his mailings
thoroughly enough, maybe they weren't explicit enough,
whatever.. I did not follow what rst was up to. I certainly didn't
realise it was a reworked but functionally compatible
NCSA type server (if I worded that badly, I hope you know what I was
trying to say).

It's a shame that nobody using Shambhala (who must have realised
what was going on) didn't raise these issues weeks ago. I can only
presume that rst was too modest to push Shambhala, or at least
discussion of it, onto use more vigourously.

I remember saying words to the effect of "this is what I plan to
do, stop me if you think this isn't a good idea". Why the hell didn't
anyone say something ? ... did others get the same impression about
rst's work as I did ?. Come on people, if you want to be part of this
group, collaborate !.

end of rant.

That said, and after looking at the code and trying (for a few minutes)
it looks great, and I've no objection to switching to it.

If we'd switched earlier, we'd be in much better shape now. The code
would have had a much broader testing.. e.g. one of the first random
URLs I clicked on exposed a minor problem that'll need fixing.

> regard this as a vote of no confidence in your work on 0.7, robh - if
> anything it's been fun watching you shove an elephant down the eye of a
> needle :)

I've learned a few things, but would much rather not have done
all that work and all those 0.7.2/3 revisions knowing what I know now.

rob ... not a happy camper.

> It's a shame that nobody using Shambhala (who must have realised
> what was going on) didn't raise these issues weeks ago. I can only
> presume that rst was too modest to push Shambhala, or at least
> discussion of it, onto use more vigourously.

You must have missed my "Shambhala Rocks!" post shortly after Shambhala2.
:-)

> I remember saying words to the effect of "this is what I plan to
> do, stop me if you think this isn't a good idea". Why the hell didn't
> anyone say something ? ... did others get the same impression about
> rst's work as I did ?. Come on people, if you want to be part of this
> group, collaborate !.
>

I think the bottom line is that RST moved much faster than anyone
could have guessed. I really just started working seriously with it
last weekend.


> That said, and after looking at the code and trying (for a few minutes)
> it looks great, and I've no objection to switching to it.

I think that is 4 +1's for shambhala! :-)

Having come to that conclusion, I'll spill my guts about known problems.

1. There seems to be a problem in content_neg WRT .var files. I will sort
this out tonight or tomorrow.

2. If I reference a URL via an ISMAP, the CWD is that of 'imagemap' and
not the URL.

Date: Fri, 30 Jun 1995 14:07:46 -0700 (PDT)
From: Brian Behlendorf <brian@organic.com>
Precedence: bulk
Reply-To: new-httpd@hyperreal.com

Weighing in a bit late, but I was off being a grad student for most of
the day... (I locked up myself up in a room to try to get some design
work done on a piece of software rather, well, different from a web
server). I might as well jump in here:

Apologies for using short names, (robh = Rob Hartill, rst = Rob Thau).

On Fri, 30 Jun 1995, Rob Hartill wrote:
> Is Shambhala a drop in replacement binary for apache ?

I think that's rst's design, yes.

It isn't yet, (no XBITHACK, no IdentityCheck, and of course the bugs)
but it has been my intention to make it so, or very nearly so... in
particular, I do regard failures to handle config file directives,
includes directives, or anything else which Apache handles properly as
bugs. There are a few known incompatibilities which I may leave, in
which case they'll have to at least be documented, but they aren't of
the sort which are likely to affect very many people.

(The most serious is that the <Limit> directives which apply to a
particular file are the innermost set, rather than the most
restrictive going all the way down. This would let you, say, deny
access to <Directory /> and reenable it selectively for subdirectories
--- this doesn't represent any loss of flexibility for people who have
set things up sensibly, but it's just *barely* possible that someone
would have to change things to keep the effect of his current setup).

I would even be a fan of ditching the
current configuration file setup for something more logical, but that's a
huge political problem.

Shambhala actually takes some steps in that direction --- srm.conf
directives work in httpd.conf, and vice versa; AddTypes, etc., in
<Directory> sections work better; and almost anything can go in
<VirtualHost> --- but those are compatible extensions. I've got a
compatibility fetish; my personal belief is that imposing any
conversion cost on the users, no matter how slight, is going to
dramatically reduce the acceptability of out stuff, whatever it is.

(On the other hand, I don't think that moving to a different basic
framework for the server would put people off, so long as we do stay
compatible --- those who are inclined to look under the hood will
(hopefully) like what they see, and the majority who aren't just won't
care).

It looks like all that Shambala lacks right now
is xbithack - and we've discussed better ways of accomplishing that
functionality (if on a request for file.html, file.shtml exists instead,
serve that up server-side-include-parsed).

Yeah, but compatibility is a fetish with me. I'll try to support it
as is, though it took me a long while to figure out how to do that
without messing things up too badly...

I think the effort that has been spent on making 0.7 stable and reentrant
has resulted in very valuable clues and wisdom as to how to properly build a
non-forking server. I also think that the future of web servers (and
apache in particular) deeply involves modular design, something which the
original NCSA base lacks.

I think the best course of action would be for you, robh, to get
shambala and apply any and all wisdom gained from working on 0.7 to
making it a solid product, and something which we could name Apache
0.8. After a few more rounds of feature-adding and debugging (and
maybe? maybe? HTTP-NG?) we can officially release an Apache 1.0.

Seconded. BTW, Randy to the contrary, I do regard 0.7 as being a more
mature piece of code than Shambhala 0.4.4 --- at this point, I'm
fairly confident that the basic Shambhala design is sound, but there
are still probably enough silly typos lurking (like the ones Randy
himself keeps finding(!)) that I do think 0.7.x is the best bet for a
release in the near-to-immediate term.

(BTW, I have been tempted to mention Shambhala to people in threads
like the "what to do about logging" discussion that periodically crops
up, most lately on www-talk, just because, along with NetSite, it does
add a new option, which is to load the new logging methods into the
server as new modules. But this would only be with the clear
disclaimer that the current code is incomplete, undebugged, and not to
be relied upon).

Some questions rst should answer before going further I guess: is making
Shambala the core of the next rev of Apache alright?

Fine with me.

Is it ready to be
put under the patch database system we have in place?

I've already started taking patches, but there are still a few things
I'd like to do with it, including possibly one fairly pervasive change
to the module API, before giving up control completely (my TODO list
is in the more recent Shambhala snapshots, if anyone wants a look; if
you think anything in particular would be a lousy idea, feel free to
holler --- NB some of that stuff, like multithreading, is obviously
past the first release anyway).

The nice thing about having a new source base would be that we could
divorce the NCSA copyright question entirely (unless there was a good
chunk of snarfed code from 1.3). I don't know if anyone wants to change
it - I certainly have no qualms about leaving it in the public domain.

Sorry, but there was --- most of the server core is a wall-to-wall
rewrite, but a lot of the code in the modules bears the stamp of the
NCSA sources. (Warming over the includes and directory indexing code
was a lot easier than redoing it from scratch, given my goal of
staying compatible).

Those are my thoughts, in any rate. I hope we can find common ground.

Same here.

rst

> You must have missed my "Shambhala Rocks!" post shortly after Shambhala2.
> :-)

I saw it, but all I read into it was that it was working well.

So what's the plan. Are we going to feed Rob T bug reports/patches/requests
or will we return to the patch'n'vote approach we once had. The only
reason it slipped out of fashion with 0.7.x was because I was the only
person in the mood to work on it and I was just replacing code to make
it pre-forking-safe. I don't want to tread on Rob's toes, if
he needs more time to work on the code alone, then that's okay, but
I'm willing to put some time in to do some coding - if indeed there's
any extra coding that needs doing... it'll take some time for everyone to
find their way around a new code base whatever happens, so Rob has some
legroom there.

Oh, and before we proceed, let's make sure everyone is happy with
the direction were heading. Does anyone have a problem with the
assumed new direction ?

-=-=

BTW, the minor problem I hinted at earlier was with a CGI script which
spawned a child to do some real work, and the parent exitted.
With Apache, the connection is immediately closed once the parent CGI exits,
with shambhala, the connection stays open until the CGI child exits.
Oh, and one other thing, I had an AddType in one of my config files, and
it had parameters after content-type, this gave a error message.


rob

> 2. If I reference a URL via an ISMAP, the CWD is that of 'imagemap' and
> not the URL.
>
> Could you be a bit more specific about what gets zorched to cgi-bin
> instead of the directory containing the map --- is it child processes,
> relative SSI <!--#include--> directives, or something else? Thanks.
>
> rst

Imagemaps that are <!--#included> in a page which have map entries
referencing relative files to the directory containing the page fail
to find the referenced URL because the current working directory
becomes that of the 'imagemap' program.

I also just verified that this has nothing to do with whether the
ISMAP is <!--#included> or not.

Is that better? Let me know if you need more.

From: Rob Hartill <hartill@ooo.lanl.gov>
Date: Fri, 30 Jun 95 16:48:01 MDT

Maybe it was rst's choice of phrases, such as "garage project" and
it having a different name, maybe I didn't read his mailings
thoroughly enough, maybe they weren't explicit enough,
whatever.. I did not follow what rst was up to. I certainly didn't
realise it was a reworked but functionally compatible
NCSA type server (if I worded that badly, I hope you know what I was
trying to say).

Sigh... I wish you would have looked... (WRT the lingo, btw ---
"garage project" doesn't necessarily mean small, at least in my
lexicon; people do build airplanes in their garages. It's just that
most of the people that start on something like that never finish; I
wasn't sure if, or when, I was going to be finished with my thing
either. Sorry about any misunderstanding).

It's a shame that nobody using Shambhala (who must have realised
what was going on) didn't raise these issues weeks ago. I can only
presume that rst was too modest to push Shambhala, or at least
discussion of it, onto use more vigourously.

Hmmm... when I first put the code up for group viewing (which was
about as soon as I had code to show; it still didn't really work worth
a damn), I did describe it as "a direction... that the group might
want to pursue", and tried to talk up its virtues a bit. I really
don't feel comfortable pushing something more aggressively than that.
Maybe that's a problem of mine; it certainly seems to have caused
trouble here. Still, I had hoped people would take that as a cue to
at least look at the thing... maybe not. (BTW, I do have this voice
in the back of my head which keeps reminding of how much I have to be
modest about... ;-).

Also, keep in mind the timing here... "the people using Shambhala"
have been me for a week and a half, and Mark and Randy for a week
less. Scale your expectations accordingly...

That said, and after looking at the code and trying (for a few minutes)
it looks great, and I've no objection to switching to it.

Glad you like it. Sorry about the other stuff...

rst

From: Rob Hartill <hartill@ooo.lanl.gov>
Date: Fri, 30 Jun 95 21:25:00 MDT

> You must have missed my "Shambhala Rocks!" post shortly after Shambhala2.
> :-)

I saw it, but all I read into it was that it was working well.

So what's the plan. Are we going to feed Rob T bug reports/patches/requests
or will we return to the patch'n'vote approach we once had.

Hmmm... past a certain point, the patch-n-vote business might be a
good thing to have regarding changes to the server core, and at least
some of the messier modules. If the stuff on my TODO list is
offensive to anybody, it might even be appropriate to restart that
process now; however, that stuff will happen somewhat faster if I
don't have to package it up incrementally, so I'd rather go that way
for a little bit more unless someone really does have serious
objections. (On the other hand, if you find a bug and fix it, please
don't let that discourage you from sending me the code!)

However, half the point of Shambhala is to make some of that process
obsolete, by allowing stuff which is not universally applicable
(e.g. database back-ends), controversial, or just half-baked, to be
shipped anyway as optional modules. (Of course, the config script
which makes it easy for webmasters to pick and choose among the
options has yet to be written, but that's the intention). Having a
middle way between tossing something in for everybody, and leaving it
out, might obviate some of the nastier votes we've had...

I don't want to tread on Rob's toes, if
he needs more time to work on the code alone, then that's okay, but
I'm willing to put some time in to do some coding - if indeed there's
any extra coding that needs doing... it'll take some time for everyone to
find their way around a new code base whatever happens, so Rob has some
legroom there.

Oh, there's no shortage of coding that needs doing...

-=-=

BTW, the minor problem I hinted at earlier was with a CGI script which
spawned a child to do some real work, and the parent exitted.
With Apache, the connection is immediately closed once the parent CGI exits,
with shambhala, the connection stays open until the CGI child exits.

Hmmm... here's a hypothesis. Shambhala is not as scrupulous about
closing spare file descriptors as it ought to be, and it does not
relocate the client file descriptors down to stdin and stdout
immediately. So, the CGI script gets the direct connection back to
the client set, and the CGI child inherits it; as long as *any*
process has the socket open, the connection stays alive. Looks like
I'll need a hook into alloc.c to allow child processes to clean up
their descriptor state before the exec...

Oh, and one other thing, I had an AddType in one of my config files, and
it had parameters after content-type, this gave a error message.

Could you send me the config line that breaks? Thanks.

rst

Date: Fri, 30 Jun 1995 19:08:28 -0500
From: Randy Terbush <randy@zyzzyva.com>

1. There seems to be a problem in content_neg WRT .var files. I will sort
this out tonight or tomorrow.

Hmmm... already sent this privately, but for anyone else who's
interested, I think this *might* improve matters:

*** mod_negotiation.c~ Sun Jun 25 17:18:03 1995
--- mod_negotiation.c Sat Jul 1 13:46:05 1995
***************
*** 809,815 ****

int is_identity_encoding (char *enc)
{
! return (!enc || !strcmp (enc, "7bit") || !strcmp (enc, "8bit")
|| !strcmp (enc, "binary"));
}

--- 809,815 ----

int is_identity_encoding (char *enc)
{
! return (!enc || !enc[0] || !strcmp (enc, "7bit") || !strcmp (enc, "8bit")
|| !strcmp (enc, "binary"));
}


2. If I reference a URL via an ISMAP, the CWD is that of 'imagemap' and
not the URL.

Could you be a bit more specific about what gets zorched to cgi-bin
instead of the directory containing the map --- is it child processes,
relative SSI <!--#include--> directives, or something else? Thanks.

rst

Cc: rst@ai.mit.edu
X-Uri: http://www.zyzzyva.com/
Date: Sat, 01 Jul 1995 14:14:55 -0500
From: Randy Terbush <randy@zyzzyva.com>

Imagemaps that are <!--#included> in a page which have map entries
referencing relative files to the directory containing the page fail
to find the referenced URL because the current working directory
becomes that of the 'imagemap' program.

Sorry --- maybe I'm being thickheaded about this, but I'm still not
quite sure what's going on here. When you say "the referenced URL",
I'm not sure who's referring to it (specifically, whether it's the URL
in the map itself, or something else). Also, it isn't quite clear how
you determined that the CWD setting is the problem. (Only the CGI
child running the imagemap script ought to have CWD set to the
directory containing 'imagemap' --- which is something the old NCSA
code did as well --- if the server isn't properly handling local
redirects served up by the 'imagemap' script, the problem may be
something else).

rst

Robbie the robot:
> Hmm, I think were going to have to put this down to lack of
> communication on the list. Had I known a month ago what I know now,
> I certainly wouldn't have done all that work on 0.7.x.

[snip]


> rob ... not a happy camper.
>

Aaaah, big hugs for rob. Don't worry, we'll try extra hard to flame you
from now on. ;)

Anyway, you know more about integrating 0.7 than anyone else does now.
I'm glad someone's making an effort ;)

Ay.

Bad boy Rob H sez:
> If everyone decides to abandon the current code base, then that's
> fine, but I just can't see it happening. We've seen how reluctant some
> people can be to switch from NCSA to Apache.
>
> Is Shambhala a drop in replacement binary for apache ?
>
> What are the feelings of all you people listening to this list ?
>
> rob
> --
> http://nqcd.lanl.gov/~hartill/
>

I'd like whatever comes out of all this to be back-compatible with NCSA,
just because everybody's uncle's dog is writing applications now with
that server platform. This doesn't mean that I want the core of the
server to be an NCSA 1.3 clone, I'm happy with just keeping the look
and feel of the thing, er, NCSAesque.

Shambala seems to be a good starting point for a new core for a server
with Apache's functionality [.I just put shambala-4.4 on my FreeBSD 2.0.5
box with the minimum of fuss in porting, just a couple of 1 line changes].

The Apache 0.6 server is a huge improvement on NCSA 1.3 and is still
my preferred choise, er, cuz I'm basically a luddite.

The Apache 0.7 is a faster option, though there still seem to be some
doubts about it's efficacy on small systems where tons of processes
might cause a problem. And am I right in thinking that the beast
needs to be SIGged every once in a while just to stop it eating
the kernel?!

[.Anyway, I've finally got my home system sorted out so I'll be in a much
better position to mess about with server design rather than sit on the
fence twiddling my thumbs, trying to sound like I know what I'm
talking about.]

Cheers,
Ay.

Andrew Wilson URL: http://www.cm.cf.ac.uk/User/Andrew.Wilson/
Elsevier Science, Oxford Office: +44 01865 843155 Mobile: +44 0589 616144