Mailing List Archive

On Sunday, Nov 10, 2002, at 14:28 US/Eastern, Dennis Cartier wrote:
> I switched to the CVS version a few days ago and disabled connection
> pools
> as I observed this behaviour and mistook it for something else. So this
> probably affects the normal release version, irrespective of
> connection pool
> usage.
>
> Anyone have any ideas about this?

A few. It is possible that mod_backhand will need to check the
validity of the request in the post read request handler, but that
seems dumb -- (the job of apache IMHO). If I can replicate this
problem, it should be fairly easy to correct.

> MulticastStats internal:80 10.1.1.255:4445,1
> AcceptStats 10.1.1.0/24
> BackhandSelfRedirect ON
> BackhandConnectionPools OFF

You have connection pools off and self redirection on. What is
"internal"? 127.0.0.1? And are you _sure_ that is a different apache
instance with mod_backhand _not_ running? I would think that some
really screwy things could happen in that case. FYI (using "internal"
there is only supported in versions after 1.2.1).

> [Sun Nov 10 14:14:31 2002] [error] mod_backhand: Check 6 -- headers(-1)
> [Sun Nov 10 14:14:31 2002] [error] mod_backhand: Tried... failed

Hmm. Check 6 failure means that it couldn't get the headers back from
the backend server it attempted to hand the request to... So C
requests from A, A decides B is better, A connects to B, asks B the
same request and B hands back bad data.

This should cause mod_backhand to retry:
9 times if connection pooling is on
0 times is connection pooling is off

When these failures happen that many subsequent times (10 and 1,
respectively) we fail to local handling.

Local handling looks something like this in 1.2.1:
ap_internal_redirect(r->uri, r);

This has a bug in it. If somehow the internal redirect chooses to
backhand the request again (which I still haven't figured out _why_
this would happen), then we get an infinite loop.

In the CVS version the local handling looks like this:
if(!ap_table_get((r->main)?r->main->notes:r->notes,
"BackhandHandlerExecuted")) {
ap_table_setn((r->main)?r->main->notes:r->notes,
"BackhandHandlerExecuted", "Done");
ap_internal_redirect(r->uri, r);
}

This should check to see if Backhand has run already and if not mark
the request indicating that Backhand has run once and then do the
redirect -- preventing the loop.


--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------

I have a bit more info regarding this. I tried removing various modules
from the scenario (mod_ssl, mod_gzip, etc.), but I never found any
combination
that allowed a corrupt request to be entered without causing the cascade.

>On Sunday, Nov 10, 2002, at 14:28 US/Eastern, Dennis Cartier wrote:
>> I switched to the CVS version a few days ago and disabled connection
>> pools
>> as I observed this behaviour and mistook it for something else. So this
>> probably affects the normal release version, irrespective of
>> connection pool
>> usage.
>>
>> Anyone have any ideas about this?
>
>A few. It is possible that mod_backhand will need to check the
>validity of the request in the post read request handler, but that
>seems dumb -- (the job of apache IMHO). If I can replicate this
>problem, it should be fairly easy to correct.
>
>> MulticastStats internal:80 10.1.1.255:4445,1
>> AcceptStats 10.1.1.0/24
>> BackhandSelfRedirect ON
>> BackhandConnectionPools OFF
>
>You have connection pools off and self redirection on. What is
>"internal"? 127.0.0.1? And are you _sure_ that is a different apache
>instance with mod_backhand _not_ running? I would think that some
>really screwy things could happen in that case. FYI (using "internal"
>there is only supported in versions after 1.2.1).

I have connection pools off for no good reason. I mistakenly thought a
couple
of days ago that the CVS version was messing up with them on. It was not, it
was this other cascade bug. I guess I can re-enable them now :)

SelfRedirection is on to make sure that ALL requests getmarked as backhand
proxied
so I can use it as a flag to log it with mod_log_spread, and compress it
with
mod_gzip. I have to keep mod_ssl and mod_gzip away from each other as they
do not
mix well. Since the backhanding takes place in the clear, this works as long
as every
request is proxied.

I share my Apache configs between all my nodes, so I use the name 'internal'
which
gets resolved through the hosts file on each machine to be the address of
the inside
interface. Each node is multihomed on a public subnet, and a private one.
Internal is
its private address.

>
>> [Sun Nov 10 14:14:31 2002] [error] mod_backhand: Check 6 -- headers(-1)
>> [Sun Nov 10 14:14:31 2002] [error] mod_backhand: Tried... failed
>
>Hmm. Check 6 failure means that it couldn't get the headers back from
>the backend server it attempted to hand the request to... So C
>requests from A, A decides B is better, A connects to B, asks B the
>same request and B hands back bad data.
>
>This should cause mod_backhand to retry:
> 9 times if connection pooling is on
> 0 times is connection pooling is off
>
>When these failures happen that many subsequent times (10 and 1,
>respectively) we fail to local handling.
>
>Local handling looks something like this in 1.2.1:
>ap_internal_redirect(r->uri, r);
>
>This has a bug in it. If somehow the internal redirect chooses to
>backhand the request again (which I still haven't figured out _why_
>this would happen), then we get an infinite loop.
>
>In the CVS version the local handling looks like this:
> if(!ap_table_get((r->main)?r->main->notes:r->notes,
> "BackhandHandlerExecuted")) {
> ap_table_setn((r->main)?r->main->notes:r->notes,
> "BackhandHandlerExecuted", "Done");
> ap_internal_redirect(r->uri, r);
> }
>
>This should check to see if Backhand has run already and if not mark
>the request indicating that Backhand has run once and then do the
>redirect -- preventing the loop.

I noticed one other thing. Under 1.2.1 when this continuous cascade
occurred,
the logs showed entries with "backhand:backhand:backhand ..." for hundreds
of
lines. Todays logs (from CVS version) still have a few showing multiple
flags
attached:
==== Example ====
[Sun Nov 10 10:34:03 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:backhand:backhand:backhand:backhand:backhand:/icons/backhand.jpg,
backhand/redirect }


Dennis

>
>
>>--
>Theo Schlossnagle
>Principal Consultant
>OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
>Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
>1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
>2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

On Sunday, Nov 10, 2002, at 14:28 US/Eastern, Dennis Cartier wrote:
> [ .. snip .. ]
> just a corrupt request. I exited the telnet and was shocked to see
> every
> machine in the cluster immediately spike to 150 (my maximum children
> setting) in 2 seconds. Then the apache children just stayed in 'W'
> mode.
>
> [ .. snip .. ]
> mod_Backhand/mod_SSL/mod_GZIP/mod_log_spread all working together
> properly.
>
> [ .. snip .. ]
> I switched to the CVS version a few days ago and disabled connection
> pools
> as I observed this behaviour and mistook it for something else. So this
> probably affects the normal release version, irrespective of
> connection pool
> usage.

Okay. I have a very similar set up to you on this end sans the
mod_gzip. I have mod_backhand/mod_ssl/mod_log_spread/mod_rewrite all
working smoothly together (mod_rewrite + mod_backhand is a pain in the
ass). Anyway, I attempted to replicate you error by issuing a request
like: "/lalala.html" to my server (no GET or HTTP/1.0) to see if I
could "throw it through a loop". Alas, I got the expected error back
-- being "malformed request" and everything was fine.

However, the next morning, I was pages about Spread problems on that
machine and logged into fine the load on its way up from 2 to 158. I
did notice two things

(1) one of the back-end machines was causing a front-end mod_backhand
error. As the load was 158 when I logged in there was about zero
troubleshooting I could do. But I did have quite a few: "could not get
valid connection -- forced local" in my error log, which would lead me
to believe that there is some type of loop.

(2) Spread was hosed. I could spuser in, but when I joined a group it
took over 20 seconds to return my membership message -- even on the
other machines in the cluster with loads between 0 and 1.

So, my guess is that it may be caused by (1), but the several severe
problems I have had in the (2) arena have me a little confused as to
which is the cause and which the effect. For the case of (1), I have
patched mod_backhand.c to try to prevent those loops (and detect them).
If the patch works, I will commit it to CVS. For the bold, a patch of
these changes (against mod_backhand.c revision 1.50) is below.

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------



RCS file: /storage/cvs/jesus/mod_backhand/mod_backhand.c,v
retrieving revision 1.50
diff -u -3 -r1.50 mod_backhand.c
--- mod_backhand.c 13 Sep 2002 00:40:42 -0000 1.50
+++ mod_backhand.c 13 Nov 2002 05:35:16 -0000
@@ -913,6 +913,11 @@
ap_table_setn((r->main)?r->main->notes:r->notes,
"BackhandHandlerExecuted", "Done");
ap_internal_redirect(r->uri, r);
+ } else {
+ ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, NULL,
+ "mod_backhand: "
+ "attempted mod_backhand redirection twice. BUG.");
+ return DECLINED;
}
if(loglevel & MBLL_MBCS3) {
ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, NULL,
@@ -1335,7 +1340,7 @@
const char *rhost = ap_table_get(subr->notes,
"Backhand-Redirect-Host");
if(rhost) ap_table_set(r->notes, "Backhand-Redirect-Host",
rhost);
r->uri = ap_pstrcat(r->pool, "backhand:", r->uri, NULL);
- r->filename = ap_pstrdup(r->pool, r->uri);
+ r->filename = ap_pstrcat(r->pool, "backhand:", r->files, NULL);
r->handler = "backhand-redirection";
if(loglevel & MBLL_DCSN1) {
ap_log_error(APLOG_MARK, APLOG_NOTICE|APLOG_NOERRNO, NULL,
@@ -1379,6 +1384,8 @@
return DECLINED;
}
memmove(r->uri, &r->uri[9], strlen(&r->uri[9])+1);
+ if(strncmp(r->filename, "backhand:", 9)==0)
+ memmove(r->filename, &r->filename[9], strlen(&r->filename[9])+1);
return OK;
}

Hello,

Thanks for the help Theo. I was not able to compile once the patch was
applied.
The r->files in the line below is not defined in my version

+ r->filename = ap_pstrcat(r->pool, "backhand:", r->files, NULL);

I have a CVS version from late last week. Did you perhaps mean r->filename?

I tried using r->filename for the undefined structure memeber, but the
cascade still starts immediately.

I have some more info though (when compiled with r->filename!):

1. The error message you added in the patch never gets triggered.
2. The looping seems to initially be a tight loop, looping after
"mod_backhand: Check 3 --" is displayed in the log. Eventually (when all the
children started?) the loop seems to grow to include the check 6 portion and
logs a failure after that point. I have included the transition point of the
logs below.
3. I tried adjusting the <LocationMatch> that is backhanding everything from
".*" to "/.+" to see if it would somehow affect the looping. With this
setting the cascade deos not start, but the child segfaults and exits. I
tried using gdb to extract some useful info from apache, but was unable to
acheive anything.


Dennis


=== Logfile Loop Transition ===
sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21


[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 3 --

sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21


[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:55 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:55 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:55 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [notice] (Type =>) = *{ backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:55 2002] [error] mod_backhand: Check 3 --

sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21


[Wed Nov 13 10:09:56 2002] [error] server reached MaxClients setting,
consider raising the MaxClients setting
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 6 -- headers(-1)
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Tried... failed
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 3 --

sdfsdfsdf
Connection: Keep-Alive
BackhandProxied: 10.1.1.21


[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 6 -- headers(-1)
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Tried... failed
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Translate =>) = { /, (null), (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { (null), /, (null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [notice] (Translate <=) = { backhand:/,
backhand-redirection, (null) }
[Wed Nov 13 10:09:59 2002] [notice] (Type <=) = { backhand-redirection, /,
(null) }
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:09:59 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:09:59 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:09:59 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/, backhand/redirect }
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 1
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 2
[Wed Nov 13 10:09:59 2002] [error] mod_backhand: Check 3 --


=== Segmentation Fault Example ===
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] Create log to group apache for daemon 0
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- UnixSocketDir set to
/var/backhand/backhand
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- Broadcast
10.1.1.255:4445 added
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- Multicast accept
10.1.1.0/24
[Wed Nov 13 10:32:51 2002] [notice] mod_backhand -- Multicast accept
10.1.1.0/24
[Wed Nov 13 10:32:51 2002] [notice] backhand_init(16615) spawning moderator
(PID 16616)
[Wed Nov 13 10:32:52 2002] [notice] waiting for mod_backhand moderator to
start
[Wed Nov 13 10:32:52 2002] [notice] mod_backhand moderator ready to go
[Wed Nov 13 10:32:52 2002] [notice] Apache/1.3.27 (Unix) mod_gzip/1.3.26.1a
mod_backhand/1.2.1 mod_ssl/2.8.12 OpenSSL/0.9.6g mod_jk/1.2.0 configured --
resuming normal operations
[Wed Nov 13 10:32:52 2002] [notice] Accept mutex: sysvsem (Default: sysvsem)
[Wed Nov 13 10:33:09 2002] [notice] (Translate <=) = { /index.html, (null),
(null) }
[Wed Nov 13 10:33:09 2002] [notice] (Translate =>) = { /index.html, (null),
(null) }
[Wed Nov 13 10:33:09 2002] [notice] (Type <=) = { (null), /index.html,
(null) }
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null) [byAge(5)]
(1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null)
[byRandom(NULL)] (1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null)
[byLogWindow(NULL)] (1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] Func executed for (null)
[byBusyChildren(NULL)] (1 -> 1)
[Wed Nov 13 10:33:09 2002] [notice] New server list: [ 0 ]
[Wed Nov 13 10:33:09 2002] [notice] All funcs executed -> 10.1.1.21
[Wed Nov 13 10:33:09 2002] [notice] (Type =>) = *{ backhand-redirection,
backhand:/index.html, backhand/redirect }
[Wed Nov 13 10:33:09 2002] [notice] (Translate <=) = {
backhand:/backhand:/index.html, (null), (null) }
[Wed Nov 13 10:33:09 2002] [notice] child pid 16618 exit signal Segmentation
fault (11)


-----Original Message-----
From: backhand-users-admin@lists.backhand.org
[mailto:backhand-users-admin@lists.backhand.org]On Behalf Of Theo
Schlossnagle
Sent: Wednesday, November 13, 2002 12:39 AM
To: backhand-users@lists.backhand.org
Cc: Theo Schlossnagle
Subject: Re: [m_b_users] Corrupt request causes Denial of Service
cascade



On Sunday, Nov 10, 2002, at 14:28 US/Eastern, Dennis Cartier wrote:

Okay. I have a very similar set up to you on this end sans the
mod_gzip. I have mod_backhand/mod_ssl/mod_log_spread/mod_rewrite all
working smoothly together (mod_rewrite + mod_backhand is a pain in the
ass). Anyway, I attempted to replicate you error by issuing a request
like: "/lalala.html" to my server (no GET or HTTP/1.0) to see if I
could "throw it through a loop". Alas, I got the expected error back
-- being "malformed request" and everything was fine.

However, the next morning, I was pages about Spread problems on that
machine and logged into fine the load on its way up from 2 to 158. I
did notice two things

(1) one of the back-end machines was causing a front-end mod_backhand
error. As the load was 158 when I logged in there was about zero
troubleshooting I could do. But I did have quite a few: "could not get
valid connection -- forced local" in my error log, which would lead me
to believe that there is some type of loop.

(2) Spread was hosed. I could spuser in, but when I joined a group it
took over 20 seconds to return my membership message -- even on the
other machines in the cluster with loads between 0 and 1.

So, my guess is that it may be caused by (1), but the several severe
problems I have had in the (2) arena have me a little confused as to
which is the cause and which the effect. For the case of (1), I have
patched mod_backhand.c to try to prevent those loops (and detect them).
If the patch works, I will commit it to CVS. For the bold, a patch of
these changes (against mod_backhand.c revision 1.50) is below.

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------



RCS file: /storage/cvs/jesus/mod_backhand/mod_backhand.c,v
retrieving revision 1.50
diff -u -3 -r1.50 mod_backhand.c
--- mod_backhand.c 13 Sep 2002 00:40:42 -0000 1.50
+++ mod_backhand.c 13 Nov 2002 05:35:16 -0000
@@ -913,6 +913,11 @@
ap_table_setn((r->main)?r->main->notes:r->notes,
"BackhandHandlerExecuted", "Done");
ap_internal_redirect(r->uri, r);
+ } else {
+ ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, NULL,
+ "mod_backhand: "
+ "attempted mod_backhand redirection twice. BUG.");
+ return DECLINED;
}
if(loglevel & MBLL_MBCS3) {
ap_log_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, NULL,
@@ -1335,7 +1340,7 @@
const char *rhost = ap_table_get(subr->notes,
"Backhand-Redirect-Host");
if(rhost) ap_table_set(r->notes, "Backhand-Redirect-Host",
rhost);
r->uri = ap_pstrcat(r->pool, "backhand:", r->uri, NULL);
- r->filename = ap_pstrdup(r->pool, r->uri);
+ r->filename = ap_pstrcat(r->pool, "backhand:", r->files, NULL);
r->handler = "backhand-redirection";
if(loglevel & MBLL_DCSN1) {
ap_log_error(APLOG_MARK, APLOG_NOTICE|APLOG_NOERRNO, NULL,
@@ -1379,6 +1384,8 @@
return DECLINED;
}
memmove(r->uri, &r->uri[9], strlen(&r->uri[9])+1);
+ if(strncmp(r->filename, "backhand:", 9)==0)
+ memmove(r->filename, &r->filename[9], strlen(&r->filename[9])+1);
return OK;
}


_______________________________________________
backhand-users mailing list
backhand-users@lists.backhand.org
http://lists.backhand.org/mailman/listinfo/backhand-users

On Wednesday, Nov 13, 2002, at 10:33 US/Eastern, Dennis Cartier wrote:
> The r->files in the line below is not defined in my version
>
> + r->filename = ap_pstrcat(r->pool, "backhand:", r->files, NULL);
>
> I have a CVS version from late last week. Did you perhaps mean
> r->filename?

I did... forgot to change that on my laptop.

> I tried using r->filename for the undefined structure memeber, but the
> cascade still starts immediately.


> I have some more info though (when compiled with r->filename!):
>
> 1. The error message you added in the patch never gets triggered.

Good. I thought I was loosing my mind.

> 2. The looping seems to initially be a tight loop, looping after
> "mod_backhand: Check 3 --" is displayed in the log. Eventually (when
> all the
> children started?) the loop seems to grow to include the check 6
> portion and
> logs a failure after that point. I have included the transition point
> of the
> logs below.

So, the logs indicate that it is redirecting to 10.1.1.21, what IP is
_this_ Apache instance listening on? Make sure the two apache
instances are writing to different log files and then tail -f the
"internal" log file and connect to "10.1.1.21 80" and make a request...
Make sure the logs show up in the right place. Also, of course, you
can't be running mod_backhand on the internal Apache instance.

> [Wed Nov 13 10:33:09 2002] [notice] All funcs executed -> 10.1.1.21
> [Wed Nov 13 10:33:09 2002] [notice] (Type =>) = *{
> backhand-redirection,
> backhand:/index.html, backhand/redirect }
> [Wed Nov 13 10:33:09 2002] [notice] (Translate <=) = {
> backhand:/backhand:/index.html, (null), (null) }
> [Wed Nov 13 10:33:09 2002] [notice] child pid 16618 exit signal
> Segmentation
> fault (11)

Those "Type =>" and "Translate <=" calls a freaky. I hope they are
from different children. It looks like some really strange hook
execution is happening. I run mod_jk with mod_backhand and it works
fine. For each place "backhand:" is prepended to the uri/filename,
there is another place where it is removed. You must be triggering the
first several times and the module ordering or configuration must be
shortcircuiting the removal.

Before I patch this up trying fix a problem I can't replicate, can you
post your entire httpd.conf (blank lines and comments removed)? Also
post the exact request line you pop into a "telnet myhost 80" that will
trigger this behaviour.

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------

> -----Original Message-----
> From: backhand-users-admin@lists.backhand.org
> [mailto:backhand-users-admin@lists.backhand.org]On Behalf Of Theo
> Schlossnagle
> Sent: Wednesday, November 13, 2002 11:21 AM
> To: backhand-users@lists.backhand.org
> Cc: Theo Schlossnagle
> Subject: Re: [m_b_users] Corrupt request causes Denial of Service
> cascade
>
> I did... forgot to change that on my laptop.

OK, good. That means my recent tests were valid.

>
> So, the logs indicate that it is redirecting to 10.1.1.21, what IP is
> _this_ Apache instance listening on? Make sure the two apache
> instances are writing to different log files and then tail -f the
> "internal" log file and connect to "10.1.1.21 80" and make a request...
> Make sure the logs show up in the right place. Also, of course, you
> can't be running mod_backhand on the internal Apache instance.
>

Hmmm. I just realized that I had all the other nodes were shutdown while I
was collecting the previous samples, so the only apache instance running was
10.1.1.21. This got me to thinking, how could it cascade on only 1 node? It
turns out if I have SelfRedirect On, then the cascade can occur on just 1
node. Not sure if that helps.

The cluster layout is like this, single layer tier, 1 Apache running per
node that listens on 1 internal address (10.1.1.X) and 2 external addresses
(1 static & firewalled and 1 VIP from wackamole). mod_backhand uses the
internal network to broadcast over and backhand through. It is identical to
your multi-point cluster example in the 2001 presentation.

>
> Those "Type =>" and "Translate <=" calls a freaky. I hope they are
> from different children. It looks like some really strange hook
> execution is happening. I run mod_jk with mod_backhand and it works
> fine. For each place "backhand:" is prepended to the uri/filename,
> there is another place where it is removed. You must be triggering the
> first several times and the module ordering or configuration must be
> shortcircuiting the removal.
>
> Before I patch this up trying fix a problem I can't replicate, can you
> post your entire httpd.conf (blank lines and comments removed)? Also
> post the exact request line you pop into a "telnet myhost 80" that will
> trigger this behaviour.
>

==== httpd.conf ====
ServerType standalone
ServerRoot "/var/backhand"
PidFile /var/backhand/logs/httpd.pid
ScoreBoardFile /var/backhand/logs/httpd.scoreboard
Timeout 300
KeepAlive off
MaxKeepAliveRequests 1000
KeepAliveTimeout 5
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 150
MaxRequestsPerChild 250
LoadModule status_module libexec/mod_status.so
LoadModule info_module libexec/mod_info.so
LoadModule jk_module libexec/mod_jk.so
LoadModule ssl_module libexec/libssl.so
LoadModule backhand_module libexec/mod_backhand.so
LoadModule log_spread_module libexec/mod_log_spread.so
LoadModule gzip_module libexec/mod_gzip.so
ClearModuleList
AddModule mod_env.c
AddModule mod_mime.c
AddModule mod_negotiation.c
AddModule mod_status.c
AddModule mod_info.c
AddModule mod_include.c
AddModule mod_dir.c
AddModule mod_cgi.c
AddModule mod_asis.c
AddModule mod_imap.c
AddModule mod_actions.c
AddModule mod_alias.c
AddModule mod_access.c
AddModule mod_auth.c
AddModule mod_so.c
AddModule mod_setenvif.c
AddModule mod_jk.c
AddModule mod_ssl.c
AddModule mod_backhand.c
AddModule mod_log_spread.c
AddModule mod_gzip.c
Listen 443
Listen 80
User nobody
Group nogroup
ServerAdmin root@noc
DocumentRoot "/var/backhand/htdocs"
SpreadDaemon 4803
ExtendedStatus On
NameVirtualHost *:80
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
mod_gzip: %{mod_gzip_result}n In:%{mod_gzip_input_size}n
Out:%{mod_gzip_output_size}n:%{mod_gzip_compression_ratio}n pct." combined
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
Include conf/mod_gzip.conf
Include conf/mod_jk.conf
Include conf/mod_backhand.conf
<Directory "/var/backhand/htdocs">
SetEnvIf BackhandProxied "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" proxied
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>
<IfModule mod_dir.c>
DirectoryIndex index.html
</IfModule>
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</Files>
<Location /server-status>
SetHandler server-status
Order Deny,Allow
Deny from all
Allow from 192.168.1 192.168.0 10.1.1
Backhand off
</Location>
<Location /server-info>
SetHandler server-info
Order Deny,Allow
Deny from all
Allow from 192.168.1 192.168.0 10.1.1
Backhand off
</Location>
UseCanonicalName On
<IfModule mod_mime.c>
TypesConfig /var/backhand/conf/mime.types
</IfModule>
DefaultType text/plain
<IfModule mod_mime_magic.c>
MIMEMagicFile /var/backhand/conf/magic
</IfModule>
HostnameLookups Off
ErrorLog /var/backhand/logs/error_log
LogLevel warn
ServerSignature On
<IfModule mod_alias.c>
Alias /icons/ "/var/backhand/icons/"
<Directory "/var/backhand/icons">
Options Indexes MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>
Alias /manual/ "/var/backhand/htdocs/manual/"
<Directory "/var/backhand/htdocs/manual">
Options Indexes FollowSymlinks MultiViews
AllowOverride None
Order allow,deny
Allow from all
</Directory>
ScriptAlias /cgi-bin/ "/var/backhand/cgi-bin/"
<Directory "/var/backhand/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
</IfModule>
<IfModule mod_autoindex.c>
IndexOptions FancyIndexing
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip
AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*
AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif core
AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^
DefaultIcon /icons/unknown.gif
ReadmeName README
HeaderName HEADER
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t
</IfModule>
<IfModule mod_mime.c>
AddEncoding x-compress Z
AddEncoding x-gzip gz tgz
AddLanguage da .dk
AddLanguage nl .nl
AddLanguage en .en
AddLanguage et .ee
AddLanguage fr .fr
AddLanguage de .de
AddLanguage el .el
AddLanguage he .he
AddCharset ISO-8859-8 .iso8859-8
AddLanguage it .it
AddLanguage ja .ja
AddCharset ISO-2022-JP .jis
AddLanguage kr .kr
AddCharset ISO-2022-KR .iso-kr
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddCharset ISO-8859-2 .iso-pl
AddLanguage pt .pt
AddLanguage pt-br .pt-br
AddLanguage ltz .lu
AddLanguage ca .ca
AddLanguage es .es
AddLanguage sv .sv
AddLanguage cz .cz
AddLanguage ru .ru
AddLanguage zh-tw .tw
AddLanguage tw .tw
AddCharset Big5 .Big5 .big5
AddCharset WINDOWS-1251 .cp-1251
AddCharset CP866 .cp866
AddCharset ISO-8859-5 .iso-ru
AddCharset KOI8-R .koi8-r
AddCharset UCS-2 .ucs2
AddCharset UCS-4 .ucs4
AddCharset UTF-8 .utf8
<IfModule mod_negotiation.c>
LanguagePriority en da nl et fr de el it ja kr no pl pt pt-br ru ltz
ca es sv tw
</IfModule>
AddType application/x-tar .tgz
AddType image/x-icon .ico
</IfModule>
<IfModule mod_setenvif.c>
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0
</IfModule>
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
<IfModule mod_ssl.c>
SSLPassPhraseDialog builtin
SSLSessionCache dbm:/var/backhand/logs/ssl_scache
SSLSessionCacheTimeout 300
SSLMutex file:/var/backhand/logs/ssl_mutex
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
SSLLog /var/backhand/logs/ssl_engine_log
SSLLogLevel info
</IfModule>
<VirtualHost _default_:443>
DocumentRoot "/var/backhand/htdocs"
ServerName tomcat1
ServerAdmin root@tomcat1
ErrorLog /var/backhand/logs/error_log
TransferLog /var/backhand/logs/access_log
<IfModule mod_ssl.c>
SSLCipherSuite
ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /var/backhand/conf/ssl.crt/server.crt
SSLCertificateKeyFile /var/backhand/conf/ssl.key/server.key
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/backhand/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
SSLEngine on
</IfModule>
SetEnvIf BackhandProxied "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" proxied
CustomLog $apache combined env=proxied
</VirtualHost>
<VirtualHost _default_:80>
DocumentRoot /var/backhand/htdocs
SetEnvIf BackhandProxied "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" proxied
CustomLog $apache combined env=proxied
</VirtualHost>
<VirtualHost *:80>
DocumentRoot /var/backhand/htdocs
ServerName rbc.blkbk.com
JkMount /rbc/jb/* ajp13
SetEnvIf BackhandProxied "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" proxied
CustomLog $apache combined env=proxied
</VirtualHost>
<VirtualHost *:80>
DocumentRoot /var/backhand/htdocs
ServerName consumer.blkbk.com
JkMount /consumer/jb/* ajp13
SetEnvIf BackhandProxied "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" proxied
CustomLog $apache combined env=proxied
</VirtualHost>


==== mod_backhand.conf ===
<IfModule mod_backhand.c>
UnixSocketDir /var/backhand/backhand
MulticastStats internal:80 10.1.1.255:4445,1
AcceptStats 10.1.1.0/24
BackhandSelfRedirect ON
BackhandConnectionPools OFF
BackhandModeratorPIDFile /var/backhand/backhand/moderator.pid
<LocationMatch "/.+">
BackhandLogLevel +netall
BackhandLogLevel +mbcsall
BackhandLogLevel +dcsnall
Backhand byAge 5
Backhand byRandom
Backhand byLogWindow
Backhand byBusyChildren
</LocationMatch>
<LocationMatch ".+/jb/">
BackhandLogLevel +netall
BackhandLogLevel +mbcsall
BackhandLogLevel +dcsnall
Backhand byAge 5
BackhandFromSO libexec/byService.so byService 8009
Backhand bySession RMID=
Backhand byRandom
Backhand byLogWindow
Backhand byLoad
</LocationMatch>
<Location "/backhand/">
Order Deny,Allow
Deny from all
Allow from 192.168.1 192.168.0 10.1.1
Backhand off
SetHandler backhand-handler
</Location>
</IfModule>


==== mod_jk.conf ====
<IfModule mod_jk.c>
JkWorkersFile conf/workers.properties
JkLogFile /tmp/mod_jk.log
JkLogLevel debug
</IfModule>


==== mod_gzip.conf ====
<IfModule mod_gzip.c>
mod_gzip_on Yes
mod_gzip_can_negotiate Yes
mod_gzip_static_suffix .gz
AddEncoding gzip .gz
mod_gzip_update_static No
mod_gzip_command_version '/mod_gzip_status'
mod_gzip_temp_dir /tmp
mod_gzip_keep_workfiles No
mod_gzip_minimum_file_size 500
mod_gzip_maximum_file_size 750000
mod_gzip_maximum_inmem_size 750000
mod_gzip_min_http 1000
mod_gzip_handle_methods GET POST
mod_gzip_item_exclude reqheader "User-agent: Mozilla/4.0[678]"
mod_gzip_item_include file \.html$
mod_gzip_item_exclude file \.js$
mod_gzip_item_exclude file \.css$
mod_gzip_item_include file \.pl$
mod_gzip_item_include handler ^cgi-script$
mod_gzip_item_include uri /test/servlet/*
mod_gzip_item_include uri /*/jb/
mod_gzip_item_include file \.jsp$
mod_gzip_item_include uri /jmx-console
mod_gzip_item_include mime ^text/html$
mod_gzip_item_include mime ^text/plain$
mod_gzip_item_include mime ^httpd/unix-directory$
mod_gzip_item_exclude mime ^image/
mod_gzip_dechunk Yes
LogFormat "%h %l %u %t \"%V %r\" %<s %b mod_gzip:
%{mod_gzip_result}n In:%{mod_gzip_input_size}n -<
Out:%{mod_gzip_output_size}n = %{mod_gzip_compression_ratio}n pct."
common_with_mod_gzip_info2
CustomLog logs/mod_gzip.log common_with_mod_gzip_info2
mod_gzip_add_header_count Yes
mod_gzip_send_vary Yes
</IfModule>

I just realized I did not give you a specific request to test with. Actually
it occurs with any corrupt request Eg. (http://sdsdfsdfsff) so any old input
will do, as long as it is not a valid request Eg. (GET http://asdasdasdas
HTTP/1.0) would not cause the cascade.

Dennis

Hi Theo,

I found the module that was causing the SegFault with the patch. It was
mod_dir!

After I removed it, typing garbage into a connection no longer causes a
segfault, you get back the expected response about the method not being
implemented.

This works as long as I backhand based on '/.+' rather than '.+' . If I
backhand on '.+', than it still causes the cascade of backhand's to max
children.

There is still a bug, but at least I can avoid it now without having
children dying for every request for '/'.

Thanks for your help.

Dennis