Mailing List Archive

Denial of Service Cascade continues ...
Hello,

I guess I spoke too soon when I said I had a workaround for the cascade
issues. I have since had 3 Apache's maxout their children. The cascade does
not spread like before, from node to node. It seems to be restricted to the
local node (I think).

This all occured on a cluster that is for the most part idle, with no
traffic, just the occaisonal passer by come to kick the tires and look for
any gaping security holes.

The patch that I had applied (from Theo), does not seem to help this problem
so I have removed it. Just the change in the backhand criteria that I
mentioned, deals with the corrupt request portion.

I have not idea why the cascade is still occuring, must be seperate yet
related bugs.

Theo, when you get back from ApacheCon, can I have a copy of your working
config? I am willing to forego mod_gzip and possibly use a filter in the
Java portion to compress instead if it helps to stablize the operation.

Dennis Cartier

---
The man with tremendous wealth is a paradox: he has to be smart enough to
get it, yet stupid enough to want it. - Chesterton
Denial of Service Cascade continues ... [ In reply to ]
mod_gzip shouldn't be a problem. Do you have BackhandSelfRedirect On?
If so why? I could see there being some bad cascading problems if you
redirect to youself "proper".

On Saturday, Nov 16, 2002, at 10:13 US/Eastern, Dennis Cartier wrote:
> I guess I spoke too soon when I said I had a workaround for the cascade
> issues. I have since had 3 Apache's maxout their children. The cascade
> does
> not spread like before, from node to node. It seems to be restricted
> to the
> local node (I think).

--
Theo Schlossnagle
Principal Consultant
OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

-----------------------------------------------------------
-- Presenting at ApacheCon --
-- November 18th, 2002 --
-- Las Vegas, Nevada --
-- Backhand: understanding and building HA/LB clusters --
-- http://apachecon.com/2002/US/ --
-- --
-- Learn all there is to know about high availability --
-- internet systems and load balancing techniques --
-- focusing on applications driven by the Apache web --
-- server! --
-----------------------------------------------------------
Denial of Service Cascade continues ... [ In reply to ]
Hi Theo,

Yes, I had BackhandSelfRedirect On. It was the only way I could get
mod_gzip, mod_ssl and the logging to all work together. I was under the
impression that this would be safe because the connection once backhanded,
would not be backhanded again, but would be serviced. Is this not the way it
is suppose to work?

In any case, I adjusted my config to turn BackhandSelfRedirect Off, and
added a remove self to accomplish the same thing. The cascade still occurs
though. I left the servers online, but idle, and the usual scanning scripts
looking for residual Nimda infections, caused a cascade. I tried entering
the same URL after restarting the servers (or as close to the URL as I could
get) but just got a 404. Hmmm, still a possible issue.

I noticed you had made some changes in the CVS version, so I have updated to
that version. I will monitor the servers for a few days and let you know if
they cascade.

Dennis

> -----Original Message-----
> From: backhand-users-admin@lists.backhand.org
> [mailto:backhand-users-admin@lists.backhand.org]On Behalf Of Theo
> Schlossnagle
> Sent: Saturday, November 16, 2002 1:16 PM
> To: backhand-users@lists.backhand.org
> Cc: Theo Schlossnagle
> Subject: Re: [m_b_users] Denial of Service Cascade continues ...
>
>
> mod_gzip shouldn't be a problem. Do you have BackhandSelfRedirect On?
> If so why? I could see there being some bad cascading problems if you
> redirect to youself "proper".
>
> On Saturday, Nov 16, 2002, at 10:13 US/Eastern, Dennis Cartier wrote:
> > I guess I spoke too soon when I said I had a workaround for the cascade
> > issues. I have since had 3 Apache's maxout their children. The cascade
> > does
> > not spread like before, from node to node. It seems to be restricted
> > to the
> > local node (I think).
>
> --
> Theo Schlossnagle
> Principal Consultant
> OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
> Phone: +1 410 872 4910 x201 Fax: +1 410 872 4911
> 1024D/82844984/95FD 30F1 489E 4613 F22E 491A 7E88 364C 8284 4984
> 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>
> -----------------------------------------------------------
> -- Presenting at ApacheCon --
> -- November 18th, 2002 --
> -- Las Vegas, Nevada --
> -- Backhand: understanding and building HA/LB clusters --
> -- http://apachecon.com/2002/US/ --
> -- --
> -- Learn all there is to know about high availability --
> -- internet systems and load balancing techniques --
> -- focusing on applications driven by the Apache web --
> -- server! --
> -----------------------------------------------------------
>
>
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users
>
>
Denial of Service Cascade continues ... [ In reply to ]
> I noticed you had made some changes in the CVS version, so I have
> updated to
> that version. I will monitor the servers for a few days and let
> you know if
> they cascade.

Hi Theo,

The server pool cascaded this afternoon, again due to a roving Nimda
scanner. I captured snapshots of the backhand screen and the server-status
screens incase some clues can be gained from them.

Backhand screen: http://www4.blkbk.com/tomcat1_backhand.html

Tomcat1 status: http://www4.blkbk.com/tomcat1_status.html
Tomcat2 status: could not complete, no children available
Tomcat3 status: http://www4.blkbk.com/tomcat3_status.html
Tomcat4 status: http://www4.blkbk.com/tomcat4_status.html

I had a look at the logs, there have been many Nimda scanners that have hit
the servers between the restart on Friday and today. The cascade did not
start until today however. I am wondering if it is time related, or perhaps
requires a special type of Nimda URL to cause the problem. Of course the
actual request that started it did not get logged, Apache becomes very
unusable once it cascades.

I can post info from my logs as well if it is helpful.

Is anyone else having Nimda trouble?

I suppose I can filter it in Apache to stop the Nimda script, but I am
worried that other situations may arise that allow a cascade to occur and
would have to be filter once noticed.

I am willing to keep my cluster in pre-production mode to help isolate what
the problem is if it will help.

Dennis