Mailing List Archive

On Mon, 4 Oct 2004 11:08:53 -0700, Grant <emailgrant@gmail.com> wrote:
> My server's CPU is 0.0% idle and it looks like a bunch of apache2
> processes taking up about 5% each is the problem. Does anyone know
> what's going on here?
>
> - Grant
>
> --
> gentoo-user@gentoo.org mailing list
>
>

Script kiddies banging away probably. Look at the logs in /var/log.
There should be some info there

--
gentoo-user@gentoo.org mailing list

> > My server's CPU is 0.0% idle and it looks like a bunch of apache2
> > processes taking up about 5% each is the problem. Does anyone know
> > what's going on here?
> >
> > - Grant

Restarting apache2 seems to have stopped it. I got a ton of these in
my error_log:

-----
[warn] child process 32110 still did not exit, sending a SIGTERM
-----

and right before this happened I was getting crap like this:

-----
client denied by server configuration: /usr/share/doc/
client denied by server configuration: /usr/share/doc/NonExistent.html
File does not exist:
/var/www/localhost/htdocs/shop/images/thumb/biztalkhttpreceive.dll
File does not exist:
/var/www/localhost/htdocs/shop/images/navigation/biztalkhttpreceive.dll
-----

What can I do to keep this from happening?

- Grant

--
gentoo-user@gentoo.org mailing list

Grant wrote:
> My server's CPU is 0.0% idle and it looks like a bunch of apache2
> processes taking up about 5% each is the problem. Does anyone know
> what's going on here?

if it was DOS, you wouldn't be able to do a process list and your web
server wouldn't deliver it's pages.

do a

# netstat -na | grep ESTAB

to see who's connected.

The do a

# netstat -na | grep WAIT

to see what connections are left hanging.

If you have a lot of hung connections they may (or may not) be causing a
load on your apache even though it should be idle. Try doing a

# apache2ctl graceful

or

# apache2ctl restart

this will end your apache processes and reload them.

Also, check the status of the apache processes. If you do a

# ps auxw | grep apache2

You should see a bunch of "S" and maybe a few "R" next to a date/time
field. That is the status of the process.

If you see a lot of "D" statuses, then you're either running out of
memory and swapping, or your filesystem is corrupted or your kernel oops
on a filesystem module (like reiser4). You'll have to reboot if it's the
latter two - otherwise get more RAM.

--
gentoo-user@gentoo.org mailing list

On Mon, 4 Oct 2004 11:17:16 -0700, Grant <emailgrant@gmail.com> wrote:
> > > My server's CPU is 0.0% idle and it looks like a bunch of apache2
> > > processes taking up about 5% each is the problem. Does anyone know
> > > what's going on here?
> > >
> > > - Grant
>
> Restarting apache2 seems to have stopped it. I got a ton of these in
> my error_log:
>
> -----
> [warn] child process 32110 still did not exit, sending a SIGTERM
> -----
>
> and right before this happened I was getting crap like this:
>
> -----
> client denied by server configuration: /usr/share/doc/
> client denied by server configuration: /usr/share/doc/NonExistent.html
> File does not exist:
> /var/www/localhost/htdocs/shop/images/thumb/biztalkhttpreceive.dll
> File does not exist:
> /var/www/localhost/htdocs/shop/images/navigation/biztalkhttpreceive.dll
> -----
>
> What can I do to keep this from happening?
>
>
>
> - Grant

Dunno - I'm no guru. When I ran Apache at home for 12 months I saw
this stuff all the time. Most of the time it was coming from Comcast
clients (I'm also on Comcast at home) and I think it was just Windows
boxes infected with viruses, or that was my thought at the time.
Usually there were lines looking for certain Windows files that I
figured were know exploits.

If you see it happening always from the same IP address then you can
ask your ISP to block, but other then that I do not know

--
gentoo-user@gentoo.org mailing list

Grant wrote:

> /var/www/localhost/htdocs/shop/images/navigation/biztalkhttpreceive.dll
> What can I do to keep this from happening?

looks like you installed something without properly setting it up.. it
looks like it's trying to load a windows .dll.

If you didn't install any type of shopping cart, then someone else may
have tried. If that someone else is unknown to you, then you have bigger
problems.

--
gentoo-user@gentoo.org mailing list

Billy wrote:

> looks like you installed something without properly setting it up.. it
> looks like it's trying to load a windows .dll.

bah.. scratch that.. I thought that was a loading message. After
re-reading the email that's just a 404 error being returned to a client.

If you have that many bad clients causing enough usage on your web box
to account for visible processor usage, then it looks like you're in a
bad internet neibhorhood. Everyone around you might just be infected
with something and they're looking for more hosts.

http://www.securiteam.com/windowsntfocus/5KP0L15B5G.html


--
gentoo-user@gentoo.org mailing list

On Mon, Oct 04, 2004 at 11:20:05AM -0700, Mark Knecht wrote the following:
> On Mon, 4 Oct 2004 11:17:16 -0700, Grant <emailgrant@gmail.com> wrote:
> > > > My server's CPU is 0.0% idle and it looks like a bunch of apache2
> > > > processes taking up about 5% each is the problem. Does anyone know
> > > > what's going on here?
> > > >
> > > > - Grant
> >
> > Restarting apache2 seems to have stopped it. I got a ton of these in
> > my error_log:
> >
> > -----
> > [warn] child process 32110 still did not exit, sending a SIGTERM
> > -----
> >
> > and right before this happened I was getting crap like this:
> >
> > -----
> > client denied by server configuration: /usr/share/doc/
> > client denied by server configuration: /usr/share/doc/NonExistent.html
> > File does not exist:
> > /var/www/localhost/htdocs/shop/images/thumb/biztalkhttpreceive.dll
> > File does not exist:
> > /var/www/localhost/htdocs/shop/images/navigation/biztalkhttpreceive.dll
> > -----
> >
> > What can I do to keep this from happening?
> >
> >
> >
> > - Grant
>
> Dunno - I'm no guru. When I ran Apache at home for 12 months I saw
> this stuff all the time. Most of the time it was coming from Comcast
> clients (I'm also on Comcast at home) and I think it was just Windows
> boxes infected with viruses, or that was my thought at the time.
> Usually there were lines looking for certain Windows files that I
> figured were know exploits.
>
> If you see it happening always from the same IP address then you can
> ask your ISP to block, but other then that I do not know
>
> --
> gentoo-user@gentoo.org mailing list
>

I would suggest installing sugarplum and use mod_rewrite to send all those
requests to a page full of crap. It won't stop the page accesses but it
should deter (after a while) any script kiddie from trying to exploit the
server. At least, I think it should. Whether or not it does...*shrugs*

And you get the added benefit where any spambots come by, they can be
instantly redirected to those pages with all the bogus emails which will
really mess up the spammers database of legit addresses (also, this is a
good way to give your worst enemy an unending deluge of spam *winkwink*)

--
AIM: pres CTHULHU | ICQ: 18115568 | Yahoo: pagan_prince
Jabber: DarkKnightRadick@(jabber.org|amessage.at) | Libertarian @ Large
PGP: 0x642F7BDA | < http://groups.yahoo.com/group/tennesseans-for-badnarik/ >
< http://mc-luug.homelinux.org/mailman/listinfo/mc-luug >

quoth the Mark Knecht:
> If you see it happening always from the same IP address then you can
> ask your ISP to block, but other then that I do not know

No need to bug your ISP, just drop packets from the offending IPs in your
firewall. Everyone that runs a public webserver (including myself) collects
logs full of this crap.

What I am not clear on though...is whether these are actually active
script-kiddie attacks, or just zombie PCs with no human interaction hammering
away at whatever box they can find.

I get lot's of log entries like:

"GET /scripts/..%255c%255c../winnt/system32/cmd.exe?/c+dir"

clearly trying to exploit a windows box. Seems to me any script kiddie that
isn't a *total* idiot will be able to figure out quite easily the OS of his
target, which leads me to believe that it is a zombie PC launching these
"attacks".

The upshot of this is that you can spend 12 hours a day manually blocking IP
addresses, and all you really accomplish is blocking an IP, or block of IPs
used by some fool that doesn't know his wintendo box is full of viruses.

My advice: just ignore, and be thankful you run Linux. If you want to sort all
this cruft out of your logs just do something like:

# grep -v ".exe" access_log > good_log

Not perfect, as you may filter some legit requests doing this....

-d
--
Part of the problem since 1976
http://badcomputer.no-ip.com
Get my public key from
http://keyserver.linux.it/pks/lookup?op=index&search=bulliver
"...the number of UNIX installations has grown to 10, with more expected..."
- Dennis Ritchie and Ken Thompson, June 1972

On Mon, 4 Oct 2004 15:38:24 -0700, Darren Kirby
<bulliver@badcomputer.no-ip.com> wrote:
> quoth the Mark Knecht:
> > If you see it happening always from the same IP address then you can
> > ask your ISP to block, but other then that I do not know
>
> No need to bug your ISP, just drop packets from the offending IPs in your
> firewall. Everyone that runs a public webserver (including myself) collects
> logs full of this crap.

Sure. Makes sense. I'm not personally that sophistacated and only ran
Apache so that a few other people could listen to my music and
comment. I didn't have a 5% problem like Grant so it never became such
and issue.

>
> What I am not clear on though...is whether these are actually active
> script-kiddie attacks, or just zombie PCs with no human interaction hammering
> away at whatever box they can find.
>
> I get lot's of log entries like:
>
> "GET /scripts/..%255c%255c../winnt/system32/cmd.exe?/c+dir"
>
> clearly trying to exploit a windows box. Seems to me any script kiddie that
> isn't a *total* idiot will be able to figure out quite easily the OS of his
> target, which leads me to believe that it is a zombie PC launching these
> "attacks".

I think they are 'sombies', in the sense that they are infected PCs. I
noticed that on a few PCs these requests came only for 3 hours in the
evening, or some other somewhat managed times, like the people turned
their PCs on and didn't know they were infected.

>
> The upshot of this is that you can spend 12 hours a day manually blocking IP
> addresses, and all you really accomplish is blocking an IP, or block of IPs
> used by some fool that doesn't know his wintendo box is full of viruses.
>
> My advice: just ignore, and be thankful you run Linux.

Very true!

> If you want to sort all
> this cruft out of your logs just do something like:
>
> # grep -v ".exe" access_log > good_log

Good stuff. Thanks!


Thanks,
Mark

--
gentoo-user@gentoo.org mailing list

I forgot to address Grant's original concern. My previous message was intended
in a broad scope, where these attacks aren't actually causing resource
problems.

If you have a good idea of the amount of legitemate traffic your server
receives you can edit apache2.conf and set a hard limit on the number of
server child processes:

<IfModule worker.c>
StartServers 2
MaxClients 150 <-- edit this to a reasonable amount
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

Not an ideal solution either, but it will keep your server from draining the
resources of your computer. Unfortunately this may affect legitemate requests
if an attack is underway at the same time.
-d
--
Part of the problem since 1976
http://badcomputer.no-ip.com
Get my public key from
http://keyserver.linux.it/pks/lookup?op=index&search=bulliver
"...the number of UNIX installations has grown to 10, with more expected..."
- Dennis Ritchie and Ken Thompson, June 1972