Came in this morning to find hyperreal's 0.7.2h dead:
1) only the parent was still around
2) it was sucking down CPU as fast as possible
3) A kill -HUP didn't restart it
4) ktrace didn't report any system calls being made
5) no CGI scripts running that I could tell
6) the server had stopped serving up requests about 9:30am our time
7) first errant 403 request came at 8:53am, shortly after a 500 server
error for the home page
8) looks like the "403" virus spread to children quickly - all responses
returning 403 by 9:12am
9) 302 responses for directory requests not ending with / were somehow
able to be accomplished after the 403 virus had spread
10) none of the 403 responses had the IP number translated to a hostname
11) error log has lots of errors around the time 403's started with
"httpd: could not create IPC pipe". After the first one appeared, then
I started getting the "403" errors recorded to the error log.
For those with hyperreal accounts, the damage can be found in
/usr/local/www.tools/apache/logs/error_log
/var/log/httpd (today)
/export/logarchives/1995/06/19.httpd.gz (tomorrow)
If anyone can suggest anything else that needs to be looked for when this
happens, let me know. Hopefully this report has enough details to let
those who know the IPC code find the problem.
Brian
--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/
1) only the parent was still around
2) it was sucking down CPU as fast as possible
3) A kill -HUP didn't restart it
4) ktrace didn't report any system calls being made
5) no CGI scripts running that I could tell
6) the server had stopped serving up requests about 9:30am our time
7) first errant 403 request came at 8:53am, shortly after a 500 server
error for the home page
8) looks like the "403" virus spread to children quickly - all responses
returning 403 by 9:12am
9) 302 responses for directory requests not ending with / were somehow
able to be accomplished after the 403 virus had spread
10) none of the 403 responses had the IP number translated to a hostname
11) error log has lots of errors around the time 403's started with
"httpd: could not create IPC pipe". After the first one appeared, then
I started getting the "403" errors recorded to the error log.
For those with hyperreal accounts, the damage can be found in
/usr/local/www.tools/apache/logs/error_log
/var/log/httpd (today)
/export/logarchives/1995/06/19.httpd.gz (tomorrow)
If anyone can suggest anything else that needs to be looked for when this
happens, let me know. Hopefully this report has enough details to let
those who know the IPC code find the problem.
Brian
--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com brian@hyperreal.com http://www.[hyperreal,organic].com/