Mailing List Archive: [mod_backhand-users] child processes segfaulting, wrong client ip-adresses

[mod_backhand-users] child processes segfaulting, wrong client ip-adresses

Feb 3, 2001, 8:09 AM

Post #1 of 4 (799 views)

Hello List!

after some configuration work, i finally managed to get mod_backhand up
and running.

Unfortunately, two problems still remain, and I have no idea how to
investigate any further:

The first one are occasionaly dying processes at the machine working as
proxy. Currently, this
machine handles approx. a few requests a second or less, so load should
not be a problem. But in
the apache error log looks like this:

<snip>
[Sat Feb 3 10:13:01 2001] [notice] child pid 15606 exit signal
Segmentation fault (11)
[Sat Feb 3 11:10:33 2001] [notice] child pid 15609 exit signal
Segmentation fault (11)
[Sat Feb 3 11:16:59 2001] [notice] child pid 15598 exit signal
Segmentation fault (11)
[Sat Feb 3 11:17:01 2001] [notice] child pid 16828 exit signal
Segmentation fault (11)
[Sat Feb 3 11:30:37 2001] [notice] child pid 16947 exit signal
Segmentation fault (11)
[Sat Feb 3 11:31:01 2001] [notice] child pid 16130 exit signal
Segmentation fault (11)
[Sat Feb 3 11:47:19 2001] [notice] child pid 16685 exit signal
Segmentation fault (11)
[Sat Feb 3 11:58:51 2001] [notice] child pid 16885 exit signal
Segmentation fault (11)
[Sat Feb 3 12:04:46 2001] [notice] child pid 17381 exit signal
Segmentation fault (11)
[Sat Feb 3 12:13:21 2001] [notice] child pid 16273 exit signal
Segmentation fault (11)
[Sat Feb 3 12:16:04 2001] [notice] child pid 16971 exit signal
Segmentation fault (11)
[Sat Feb 3 12:41:11 2001] [notice] child pid 17856 exit signal
Segmentation fault (11)
[Sat Feb 3 12:46:22 2001] [notice] child pid 17897 exit signal
Segmentation fault (11)
[Sat Feb 3 13:16:06 2001] [notice] child pid 15600 exit signal
Segmentation fault (11)
[Sat Feb 3 13:16:10 2001] [notice] child pid 18133 exit signal
Segmentation fault (11)
[Sat Feb 3 13:32:37 2001] [notice] child pid 18913 exit signal
Segmentation fault (11)
[Sat Feb 3 13:32:43 2001] [notice] child pid 16331 exit signal
Segmentation fault (11)
[Sat Feb 3 13:46:44 2001] [notice] child pid 17036 exit signal
Segmentation fault (11)
[Sat Feb 3 13:59:15 2001] [notice] child pid 19322 exit signal
Segmentation fault (11)
[Sat Feb 3 13:59:15 2001] [notice] child pid 19321 exit signal
Segmentation fault (11)
[Sat Feb 3 13:59:15 2001] [notice] child pid 19320 exit signal
Segmentation fault (11)
[Sat Feb 3 13:59:15 2001] [notice] child pid 18893 exit signal
Segmentation fault (11)
<snip>

This segfaulting did not occur before i installed mod_backhand. It only
happens at the server
answering the requests from remote, those working as proxies on other
machines are perfectly
fine.

I'm not an expert in debugging running applications, so any hint how to
receive more information
about the dying processes would be appreciated.

Second: It seems that mod_backhand gets sometimes confused about the
clients' IP adresses.

If I reload repeatedly the /backhand/ status handler, it sometimes
(approx 10%) reports
the wrong REMOTE_ADDR. Same happens to php-scripts echoing or processing
the REMOTE_ADDR.
These wrong addresses usually belong to clients who are at the same
time making requests to the system. As we do things like content
customisation on an clients IP address
base, this is no good (Even the logging at the proxy-servers shows the
wrong adresses, while the external
server logs the right one)

Here's the setup:
(fake official IPs and domain names)

machine(n)
eth0: 1.2.3.21(n) 1.2.3.0/24
eth1: 192.168.14.4(n) 192.168.14.0/24
Multicast not explicitly bound to an interface via route, but tcpdump
shows the traffic at eth0, no traffic at eth1.

Linux version 2.2.16-SMP (root@SMP_X86.suse.de) (gcc version 2.95.2
19991024 (release))
2 CPUs, 2GB RAM, HW-Raid UW-SCSI drives, 100BaseT 3Com network adaptors.

One more machine (similar configuration) working as log server via
spread

Apache 1.3.14 with PHP 4.04RC6, mod_backhand 1.1.1p2, mod_log_spread

PHP is commonly used at most pages, Session management is active and
vital, PHPSESSID is processed as described in README.bySession.
php makes connections to Postgres (local and remote) and Oracle (remote)
Databases.

Excerpts from configuration:

(...)
LoadModule vhost_alias_module libexec/mod_vhost_alias.so
LoadModule env_module libexec/mod_env.so
LoadModule define_module libexec/mod_define.so
LoadModule config_log_module libexec/mod_log_config.so
LoadModule mime_module libexec/mod_mime.so
LoadModule negotiation_module libexec/mod_negotiation.so
LoadModule status_module libexec/mod_status.so
LoadModule info_module libexec/mod_info.so
LoadModule includes_module libexec/mod_include.so
LoadModule autoindex_module libexec/mod_autoindex.so
LoadModule dir_module libexec/mod_dir.so
LoadModule cgi_module libexec/mod_cgi.so
LoadModule asis_module libexec/mod_asis.so
LoadModule imap_module libexec/mod_imap.so
LoadModule action_module libexec/mod_actions.so
LoadModule alias_module libexec/mod_alias.so
LoadModule access_module libexec/mod_access.so
LoadModule auth_module libexec/mod_auth.so
LoadModule anon_auth_module libexec/mod_auth_anon.so
LoadModule dbm_auth_module libexec/mod_auth_dbm.so
LoadModule db_auth_module libexec/mod_auth_db.so
LoadModule digest_module libexec/mod_digest.so
LoadModule expires_module libexec/mod_expires.so
LoadModule headers_module libexec/mod_headers.so
LoadModule unique_id_module libexec/mod_unique_id.so
LoadModule setenvif_module libexec/mod_setenvif.so
LoadModule php4_module libexec/libphp4.so
LoadModule backhand_module libexec/mod_backhand.so
LoadModule log_spread_module libexec/mod_log_spread.so

(..)

<IfModule mod_backhand.c>
UnixSocketDir /var/state/backhand
MulticastStats 192.168.14.40 225.220.221.20:4445,1
AcceptStats 192.168.14.0/24
AcceptStats 1.2.3.0/24 <-- Without this line the 192.168.14.4x
servers do not see each other - strange
<Location "/backhand/">
SetHandler backhand-handler
</Location>
#BackhandLogLevel +dcsnall
#BackhandLogLevel +mbcsall
BackhandSelfRedirect On
</IfModule>

(..)

Listen 1.2.3.210:80
NameVirtualHost 1.2.3.210:80

Listen 192.168.14.40:80
NameVirtualHost 192.168.14.40:80

<VirtualHost 1.2.3.210:80>
ServerName www.domain.de
DocumentRoot /home/customer/domain/htdocs
ErrorLog /home/customer/logs/domain.de-error.log
CustomLog /home/customer/logs/domain.de-access.log combined
ErrorDocument 404 /bin/error.html

<IfModule mod_log_spread.c>
CustomLog $#vhost combined
</IfModule>

AliasMatch ^/([^/\.]+)[/]*$ /home/customer/domain/htdocs/$1.html
(... some more alias matches and <File> directives ...)
Redirect /index_d.html http://www.domain.de/
Redirect /index_e.html http://www.domain.de/

<Directory /home/customer/domain/htdocs>

(..)

<IfModule mod_php4.c>
php_admin_flag safe_mode off
php_value include_path .:/home/dmag/php
php_value auto_prepend_file /home/kunden/dmag/php/prepend.php3
php_value gpc_order PGC
</IfModule>

<IfModule mod_backhand.c>
Backhand byAge
Backhand byLoad
Backhand bySession
</IfModule>

</Directory>

</VirtualHost>

(Following identical setup for virtual server listening on
192.168.14.40)

(...)

Same configuration applies to the other machines, with external and
internal adresses incremented +1
(currently 1 more machine, as we are in evaluation phase, we plan to set
up 7 servers with DNS round robin to
the external virtual servers)

Thank you in advance!

Tilman

[mod_backhand-users] child processes segfaulting, wrong client ip-adresses [ In reply to ]

jesus at omniti

Feb 3, 2001, 6:38 PM

Post #2 of 4 (773 views)

Permalink

Tilman Kastner wrote:
> after some configuration work, i finally managed to get mod_backhand up
> and running.
>
> Unfortunately, two problems still remain, and I have no idea how to
> investigate any further:
>
> The first one are occasionaly dying processes at the machine working as
> proxy. Currently, this
> machine handles approx. a few requests a second or less, so load should
> not be a problem. But in
> the apache error log looks like this:
>
> <snip>
> [Sat Feb 3 10:13:01 2001] [notice] child pid 15606 exit signal
> Segmentation fault (11)
> ... many many more ....
> [Sat Feb 3 13:59:15 2001] [notice] child pid 18893 exit signal
> Segmentation fault (11)
> <snip>

Ouch. Well, I would wager this is mod_backhand. It looks like to proxying
children are dying. I thought I had cleaned up all of these, but I guess I
didn't or introduced a new one. Turning on debugging will help better define
what is breaking -- see below.

Also, what are your Backhand directives?
Backhand bySession is one of them I assume, what are the others?

> This segfaulting did not occur before i installed mod_backhand. It only
> happens at the server
> answering the requests from remote, those working as proxies on other
> machines are perfectly
> fine.

I don't understand your setup then... A request only ever gets redirected
once. So, how do you know the other proxies wouldn't exhibit this problem is
they were queried directly. If they are only queried via a "backhanded"
request, then they will no proxy (this eliminates requests from ping-ponging
indefinitely.) mod_backhand will only ever act like a proxy is the request
originated from a "remote" machine.

> I'm not an expert in debugging running applications, so any hint how to
> receive more information
> about the dying processes would be appreciated.

Apache modules are actually pretty difficult to debug. If you can repeat this
problem with *very little* traffic, then it is easier. If you can turn the
instance up and make only a few requests and repeat the problem, then it
shouldn't be too hard to troubleshoot. Otherwise, the error log will be
pretty big :-)

First, since we think it is mod_backhand that is at fault, turn on all of the
debugging:
BackhandLogLevel +mbcsall
BackhandLogLevel +netall
BackhandLogLevel +dcsnall

Delete your error log (so it starts fresh) and then run Apache. Let requests
come in until you see a segfault and then turn off apache. Save the error
file and send it to me via email. If you look in there, the last mod_backhand
debugging statement before the segfault should help narrow the window of code
that could have problems.

> Second: It seems that mod_backhand gets sometimes confused about the
> clients' IP adresses.
>
> If I reload repeatedly the /backhand/ status handler, it sometimes
> (approx 10%) reports
> the wrong REMOTE_ADDR. Same happens to php-scripts echoing or processing
> the REMOTE_ADDR.

You you notice any similarities in the reuqests that trigger these. For
instance are they actually backhanded? If you backhand your whole site, it
will end up backhanding your /backhand/ page, an annoying side effect. (Add
"Backhand off" inside your <Location /backhand/> clause to fix this. Do you
notice that the REMOTE_ADDR is only wrong when the response comes from a
machine you didn't send the request to (it was backhanded) or vice-versa?

> These wrong addresses usually belong to clients who are at the same
> time making requests to the system. As we do things like content
> customisation on an clients IP address
> base, this is no good (Even the logging at the proxy-servers shows the
> wrong adresses, while the external
> server logs the right one)

I am confused about your set up. What proxy servers? Are you running
mod_backhand on all the machines and running mod_proxy on the back-end
machines as well? That would be pretty ineffecient and I have never tried
that. mod_backhand running on a "front-end" machine is effectively a proxy
server. It will give you all of the advantages of running a two-tiered proxy
set up.

So all machines are publicly accessible? (given they have 1.2.3.x) I am
confused. :-( When people use mod_backhand and they say "proxy," they are
referring to mod_backhand's built-in proxy mechanism, are you?

Which page more adequately describes your set up?
(1) http://www.backhand.org/ApacheCon2000/EU/img13.htm
(2) http://www.backhand.org/ApacheCon2000/EU/img8.htm

> Here's the setup:
> (fake official IPs and domain names)
> machine(n)
> eth0: 1.2.3.21(n) 1.2.3.0/24
> eth1: 192.168.14.4(n) 192.168.14.0/24
> Multicast not explicitly bound to an interface via route, but tcpdump
> shows the traffic at eth0, no traffic at eth1.

The multicast packets going out the "wrong" interface is a Linux-ism.

> Linux version 2.2.16-SMP (root@SMP_X86.suse.de) (gcc version 2.95.2
> 19991024 (release))
> 2 CPUs, 2GB RAM, HW-Raid UW-SCSI drives, 100BaseT 3Com network adaptors.
>
> One more machine (similar configuration) working as log server via
> spread
> Apache 1.3.14 with PHP 4.04RC6, mod_backhand 1.1.1p2, mod_log_spread
> PHP is commonly used at most pages, Session management is active and
> vital, PHPSESSID is processed as described in README.bySession.
> php makes connections to Postgres (local and remote) and Oracle (remote)
> Databases.

This is a brand new candidacy function, so perhaps the bug lies here. It has
not been tested heavily as the README points out. I assume you are using
cookies for session management, right?

> <IfModule mod_backhand.c>
> UnixSocketDir /var/state/backhand
> MulticastStats 192.168.14.40 225.220.221.20:4445,1
> AcceptStats 192.168.14.0/24
> AcceptStats 1.2.3.0/24 <-- Without this line the 192.168.14.4x
> servers do not see each other - strange

This is not strange. You are using Linux perhaps? The multicast address you
are using is going out eth0... So when it comes in, the source IP is
1.2.3.x... This can be fixed in Linux, by explicitly routing multicast
addresses out of the eth1 interface.

A very quick fix is to use broadcast:
MulticastStats 192.168.14.40 192.168.14.255:4445
This will use broadcast packets instead, which should work fine.

> BackhandSelfRedirect On

Be aware that this directive will redirect the request to the local machine
using a proxy. Unless you are running two Apache instances on the same
machine, this doesn't make much since. It seems like you are running two
virtual hosts out of the same apache instance. Why? What to you stand to
gain over running everything Apache-related over the 1.2.3.x interface?

If you are trying to use mod_backhand a two-tiered http accelerator, you need
to run a "lean-and-mean" version of Apache will nothing but mod_backhand and
Listen on 1.2.3.x:80 and then run the big-and-bulky Apahce+mod_* (without
mod_backhand) and Listen on 192.168.14.*:80. Turn BackhandSelfRedirect On, up
the TCP buffers and then you will see the "HTTP acceleration" that is promised
with this type of two-tiered approach.

--
Theo Schlossnagle
1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7

[mod_backhand-users] child processes segfaulting, wrong client ip-adresses [ In reply to ]

sean-apache-backhand-users at chittenden

Feb 3, 2001, 6:57 PM

Post #3 of 4 (768 views)

Permalink

--ep0oHQY+/Gbo/zt0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Can I venture to guess that it's a permissions issue? I've had this probl=
em before with permission problems. -sc

On Sat, Feb 03, 2001 at 08:38:43PM -0500, Theo E. Schlossnagle wrote:
> Delivered-To: sean-apache-modbackhand-users@chittenden.org
> Date: Sat, 03 Feb 2001 20:38:43 -0500
> From: "Theo E. Schlossnagle" <jesus@omniti.com>
> Organization: Center for Networking and Distributed Systems
> X-Mailer: Mozilla 4.76 [en] (X11; U; Linux 2.2.18 i686)
> X-Accept-Language: en
> To: Tilman Kastner <kastner@devicen.de>
> CC: "backhand-users@lists.backhand.org" <backhand-users@lists.backhand.or=
> Subject: Re: [mod_backhand-users] child processes segfaulting, wrong clie=
nt=20
> ip-adresses
> Errors-To: backhand-users-admin@lists.backhand.org
> X-BeenThere: backhand-users@lists.backhand.org
> X-Mailman-Version: 2.0beta2
> Precedence: bulk
> List-Id: mod_backhand -- users list <backhand-users.lists.backhand.org>
>=20
> Tilman Kastner wrote:
> > after some configuration work, i finally managed to get mod_backhand up
> > and running.
> >=20
> > Unfortunately, two problems still remain, and I have no idea how to
> > investigate any further:
> >=20
> > The first one are occasionaly dying processes at the machine working as
> > proxy. Currently, this
> > machine handles approx. a few requests a second or less, so load should
> > not be a problem. But in
> > the apache error log looks like this:
> >=20
> > <snip>
> > [Sat Feb 3 10:13:01 2001] [notice] child pid 15606 exit signal
> > Segmentation fault (11)
> > ... many many more ....
> > [Sat Feb 3 13:59:15 2001] [notice] child pid 18893 exit signal
> > Segmentation fault (11)
> > <snip>
>=20
> Ouch. Well, I would wager this is mod_backhand. It looks like to proxyi=
ng
> children are dying. I thought I had cleaned up all of these, but I guess=
I
> didn't or introduced a new one. Turning on debugging will help better de=
fine
> what is breaking -- see below.
>=20
> Also, what are your Backhand directives?
> Backhand bySession is one of them I assume, what are the others?
>=20
> > This segfaulting did not occur before i installed mod_backhand. It only
> > happens at the server
> > answering the requests from remote, those working as proxies on other
> > machines are perfectly
> > fine.
>=20
> I don't understand your setup then... A request only ever gets redirected
> once. So, how do you know the other proxies wouldn't exhibit this proble=
m is
> they were queried directly. If they are only queried via a "backhanded"
> request, then they will no proxy (this eliminates requests from ping-pong=
ing
> indefinitely.) mod_backhand will only ever act like a proxy is the reque=
st
> originated from a "remote" machine.
>=20
> > I'm not an expert in debugging running applications, so any hint how to
> > receive more information
> > about the dying processes would be appreciated.
>=20
> Apache modules are actually pretty difficult to debug. If you can repeat=
this
> problem with *very little* traffic, then it is easier. If you can turn t=
he
> instance up and make only a few requests and repeat the problem, then it
> shouldn't be too hard to troubleshoot. Otherwise, the error log will be
> pretty big :-)
>=20
> First, since we think it is mod_backhand that is at fault, turn on all of=
the
> debugging:
> BackhandLogLevel +mbcsall
> BackhandLogLevel +netall
> BackhandLogLevel +dcsnall
>=20
> Delete your error log (so it starts fresh) and then run Apache. Let reque=
sts
> come in until you see a segfault and then turn off apache. Save the error
> file and send it to me via email. If you look in there, the last mod_bac=
khand
> debugging statement before the segfault should help narrow the window of =
code
> that could have problems.
>=20
> > Second: It seems that mod_backhand gets sometimes confused about the
> > clients' IP adresses.
> >=20
> > If I reload repeatedly the /backhand/ status handler, it sometimes
> > (approx 10%) reports
> > the wrong REMOTE_ADDR. Same happens to php-scripts echoing or processing
> > the REMOTE_ADDR.
>=20
> You you notice any similarities in the reuqests that trigger these. For
> instance are they actually backhanded? If you backhand your whole site, =
it
> will end up backhanding your /backhand/ page, an annoying side effect. (A=
dd
> "Backhand off" inside your <Location /backhand/> clause to fix this. Do =
you
> notice that the REMOTE_ADDR is only wrong when the response comes from a
> machine you didn't send the request to (it was backhanded) or vice-versa?
>=20
> > These wrong addresses usually belong to clients who are at the same
> > time making requests to the system. As we do things like content
> > customisation on an clients IP address
> > base, this is no good (Even the logging at the proxy-servers shows the
> > wrong adresses, while the external
> > server logs the right one)
>=20
> I am confused about your set up. What proxy servers? Are you running
> mod_backhand on all the machines and running mod_proxy on the back-end
> machines as well? That would be pretty ineffecient and I have never tried
> that. mod_backhand running on a "front-end" machine is effectively a pro=
xy
> server. It will give you all of the advantages of running a two-tiered p=
roxy
> set up.
>=20
> So all machines are publicly accessible? (given they have 1.2.3.x) I am
> confused. :-( When people use mod_backhand and they say "proxy," they are
> referring to mod_backhand's built-in proxy mechanism, are you?
>=20
> Which page more adequately describes your set up?
> (1) http://www.backhand.org/ApacheCon2000/EU/img13.htm
> (2) http://www.backhand.org/ApacheCon2000/EU/img8.htm
>=20
> > Here's the setup:
> > (fake official IPs and domain names)
> > machine(n)
> > eth0: 1.2.3.21(n) 1.2.3.0/24
> > eth1: 192.168.14.4(n) 192.168.14.0/24
> > Multicast not explicitly bound to an interface via route, but tcpdump
> > shows the traffic at eth0, no traffic at eth1.
>=20
> The multicast packets going out the "wrong" interface is a Linux-ism.
>=20
> > Linux version 2.2.16-SMP (root@SMP_X86.suse.de) (gcc version 2.95.2
> > 19991024 (release))
> > 2 CPUs, 2GB RAM, HW-Raid UW-SCSI drives, 100BaseT 3Com network adaptors.
> >
> > One more machine (similar configuration) working as log server via
> > spread
> > Apache 1.3.14 with PHP 4.04RC6, mod_backhand 1.1.1p2, mod_log_spread
> > PHP is commonly used at most pages, Session management is active and
> > vital, PHPSESSID is processed as described in README.bySession.
> > php makes connections to Postgres (local and remote) and Oracle (remote)
> > Databases.
>=20
> This is a brand new candidacy function, so perhaps the bug lies here. It=
has
> not been tested heavily as the README points out. I assume you are using
> cookies for session management, right?
>=20
> > <IfModule mod_backhand.c>
> > UnixSocketDir /var/state/backhand
> > MulticastStats 192.168.14.40 225.220.221.20:4445,1
> > AcceptStats 192.168.14.0/24
> > AcceptStats 1.2.3.0/24 <-- Without this line the 192.168.14.4x
> > servers do not see each other - strange
>=20
> This is not strange. You are using Linux perhaps? The multicast address=
you
> are using is going out eth0... So when it comes in, the source IP is
> 1.2.3.x... This can be fixed in Linux, by explicitly routing multicast
> addresses out of the eth1 interface.
>=20
> A very quick fix is to use broadcast:
> MulticastStats 192.168.14.40 192.168.14.255:4445
> This will use broadcast packets instead, which should work fine.
>=20
> > BackhandSelfRedirect On
>=20
> Be aware that this directive will redirect the request to the local machi=
ne
> using a proxy. Unless you are running two Apache instances on the same
> machine, this doesn't make much since. It seems like you are running two
> virtual hosts out of the same apache instance. Why? What to you stand to
> gain over running everything Apache-related over the 1.2.3.x interface?
>=20
> If you are trying to use mod_backhand a two-tiered http accelerator, you =
need
> to run a "lean-and-mean" version of Apache will nothing but mod_backhand =
and
> Listen on 1.2.3.x:80 and then run the big-and-bulky Apahce+mod_* (without
> mod_backhand) and Listen on 192.168.14.*:80. Turn BackhandSelfRedirect O=
n, up
> the TCP buffers and then you will see the "HTTP acceleration" that is pro=
mised
> with this type of two-tiered approach.
>=20
> --=20
> Theo Schlossnagle
> 1024D/A8EBCF8F/13BD 8C08 6BE2 629A 527E 2DC2 72C2 AD05 A8EB CF8F
> 2047R/33131B65/71 F7 95 64 49 76 5D BA 3D 90 B9 9F BE 27 24 E7
>=20
> _______________________________________________
> backhand-users mailing list
> backhand-users@lists.backhand.org
> http://lists.backhand.org/mailman/listinfo/backhand-users

--ep0oHQY+/Gbo/zt0
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAjp8tyQACgkQn09c7x7d+q031ACgpZiXNbCo7/ami4F7erxz8biu
F+QAnRMu5sAJLpq0guzFXU3sAg3XdE8c
=lDUd
-----END PGP SIGNATURE-----

--ep0oHQY+/Gbo/zt0--

[mod_backhand-users] child processes segfaulting, wrong client ip-adresses [ In reply to ]

kastner at devicen

Feb 4, 2001, 9:49 AM

Post #4 of 4 (770 views)

Permalink

Thank you for the quick reply!

"Theo E. Schlossnagle" wrote:
>
> > <snip>
> > [Sat Feb 3 10:13:01 2001] [notice] child pid 15606 exit signal
> > Segmentation fault (11)
> > ... many many more ....
> > [Sat Feb 3 13:59:15 2001] [notice] child pid 18893 exit signal
> > Segmentation fault (11)
> > <snip>
>
> Ouch. Well, I would wager this is mod_backhand. It looks like to proxying
> children are dying. I thought I had cleaned up all of these, but I guess I
> didn't or introduced a new one. Turning on debugging will help better define
> what is breaking -- see below.
>
> Also, what are your Backhand directives?
> Backhand bySession is one of them I assume, what are the others?

Backhand by Age
Bachand byLoad

>
> > This segfaulting did not occur before i installed mod_backhand. It only
> > happens at the server
> > answering the requests from remote, those working as proxies on other
> > machines are perfectly
> > fine.
>
> I don't understand your setup then... A request only ever gets redirected
> once. So, how do you know the other proxies wouldn't exhibit this problem is
> they were queried directly. If they are only queried via a "backhanded"
> request, then they will no proxy (this eliminates requests from ping-ponging
> indefinitely.) mod_backhand will only ever act like a proxy is the request
> originated from a "remote" machine.
>

OK. The basic idea behind my setup is:

Have a frontend server at every machine listening for external requests,
spread over
all machines via DNS round robin, Big/IP, etc. Let this server backhand
every request
to the backend servers on every machine. Of course, this would make more
sense when using
a slim apache as frontend, but I wanted to test this with the existing
servers first, before
fiddling around with two apaches on every machine...

What we want is completely interchangable machines with multiple
external interfaces (Reliability)
combined with the sophisticated load balancing from mod_backhand (A big
THANK YOU to all coders out
there by the way :-) )

> > I'm not an expert in debugging running applications, so any hint how to
> > receive more information
> > about the dying processes would be appreciated.
>
> Apache modules are actually pretty difficult to debug. If you can repeat this
> problem with *very little* traffic, then it is easier. If you can turn the
> instance up and make only a few requests and repeat the problem, then it
> shouldn't be too hard to troubleshoot. Otherwise, the error log will be
> pretty big :-)

> First, since we think it is mod_backhand that is at fault, turn on all of the
> debugging:
> BackhandLogLevel +mbcsall
> BackhandLogLevel +netall
> BackhandLogLevel +dcsnall
>
> Delete your error log (so it starts fresh) and then run Apache. Let requests
> come in until you see a segfault and then turn off apache. Save the error
> file and send it to me via email. If you look in there, the last mod_backhand
> debugging statement before the segfault should help narrow the window of code
> that could have problems.

So did I. The log is 63Mb in size, tgz is 1,6Mb. I guess I should cut
out the relevant parts :-)

>
> > Second: It seems that mod_backhand gets sometimes confused about the
> > clients' IP adresses.
> >
> > If I reload repeatedly the /backhand/ status handler, it sometimes
> > (approx 10%) reports
> > the wrong REMOTE_ADDR. Same happens to php-scripts echoing or processing
> > the REMOTE_ADDR.
>
> You you notice any similarities in the reuqests that trigger these. For
> instance are they actually backhanded? If you backhand your whole site, it
> will end up backhanding your /backhand/ page, an annoying side effect. (Add
> "Backhand off" inside your <Location /backhand/> clause to fix this. Do you
> notice that the REMOTE_ADDR is only wrong when the response comes from a
> machine you didn't send the request to (it was backhanded) or vice-versa?

Only backhanded processes have this confused addresses. I put the
Backhand off directive
into the backhand <Location> handler as you mentioned, and the
REMOTE_ADDR is perfectly
fine there. For debugging I altered the backhand-test cgi-skript by
displaying the
REMOTE_ADDR. As the /cgi-bin/ directory gets backhanded, the strange
behaviour is
obvious. Sometimes even the $HTTP_BACKHANDPROXIED value is screwed up
:-(

Perhaps I loaded mad_backhand by accident ;-)

>
> > These wrong addresses usually belong to clients who are at the same
> > time making requests to the system. As we do things like content
> > customisation on an clients IP address
> > base, this is no good (Even the logging at the proxy-servers shows the
> > wrong adresses, while the external
> > server logs the right one)
>
> I am confused about your set up. What proxy servers? Are you running
> mod_backhand on all the machines and running mod_proxy on the back-end
> machines as well? That would be pretty ineffecient and I have never tried
> that. mod_backhand running on a "front-end" machine is effectively a proxy
> server. It will give you all of the advantages of running a two-tiered proxy
> set up.

no mod_proxy involved at all, I meant the proxy-capability of
mod_backhand

>
> So all machines are publicly accessible? (given they have 1.2.3.x) I am
> confused. :-( When people use mod_backhand and they say "proxy," they are
> referring to mod_backhand's built-in proxy mechanism, are you?
>
> Which page more adequately describes your set up?
> (1) http://www.backhand.org/ApacheCon2000/EU/img13.htm
> (2) http://www.backhand.org/ApacheCon2000/EU/img8.htm
(1)

>
> > Here's the setup:
> > (fake official IPs and domain names)
> > machine(n)
> > eth0: 1.2.3.21(n) 1.2.3.0/24
> > eth1: 192.168.14.4(n) 192.168.14.0/24
> > Multicast not explicitly bound to an interface via route, but tcpdump
> > shows the traffic at eth0, no traffic at eth1.
>
> The multicast packets going out the "wrong" interface is a Linux-ism.
>
> > Linux version 2.2.16-SMP (root@SMP_X86.suse.de) (gcc version 2.95.2
> > 19991024 (release))
> > 2 CPUs, 2GB RAM, HW-Raid UW-SCSI drives, 100BaseT 3Com network adaptors.
> >
> > One more machine (similar configuration) working as log server via
> > spread
> > Apache 1.3.14 with PHP 4.04RC6, mod_backhand 1.1.1p2, mod_log_spread
> > PHP is commonly used at most pages, Session management is active and
> > vital, PHPSESSID is processed as described in README.bySession.
> > php makes connections to Postgres (local and remote) and Oracle (remote)
> > Databases.
>
> This is a brand new candidacy function, so perhaps the bug lies here. It has
> not been tested heavily as the README points out. I assume you are using
> cookies for session management, right?

Yes, we use cookies. BTW: Enabling or disabling the bySession directive
makes no
difference regarding the "screwed up ip addresses" problem.

>
> > <IfModule mod_backhand.c>
> > UnixSocketDir /var/state/backhand
> > MulticastStats 192.168.14.40 225.220.221.20:4445,1
> > AcceptStats 192.168.14.0/24
> > AcceptStats 1.2.3.0/24 <-- Without this line the 192.168.14.4x
> > servers do not see each other - strange
>
> This is not strange. You are using Linux perhaps? The multicast address you
> are using is going out eth0... So when it comes in, the source IP is
> 1.2.3.x... This can be fixed in Linux, by explicitly routing multicast
> addresses out of the eth1 interface.
OK, got it.

>
> A very quick fix is to use broadcast:
> MulticastStats 192.168.14.40 192.168.14.255:4445
> This will use broadcast packets instead, which should work fine.

I have no problems with having the multicast on eth0 at the moment.
I will set up Multicast routes for eth1 when network load becomes an
issue...

>
> > BackhandSelfRedirect On
>
> Be aware that this directive will redirect the request to the local machine
> using a proxy. Unless you are running two Apache instances on the same
> machine, this doesn't make much since. It seems like you are running two
> virtual hosts out of the same apache instance. Why? What to you stand to
> gain over running everything Apache-related over the 1.2.3.x interface?

As I mentioned earlier, this is a temporary setup. Finally, we will have
two instances.

Tilman