Mailing List Archive

Hello,

On Fri, 23 Feb 2001, John P . Looney wrote:

> Hi - I was wondering if it's possible to make failover more seemless. At
> the moment, I have mon setup, polling services every few seconds.
>
> However, if a service goes down, and incoming new connections for a few
> seconds get a "connection refused", until mon can change the ip_vs tables
> with ipvsadm.

Hm, what kind of service likes very much to die? May be you
stop the service without excluding it from the cluster (or setting the
weight to 0)? You are talking for buggy services or for bad cluster
software?

> Is there a reason that no one has put some login in the kernel to "try
> open the connection, if it's there, then let it through, otherwise go onto
> the next one", like Cisco Local Directors do ?

That sounds like a kernel space proxy server. Such setup
works for NAT mode only?

> Kate
>
> --
> When I say 'free', I mean 'free': free from bond, of chain or command:
> to go where you will, even to Mordor, Saruman, if you desire. "
> -- Gandalf, paraphrasing the choice between Free and Non-free software


Regards

--
Julian Anastasov <ja@ssi.bg>

On Fri, Feb 23, 2001 at 11:48:18AM +0200, Julian Anastasov mentioned:
> > However, if a service goes down, and incoming new connections for a few
> > seconds get a "connection refused", until mon can change the ip_vs tables
> > with ipvsadm.
> Hm, what kind of service likes very much to die? May be you
> stop the service without excluding it from the cluster (or setting the
> weight to 0)? You are talking for buggy services or for bad cluster
> software?

Buggy services, or buggy hardware.

If I have two mail servers, and one machine crashes. "mon" will notice
after 2 or 3 seconds. It will run ipvsadm and take it out of the cluster.

But, until "mon" does this, anyone trying to connect will fail....

> > Is there a reason that no one has put some login in the kernel to "try
> > open the connection, if it's there, then let it through, otherwise go onto
> > the next one", like Cisco Local Directors do ?
> That sounds like a kernel space proxy server. Such setup
> works for NAT mode only?

Maybe. But is it for ipvs to do this, or should it be done by something
else ? Is there a reason why ipvs can't/shouldn't do it ? (I was thinking
of trying to do it myself, and wondered about the best way of starting..)

John

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software

On 2001-02-23T09:53:43,
"John P . Looney" <john@antefacto.com> said:

> > > Is there a reason that no one has put some login in the kernel to "try
> > > open the connection, if it's there, then let it through, otherwise go onto
> > > the next one", like Cisco Local Directors do ?
> > That sounds like a kernel space proxy server. Such setup
> > works for NAT mode only?
> Maybe. But is it for ipvs to do this, or should it be done by something
> else ? Is there a reason why ipvs can't/shouldn't do it ? (I was thinking
> of trying to do it myself, and wondered about the best way of starting..)

This depends - if what happens is that the failed node sends back an error
reply (RST or port unreachable), LVS could probably pretty easily set the
weight of the server to 0 automatically, or even completely drop it from the
service table.

However, you would still get a single refused connection because the packet
would be send back to the client. Hrm, one could drop the reply packet, and
the client would be reassigned to another real server when it sends the second
SYN packet (because it would just assume that the initial SYN was lost).

This just works with LVS NAT, as we don't see the reply packets from the real
servers in any of the other cases.

However, should the error occur at any later stage than the first packet, the
connection is broken for good, obviously.

If the server doesn't send back a connection refused but silently doesn't
reply (lets say, it was turned off), you will still need external monitoring
(mon, ldirectord) to catch this - however, as nothing was send back to the
client, it would still assume dropped packets and resend, resulting in the
connection succeeding, albeit delayed.

Sincerely,
Lars Marowsky-Brée <lmb@suse.de>

--
Perfection is our goal, excellence will be tolerated. -- J. Yahl

On Fri, Feb 23, 2001 at 11:03:27AM +0100, Lars Marowsky-Bree mentioned:
> If the server doesn't send back a connection refused but silently doesn't
> reply (lets say, it was turned off), you will still need external monitoring
> (mon, ldirectord) to catch this - however, as nothing was send back to the
> client, it would still assume dropped packets and resend, resulting in the
> connection succeeding, albeit delayed.

This is perfect. It should be assumed that failover is a non-frequent
occurance, so a non-optimal solution is preferred.

If ipvs tried to make a connection to the target server, that fails, so
it drops the reply. The client retries the connect. With standard round
robin, it'll go to the next server, and all will be fine.

But other algorithms will try send it to the same (dead) server, again
and again, and I'd be loathe to start playing with weights etc. Is it a
big job to have a test for "is the destinating server accepting packets"
for *each* and *every* packet ?

Kate

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software

On 2001-02-23T10:20:34,
"John P . Looney" <john@antefacto.com> said:

> But other algorithms will try send it to the same (dead) server, again
> and again, and I'd be loathe to start playing with weights etc. Is it a
> big job to have a test for "is the destinating server accepting packets"
> for *each* and *every* packet ?

Yes.

The best option really is:
- drop the error reply on the first packet, and deallocate any persistence
which might have been set for that server (removing the server from the list
of real servers might work nicely)
This should be an additional flag for VS/NAT services.

- have monitoring do that in case the server silently dies.

Sincerely,
Lars Marowsky-Brée <lmb@suse.de>

--
Perfection is our goal, excellence will be tolerated. -- J. Yahl

On Fri, Feb 23, 2001 at 11:27:29AM +0100, Lars Marowsky-Bree mentioned:
> The best option really is:
> - drop the error reply on the first packet, and deallocate any persistence
> which might have been set for that server (removing the server from the list
> of real servers might work nicely)
> This should be an additional flag for VS/NAT services.
>
> - have monitoring do that in case the server silently dies.

We were thinking about this.

How about getting the router to use netfilter to never send icmp
"connection refused" or "port unreachable" packets (whatever they are
called).

So, if someone connects in when the server has just gone down, they will
hear nothing. TCP will do an exponetial backoff, and retry - and will
eventually get a real host.

Also - when a host dies, and it's entries are removed with ipvsadm, is
ip_vs smart enough to remove entries of connections that still think they
are alive from it's hash tables ?

Kate

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software