Mailing List Archive

[lvs-users] ldirectord - Real server maintenance discarded by check
Hello everyone,

I have a question about ldirectord, we have a maintenance procedure where we set the real server weight to 0 so connections finish correctly, then we stop apache on the real server, sleep to let it close all its fcgi, then start the server, and finally put the weight back to 1.

The problem is when we start the server, ldirectord makes his check, sees the server is up and so add it back with a weight of 1, 1s later it rereads its configuration and see that the server should have a weight of 0, so it changes the weight back to 0.

During this short time period the server takes few thousands request and is not ready for it, it seems like a bug about the check system.

ldirectord has been upgraded to the latest version available on github, also we have tested the "maintenacedir" configuration, which behaves the same way as described above.

Here is the configuration:

# Global Directives
checktimeout=10
checkinterval=5
autoreload=yes
logfile="/var/log/ldirectord.log"
quiescent=yes
fork=yes
maintenancedir = /var/run/ldirectord_maintenance/

virtual=....3:80
real=....72:80 gate
real=....73:80 gate
real=....74:80 gate
real=....75:80 gate
service=http
request="/fcgi-bin/envmy.fcgi"
receive="Mysql table test : OK"
scheduler=rr
protocol=tcp
checktype=negotiate


Do you have any suggestion, any idea ? I have not found any bug related to this.

Thank you all by advance,

Regards,

Jeremy

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] ldirectord - Real server maintenance discarded by check [ In reply to ]
On Tue, Jan 14, 2014 at 12:26 PM, Jeremy Clerc <jclerc@weborama.com> wrote:

> Hello everyone,
>
> I have a question about ldirectord, we have a maintenance procedure where
> we set the real server weight to 0 so connections finish correctly, then we
> stop apache on the real server, sleep to let it close all its fcgi, then
> start the server, and finally put the weight back to 1.
>
> The problem is when we start the server, ldirectord makes his check, sees
> the server is up and so add it back with a weight of 1, 1s later it rereads
> its configuration and see that the server should have a weight of 0, so it
> changes the weight back to 0.
>
> During this short time period the server takes few thousands request and
> is not ready for it, it seems like a bug about the check system.
>

I was looking for the similar, but opposite functionality - on failover
when other director host takes traffic, in the beginning the weight of the
servers is 0 until they've been checked by the 'checkcommand'. So despite
using conntrack to keep connections live we get timeouts, as no traffic is
forwarded to the real servers.

So, it would be nice also to have ability to declare initial weight for the
connections with RS with positive weight until otherwise is proven by
checkcommand.

With best regards,
Timur Bakeyev.
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] ldirectord - Real server maintenance discarded by check [ In reply to ]
On 01/14/2014 06:26 AM, Jeremy Clerc wrote:
> Hello everyone,
>
> I have a question about ldirectord, we have a maintenance procedure where we set the real server weight to 0 so connections finish correctly, then we stop apache on the real server, sleep to let it close all its fcgi, then start the server, and finally put the weight back to 1.
>
> The problem is when we start the server, ldirectord makes his check, sees the server is up and so add it back with a weight of 1, 1s later it rereads its configuration and see that the server should have a weight of 0, so it changes the weight back to 0.
>
> During this short time period the server takes few thousands request and is not ready for it, it seems like a bug about the check system.
>
> ldirectord has been upgraded to the latest version available on github, also we have tested the "maintenacedir" configuration, which behaves the same way as described above.
>
> Here is the configuration:
>
> # Global Directives
> checktimeout=10
> checkinterval=5
> autoreload=yes
> logfile="/var/log/ldirectord.log"
> quiescent=yes
> fork=yes
> maintenancedir = /var/run/ldirectord_maintenance/
>
> virtual=....3:80
> real=....72:80 gate
> real=....73:80 gate
> real=....74:80 gate
> real=....75:80 gate
> service=http
> request="/fcgi-bin/envmy.fcgi"
> receive="Mysql table test : OK"
> scheduler=rr
> protocol=tcp
> checktype=negotiate
>
>
> Do you have any suggestion, any idea ? I have not found any bug related to this.
>
> Thank you all by advance,
>
> Regards,
>
> Jeremy
>
>
That seems like a bug in ldirectord. But without going through the
ldirectord code to fix this issue, you could add one more step to your
procedures:

1. set server weight to 0
2. stop apache2
** 3. create a file in 'maintenancedir' to remove the real server from
the VIP

And to bring it back, you'd wait until your work is done before removing
the maintenance file (as your last step).

The caveat here is that as far as I know, ldirectord does not
differentiate between real servers across VIPs. So if you have the same
real server in multiple VIPs, creating the maintenance file will remove
it from all those VIPs.

Hope this helps,

--
Khosrow


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] ldirectord - Real server maintenance discarded by check [ In reply to ]
----- Original Message -----
> From: "Khosrow Ebrahimpour" <khosrow.ebrahimpour@ssc-spc.gc.ca>
> To: lvs-users@linuxvirtualserver.org
> Sent: Jeudi 16 Janvier 2014 21:55:36
> Subject: Re: [lvs-users] ldirectord - Real server maintenance discarded by check
>
> On 01/14/2014 06:26 AM, Jeremy Clerc wrote:
> > Hello everyone,
> >
> > I have a question about ldirectord, we have a maintenance procedure
> > where we set the real server weight to 0 so connections finish
> > correctly, then we stop apache on the real server, sleep to let it
> > close all its fcgi, then start the server, and finally put the
> > weight back to 1.
> >
> > The problem is when we start the server, ldirectord makes his
> > check, sees the server is up and so add it back with a weight of
> > 1, 1s later it rereads its configuration and see that the server
> > should have a weight of 0, so it changes the weight back to 0.
> >
> > During this short time period the server takes few thousands
> > request and is not ready for it, it seems like a bug about the
> > check system.
> >
> > ldirectord has been upgraded to the latest version available on
> > github, also we have tested the "maintenacedir" configuration,
> > which behaves the same way as described above.
> >
> > Here is the configuration:
> >
> > # Global Directives
> > checktimeout=10
> > checkinterval=5
> > autoreload=yes
> > logfile="/var/log/ldirectord.log"
> > quiescent=yes
> > fork=yes
> > maintenancedir = /var/run/ldirectord_maintenance/
> >
> > virtual=....3:80
> > real=....72:80 gate
> > real=....73:80 gate
> > real=....74:80 gate
> > real=....75:80 gate
> > service=http
> > request="/fcgi-bin/envmy.fcgi"
> > receive="Mysql table test : OK"
> > scheduler=rr
> > protocol=tcp
> > checktype=negotiate
> >
> >
> > Do you have any suggestion, any idea ? I have not found any bug
> > related to this.
> >
> > Thank you all by advance,
> >
> > Regards,
> >
> > Jeremy
> >
> >
> That seems like a bug in ldirectord. But without going through the
> ldirectord code to fix this issue, you could add one more step to
> your
> procedures:
>
> 1. set server weight to 0
> 2. stop apache2
> ** 3. create a file in 'maintenancedir' to remove the real server
> from
> the VIP
>
> And to bring it back, you'd wait until your work is done before
> removing
> the maintenance file (as your last step).
>
> The caveat here is that as far as I know, ldirectord does not
> differentiate between real servers across VIPs. So if you have the
> same
> real server in multiple VIPs, creating the maintenance file will
> remove
> it from all those VIPs.
>
> Hope this helps,
>
> --
> Khosrow
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list -
> lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

Hello,

Thank you for your answer.

We have been doing some testing today, and there is a solution but there is still a bug.

If you look at my conf, we use "fork=yes", in the man it says :
fork = yes | no
If yes, ...... This will "increase response times to changes in real server status" in configurations with many virtual servers. .......

So one of my colleague thought about changing fork to "no", and it worked, with weight 0 or file in maintenance dir, server style stays with weight 0 after apache restart and a good check.

It appears using fork, so it why I think there is still bug, I will fill an issue on github.

Thank you again,

Jeremy

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users