Mailing List Archive

[lvs-users] setting weight to 0 (no persistence!), but still new connections
Hi all,

we have been using lvs in production with redhat rhel5 with direct routing successfully for years now.
Maintenance of server was no problem, because setting the weight to 0 avoided new connections and we only had to wait
for the current tcp-sessions to end on the maintained server.

But with our new rhel6-cluster we are having most of the time the problem that still new http-connections
are being directed to the server that has a weight of 0, although we don't use persistent connections and there is no
keepalive enabled in httpd.conf, so each tcp-connection ends after very short time.


config:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:http wlc
-> lxweb1-stage:http Local 1 0 0
-> lxweb2-stage:http Route 1 0 0



Running stress test with ab from another server:
ab -n 600000 -c 30 http://192.168.0.1/


IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:http wlc
-> lxweb1-stage:http Local 1 12 14323
-> lxweb2-stage:http Route 1 10 13868


setting weight to 0:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:http wlc
-> lxweb1-stage:http Local 0 7 14317
-> lxweb2-stage:http Route 1 13 13888


a few minutes minutes later:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.1:http wlc
-> lxweb1-stage:http Local 0 12 14304
-> lxweb2-stage:http Route 1 8 13902



Running ipvsadm -Lnc | grep -v FIN_WAIT shows, that there are only a few established connections
and some SYN_RECV. Repeating this command after a few seconds also makes clear, that there are no ports
being reused from the ESTABLISHED-connections. So I have no idea why there are still active connections on the
server with weight 0.


Any hint is appreciated
Thanks very much for your effort

Best regards
Tim







_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] setting weight to 0 (no persistence!), but still new connections [ In reply to ]
Hello,

On Thu, 25 Apr 2013, Tim.Einmahl@kba.de wrote:

> Hi all,
>
> we have been using lvs in production with redhat rhel5 with direct routing successfully for years now.
> Maintenance of server was no problem, because setting the weight to 0 avoided new connections and we only had to wait
> for the current tcp-sessions to end on the maintained server.
>
> But with our new rhel6-cluster we are having most of the time the problem that still new http-connections
> are being directed to the server that has a weight of 0, although we don't use persistent connections and there is no
> keepalive enabled in httpd.conf, so each tcp-connection ends after very short time.
>
>
> config:
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 1 0 0
> -> lxweb2-stage:http Route 1 0 0
>
>
>
> Running stress test with ab from another server:
> ab -n 600000 -c 30 http://192.168.0.1/
>
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 1 12 14323
> -> lxweb2-stage:http Route 1 10 13868
>
>
> setting weight to 0:
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 0 7 14317
> -> lxweb2-stage:http Route 1 13 13888
>
>
> a few minutes minutes later:
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 0 12 14304
> -> lxweb2-stage:http Route 1 8 13902
>
>
>
> Running ipvsadm -Lnc | grep -v FIN_WAIT shows, that there are only a few established connections
> and some SYN_RECV. Repeating this command after a few seconds also makes clear, that there are no ports
> being reused from the ESTABLISHED-connections. So I have no idea why there are still active connections on the
> server with weight 0.

Such tests from single client IP can lead to
client port reuse when no new IPVS connections are created.
Are you really sure that such port reuse does not happen?

> Any hint is appreciated
> Thanks very much for your effort
>
> Best regards
> Tim

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] setting weight to 0 (no persistence!), but still new connections [ In reply to ]
Hello Julian,

the reusage of client ports was something I considered as a problem,
but running "ipvsadm -Lnc | grep ESTABLISHED" several times while the ab-test is running shows,
that any client port is different each time I checked them.
And if reusage of client ports would be the cause of the problem, then I would not understand why
this perfectly works when running ab-stress-test from the same machine to the rhel5-cluster.
Or do I have to check this in a different way?
I am a little bit at a loss with this at the moment.


Best regards
Tim




Hello,

On Thu, 25 Apr 2013, Tim.Einmahl@kba.de wrote:

> Hi all,
>
> we have been using lvs in production with redhat rhel5 with direct routing successfully for years now.
> Maintenance of server was no problem, because setting the weight to 0 avoided new connections and we only had to wait
> for the current tcp-sessions to end on the maintained server.
>
> But with our new rhel6-cluster we are having most of the time the problem that still new http-connections
> are being directed to the server that has a weight of 0, although we don't use persistent connections and there is no
> keepalive enabled in httpd.conf, so each tcp-connection ends after very short time.
>
>
> config:
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 1 0 0
> -> lxweb2-stage:http Route 1 0 0
>
>
>
> Running stress test with ab from another server:
> ab -n 600000 -c 30 http://192.168.0.1/
>
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 1 12 14323
> -> lxweb2-stage:http Route 1 10 13868
>
>
> setting weight to 0:
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 0 7 14317
> -> lxweb2-stage:http Route 1 13 13888
>
>
> a few minutes minutes later:
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.0.1:http wlc
> -> lxweb1-stage:http Local 0 12 14304
> -> lxweb2-stage:http Route 1 8 13902
>
>
>
> Running ipvsadm -Lnc | grep -v FIN_WAIT shows, that there are only a few established connections
> and some SYN_RECV. Repeating this command after a few seconds also makes clear, that there are no ports
> being reused from the ESTABLISHED-connections. So I have no idea why there are still active connections on the
> server with weight 0.

Such tests from single client IP can lead to
client port reuse when no new IPVS connections are created.
Are you really sure that such port reuse does not happen?

> Any hint is appreciated
> Thanks very much for your effort
>
> Best regards
> Tim

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users