Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Linux Virtual Server>
  3. Users
[lvs-users] local forwarding method not working
matias.fernandez at nine
Dec 13, 2013, 4:01 AM
Post #1 of 5 (3326 views)
Permalink
Dear lvs-users

We use LVS extensively with Keepalived for running redundant
load balanced setups. As part of the load balancer we sometimes
use what is called a sorry_server in Keepalived. This maps to a
local real server in LVS speak.

In recent versions we have found a change in behaviour that
seems to break that functionality of a local sorry_server. It boils
down to LVS not setting the forward mode to "Local" for addresses for
which there interfaces on the local node.

See the following examples. First on a Ubuntu 10.04.4 LTS:

# uname -a
Linux ubuntuserver1004 2.6.32-54-server #116-Ubuntu SMP Tue Nov 12 19:37:57 UTC 2013 x86_64 GNU/Linux

# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.0.2:80 wlc
-> 127.0.0.1:80 Local 1 0 0

In this example the forward mode is correctly set to "Local". In newer
versions this doesn't seem to work any more:

# uname -a
Linux ubuntuserver1204 3.8.0-34-generic #49~precise1-Ubuntu SMP Wed Nov 13 18:05:00 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

# ipvsadm -A -t 10.10.0.2:80
# ipvsadm -a -t 10.10.0.2:80 -r 127.0.0.1
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.0.2:80 wlc
-> 127.0.0.1:80 Route 1 0 0

Note that in this later cate the forward mode in the second example
has *NOT* been set to "Local".

This behaviour contradicts the documentation in the manual, which states:

> Regardless of the packet-forwarding mechanism specified, real servers for
> addresses for which there are interfaces on the local node will be use
> the local forwarding method, then packets for the servers will be passed to upper layer on the local node. This cannot be specified by ipvsadm, rather it set by the kernel
> as real servers are added or modified.

Is the observed behaviour desired or a bug in LVS or even the Kernel?

Regards
Matías



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] local forwarding method not working [ In reply to ]
leon at linux
Dec 13, 2013, 5:25 AM
Post #2 of 5 (3249 views)
Permalink
On Fri, 2013-12-13 at 13:01 +0100, "Matías E. Fernández" wrote:
> In recent versions we have found a change in behaviour that
> seems to break that functionality of a local sorry_server. It boils
> down to LVS not setting the forward mode to "Local" for addresses for
> which there interfaces on the local node.

I experienced this as well a couple of weeks ago. Solution/workaround
was to use LVS-NAT for a sorry server where I just put the address on
the loopback interface and have apache listen on it.

Here's a piece from my ldirectord.cf (I know you're using keepalived,
but the idea is the same) :

virtual=31.x.y.69:80
fallback=192.168.x.y masq
real=172.20.x.y:80 masq 100


kind regards,

Léon



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] local forwarding method not working [ In reply to ]
matias.fernandez at nine
Dec 13, 2013, 7:38 AM
Post #3 of 5 (3250 views)
Permalink
Dear Léon

On 2013-12-13, at 14:25, L.S. Keijser <leon@linux.nl> wrote:
> I experienced this as well a couple of weeks ago. Solution/workaround
> was to use LVS-NAT for a sorry server where I just put the address on
> the loopback interface and have apache listen on it.

Thank your for the input. Your workaround works with Keepalived too.

However, I would like to know wether the behaviour was changed on
purpose either in LVS or other parts of the Kernel or if it is
a regression or bug. In the first case the documentation should
be updated and Keepalived adapted, in the second I'd like to file
a bug report.

Regards
Matías


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] local forwarding method not working [ In reply to ]
ja at ssi
Dec 13, 2013, 12:26 PM
Post #4 of 5 (3263 views)
Permalink
Hello,

On Fri, 13 Dec 2013, "Matías E. Fernández" wrote:

> Dear lvs-users
>
> We use LVS extensively with Keepalived for running redundant
> load balanced setups. As part of the load balancer we sometimes
> use what is called a sorry_server in Keepalived. This maps to a
> local real server in LVS speak.
>
> In recent versions we have found a change in behaviour that
> seems to break that functionality of a local sorry_server. It boils
> down to LVS not setting the forward mode to "Local" for addresses for
> which there interfaces on the local node.

...

> Note that in this later cate the forward mode in the second example
> has *NOT* been set to "Local".
>
> This behaviour contradicts the documentation in the manual, which states:
>
> > Regardless of the packet-forwarding mechanism specified, real servers for
> > addresses for which there are interfaces on the local node will be use
> > the local forwarding method, then packets for the servers will be passed to upper layer on the local node. This cannot be specified by ipvsadm, rather it set by the kernel
> > as real servers are added or modified.
>
> Is the observed behaviour desired or a bug in LVS or even the Kernel?

It is a change that helps the synchronization:

===
commit fc604767613b6d2036cdc35b660bc39451040a47
Author: Julian Anastasov <ja@ssi.bg>
Date: Sun Oct 17 16:38:15 2010 +0300

ipvs: changes for local real server

...
- Now LOCALNODE is not set explicitly as forwarding
method in real server to allow the connections to provide
correct forwarding method to the backup server. Not sure if
this breaks tools that expect to see 'Local' real server type.
If needed, this can be supported with new flag IP_VS_DEST_F_LOCAL.
Now it should be possible connections in backup that lost
their fwmark information during sync to be forwarded properly
to their daddr, even if it is local address in the backup server.
By this way backup could be used as real server for DR or TUN,
for NAT there are some restrictions because tuple collisions
in conntracks can create problems for the traffic.
===

My understanding is that Local method just
passed the packet to the local server listening on VIP, without
modification. The same is done by Route method. But I still
don't understand what actually breaks in Keepalived. OTOH, if
NAT method is used, the destination address/port in packet
is modified to match the real server. In such case the
local server listens on RIP:RPORT, not on VIP:VPORT.

Regards

--
Julian Anastasov <ja@ssi.bg>
Re: [lvs-users] local forwarding method not working [ In reply to ]
matias.fernandez at nine
Dec 23, 2013, 2:30 AM
Post #5 of 5 (3225 views)
Permalink
Dear lvs-user

A colleague pointed out, that I had forgotten to reply to the
group. I'm forwarding my reply to Julian below.

I will try sending a patch for the man page of ipvsadm to Simon Horman
an contact the guys from Keepalived for possible solutions/workarounds.

Regards
Matías

On 2013-12-17, at 11:08, Matías E. Fernández <matias.fernandez@nine.ch> wrote:

> Dear Julian
>
>> commit fc604767613b6d2036cdc35b660bc39451040a47
>> Author: Julian Anastasov <ja@ssi.bg>
>> Date: Sun Oct 17 16:38:15 2010 +0300
>
> Thank you for the hint. This means that the corresponding passage
> in the man page should be removed, shouldn't it? Should I send a
> patch? To whom?
>
>> My understanding is that Local method just
>> passed the packet to the local server listening on VIP, without
>> modification. The same is done by Route method.
>
> I can confirm that using the "Route" method with ipvsadm directly
> works as expected.
>
>> But I still
>> don't understand what actually breaks in Keepalived. OTOH, if
>> NAT method is used, the destination address/port in packet
>> is modified to match the real server. In such case the
>> local server listens on RIP:RPORT, not on VIP:VPORT.
>
> The problem with keepalived ist that lb_kind is used for all real_server
> and sorry_server in virtual_server. This means that if you configure
> lb_kind to NAT, the same forwarding method will be used for a a sorry_server
> that happens to be on the local node.
> In earlier versions this use to work, because the forwarding method of "Local"
> was set regardless of what may have been specified.
>
> For me it looks like the man page of ipvsadm should be corrected, and
> keepalived adapted to the new behaviour of ipvsadm.
>
> Thank you for the help.
>
> Regards
> Matías


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal