Mailing List Archive

[lvs-users] CentOS/RHEL 6 ipvs issue with DR (different interfaces and networks)
Hello everyone,

I've been running ipvs on our systems for quite a long time, we recently
upgraded some servers from CentOS 5 to CentOS 6 (which ships with kernel
2.6.32). Our setup is ipvs DR:

loadbalancer (two interfaces: eth0 public and eth1 private)
VIP configured on load balancer on eth0:0

real servers (two interfaces: eth0 public and eth1 private)
VIP configured on real servers lo:0


Requests from the web come to the loadbalancer through public interface
eth0, they get forwarded through private interface eth1 to real servers'
private interface eth1. lo:0 on real servers has VIP. Traffic is supposed
to go out from eth0 (public interface) on real servers, but there is
nothing going out. Packets are being discarded.

I ran tcpdump on real servers and saw packets come in through eth1, but
there was nothing going out, kernel was dropping the packets.

After talking with my senior guy, we figured out that RHEL/CentOS 6
behavior has changed in regards to asymmetric routes and reverse path
filtering. By default RHEL 6 now will employ strict reverse path filtering,
meaning if you have different networks and routes on the interface where
traffic comes in from, kernel will drop those packets.

You need to change the settings in order for IPVS DR to work if you have
private and public networks or different networks on different interfaces.
Those settings are:
echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

or if you would like to just do the interface where loadbalancer is sending
traffic to, then:
echo 2 > /proc/sys/net/ipv4/conf/<interface of real server ip>/rp_filter
(in my case eth1, so echo 2 > /proc/sys/net/ipv4/conf/eth1/rp_filter)

Please note value of 1 does not have the same behavior on RHEL6 as it did
on RHEL5.

Hope this helps others who encounter the issue.

Cheers,
Alex
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] CentOS/RHEL 6 ipvs issue with DR (different interfaces and networks) [ In reply to ]
On 10/18/13 12:50 AM, Alex Attarian wrote:
> You need to change the settings in order for IPVS DR to work if you have
> private and public networks or different networks on different interfaces.
> Those settings are:
> echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
>
> or if you would like to just do the interface where loadbalancer is sending
> traffic to, then:
> echo 2 > /proc/sys/net/ipv4/conf/<interface of real server ip>/rp_filter
> (in my case eth1, so echo 2 > /proc/sys/net/ipv4/conf/eth1/rp_filter)
We have a large LVS environment with RHEL6 for both balancers and real
servers. On most real servers we have rp_filter set to 1. Our network
config is pretty much the same as yours (VIPs on lo, usually from a
different subnet).

The only systems where we do have rp_filter set to 0 (I need to back and
check why we picked 0 instead of 2 - 0 does work however) are systems
which get packets from balancers using IPIP tunnels.
>
> Please note value of 1 does not have the same behavior on RHEL6 as it did
> on RHEL5.
Did you open a RedHat case on this behavior? Did they give you a BZ for it?

Not saying changing rp_filter wasn't the right thing to do, but I've not
experienced a difference in how it functions in the last 12 years,
including with RHEL6. On the few RHEL5 systems I have default rp_filter
value is 1, so I'd expect them to behave the same.

David

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] CentOS/RHEL 6 ipvs issue with DR (different interfaces and networks) [ In reply to ]
On Fri, Oct 18, 2013 at 4:07 AM, David Coulson <david@davidcoulson.net>wrote:

>
> On 10/18/13 12:50 AM, Alex Attarian wrote:
> > You need to change the settings in order for IPVS DR to work if you have
> > private and public networks or different networks on different
> interfaces.
> > Those settings are:
> > echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> > echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
> >
> > or if you would like to just do the interface where loadbalancer is
> sending
> > traffic to, then:
> > echo 2 > /proc/sys/net/ipv4/conf/<interface of real server ip>/rp_filter
> > (in my case eth1, so echo 2 > /proc/sys/net/ipv4/conf/eth1/rp_filter)
> We have a large LVS environment with RHEL6 for both balancers and real
> servers. On most real servers we have rp_filter set to 1. Our network
> config is pretty much the same as yours (VIPs on lo, usually from a
> different subnet).
>
> The only systems where we do have rp_filter set to 0 (I need to back and
> check why we picked 0 instead of 2 - 0 does work however) are systems
> which get packets from balancers using IPIP tunnels.
> >
> > Please note value of 1 does not have the same behavior on RHEL6 as it did
> > on RHEL5.
> Did you open a RedHat case on this behavior? Did they give you a BZ for it?
>
> Not saying changing rp_filter wasn't the right thing to do, but I've not
> experienced a difference in how it functions in the last 12 years,
> including with RHEL6. On the few RHEL5 systems I have default rp_filter
> value is 1, so I'd expect them to behave the same.
>
> David
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



David,

yes both values 0 and 2 work, however 0 does no source checking at all, 2
does a loose checking:

2 = basically if any interface can actually route any of the networks the
packet is coming in from, then allow.

rp_filter - INTEGER
0 - No source validation.
1 - Strict mode as defined in RFC3704 Strict Reverse Path
Each incoming packet is tested against the FIB and if the
interface
is not the best reverse path the packet check will fail.
By default failed packets are discarded.
2 - Loose mode as defined in RFC3704 Loose Reverse Path
Each incoming packet's source address is also tested against
the FIB
and if the source address is not reachable via any interface
the packet check will fail.


Yes, I actually found it on RHEL solutions site as well:
https://access.redhat.com/site/solutions/53031

Cheers,
Alex
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users