Mailing List Archive

[lvs-users] IPVS uses loopback address for tunneling outer source IP
Hello,

We are seeing some very strange thing on IPVS tunnel mode(we are
running centos 7 with kernel 3.10.0):

Occasionally, the tcpdump shows that IPVS changes tunnelled outer
source IP address to a loopback address, like 127.146.255.255, and the
tunnelled packets had been dropped . However, most of time, the outer
source IP address is the normal IP address bind to the physical
ethernet interface. When this happened, we dumped the routing tables
and iptables, and it looks this issue is not due to either routing
setting or a iptables' SNAT. Since this issue happened on two hosts,
we also doubt that this is a hardware problem. We also briefly checked
the ip_vs_tunnel_xmit() call and it looks it retrieves outer header
information from kernel's routing part. We are wondering whether this
is a desired behaviour of IPVS, due to IPVS doesn't want to send out
packets at that time?

Thanks,

Yunzhao

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] IPVS uses loopback address for tunneling outer source IP [ In reply to ]
Hello,

On Wed, 29 Aug 2018, Yunzhao li wrote:

> We are seeing some very strange thing on IPVS tunnel mode(we are
> running centos 7 with kernel 3.10.0):
>
> Occasionally, the tcpdump shows that IPVS changes tunnelled outer
> source IP address to a loopback address, like 127.146.255.255, and the
> tunnelled packets had been dropped . However, most of time, the outer
> source IP address is the normal IP address bind to the physical
> ethernet interface. When this happened, we dumped the routing tables
> and iptables, and it looks this issue is not due to either routing
> setting or a iptables' SNAT. Since this issue happened on two hosts,
> we also doubt that this is a hardware problem. We also briefly checked
> the ip_vs_tunnel_xmit() call and it looks it retrieves outer header
> information from kernel's routing part. We are wondering whether this
> is a desired behaviour of IPVS, due to IPVS doesn't want to send out
> packets at that time?

Looks like a bug already fixed in Linux 4.2, 3.10.91, 3.18.23:

commit 4754957f04f5f368792a0eb7dab0ae89fb93dcfd
Author: Julian Anastasov <ja@ssi.bg>
Date: Sat Jun 27 14:39:30 2015 +0300

ipvs: do not use random local source address for tunnels

Michael Vallaly reports about wrong source address used
in rare cases for tunneled traffic. Looks like
__ip_vs_get_out_rt in 3.10+ is providing uninitialized
dest_dst->dst_saddr.ip because ip_vs_dest_dst_alloc uses
kmalloc. While we retry after seeing EINVAL from routing
for data that does not look like valid local address, it
still succeeded when this memory was previously used from
other dests and with different local addresses. As result,
we can use valid local address that is not suitable for
our real server.

Fix it by providing 0.0.0.0 every time our cache is refreshed.
By this way we will get preferred source address from routing.

Reported-by: Michael Vallaly <lvs@nolatency.com>
Fixes: 026ace060dfe ("ipvs: optimize dst usage for real server")
Signed-off-by: Julian Anastasov <ja@ssi.bg>
Signed-off-by: Simon Horman <horms@verge.net.au>

diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index b99d80695b1f..ec30d68ccc0b 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -130,7 +130,6 @@ static struct rtable *do_output_route4(struct net *net, __be32 daddr,

memset(&fl4, 0, sizeof(fl4));
fl4.daddr = daddr;
- fl4.saddr = (rt_mode & IP_VS_RT_MODE_CONNECT) ? *saddr : 0;
fl4.flowi4_flags = (rt_mode & IP_VS_RT_MODE_KNOWN_NH) ?
FLOWI_FLAG_KNOWN_NH : 0;

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users