Mailing List Archive

[lvs-users] so-called "one arm NAT" LVS-NAT and interaction with netfilter/iptables
LVS users,

I have a "one arm NAT" configuration with so-called "local node" that I'm trying to make work with LVS.

I have two servers, syslog servers (call them A and B), and one client (we'll call C). Both A and B also run LVS with keepalived. There is a VIP address managed by keepalived. The idea is that when client C contacts the VIP, it will hit either A or B based on LVS scheduling. I'm doing both UDP and TCP protocols. For purposes of this discussion with respect to packets and the IP addresses, I'll use the format (source IP, destination IP).

When client contacts the VIP, a packet is sent with (C-IP, VIP). It arrives at A that holds the VIP at the moment, and proceeds to LOCAL_IN and LVS-NAT changes the packet to either (C-IP, A-IP) or (C-IP, B-IP). In the former case (C-IP, A-IP) case, it gets processed by the local syslog process. In the later case (C-IP, B-IP), it gets sent back out to B. However, because the packet is (C-IP, B-IP), B's replies go to (B-IP, C-IP) and doesn't get processed by the LVS-NAT anymore. Connection hangs... as the C is expecting reply from (VIP, C-IP).

I know in reading the various FAQ and postings about "one arm NAT" LVS-NAT situations is to have the "real servers" be the default GW. This won't work for me since the real servers are also the LVS servers. However, I was thinking that if I can use netfilter/iptables to source NAT the load balanced packets, I can change the (C-IP,B-IP) show up as (A-IP, B-IP) instead, and hence B's replies would go back to A (the current LVS-NAT) and sent back to C as (VIP, C-IP). However, the packet doesn't seem to traverse the "netfliter stack" as expected:

(i setup a -j LOG rule within every chain [PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING] of every table [mangle, nat, filter], and watch the packet traverse the "netfilter stack")


Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-nat-PRER: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-INPU: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-INPU: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-OUTP: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-OUTP: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-POSR: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16

This is a packet from (C-IP, VIP), goes through NAT/PREROUTING->ROUTING->MANGLE/INPUT->FILTER/INPUT->(LVS-NAT?)->ROUTING->MANGLE/OUTPUT->FILTER/OUTPUT->MANGLE/POSTROUTING.

What I see missing are MANGLE/PREROUTING and particularly interested in the missing NAT/POSTROUTING, where I was hoping to SNAT the packet.

According to this post:

http://archive.linuxvirtualserver.org/html/lvs-users/2011-03/msg00019.html

It would appear that at some point, LVS-NAT packets did traverse NAT/POSTROUTING and one could SNAT the LVS outbound packets?

I'm using:

RHEL 6.5 with RH kernel 2.6.32-431.17.1.el6.x86_64
ipvsadm-1.26-2.el6.x86_64
keepalived-1.2.13-1.el6.x86_64

here are the -j LOG iptables rules I'm using:

[root@sprdrsyas400 ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target prot opt in out source destination
4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-PRER: '

Chain INPUT (policy ACCEPT 293K packets, 29M bytes)
pkts bytes target prot opt in out source destination
4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-INPU: '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-FORW: '

Chain OUTPUT (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target prot opt in out source destination
2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-OUTP: '

Chain POSTROUTING (policy ACCEPT 362K packets, 18M bytes)
pkts bytes target prot opt in out source destination
2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-POSR: '
[root@sprdrsyas400 ~]# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 8739 packets, 500K bytes)
pkts bytes target prot opt in out source destination
7 252 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-PRER: '

Chain POSTROUTING (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-POSR: '

Chain OUTPUT (policy ACCEPT 25612 packets, 1374K bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-OUTP: '
[root@sprdrsyas400 ~]# iptables -vnL
Chain INPUT (policy ACCEPT 292K packets, 29M bytes)
pkts bytes target prot opt in out source destination
4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-INPU: '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-FORW: '

Chain OUTPUT (policy ACCEPT 361K packets, 18M bytes)
pkts bytes target prot opt in out source destination
2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-OUTP: '


And this is LVS on A:

[root@sprdrsyas400 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
UDP 10.180.88.35:514 rr ops
-> 10.180.88.29:514 Local 1 0 0
-> 10.180.88.30:514 Masq 1 0 0

I would appreciate any help...

Thanks in advance,
-Bond

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] so-called "one arm NAT" LVS-NAT and interaction with netfilter/iptables [ In reply to ]
Hi,

I use also ipvs in NAT mode with some SNAT rules to force traffic go
back to client through my ipvs gateway.
Note sure, but I guess that in your case, this should work with
localhost service.
In all cases, you should use IPVS specific iptables rules like that :

iptables -t nat -A IPVS_SNAT -m ipvs -d 10.180.88.29/32 --vaddr
10.180.88.35 --vport 514 -j SNAT --to-source 10.180.88.35 -m comment
--comment "SNAT ipvs"

I hope it will help you to find a solution.
Best regards.
--
Ivan Havlicek

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users