Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Linux Virtual Server>
  3. Users
[lvs-users] Wrong RIP is used for fwmark-based groups pointing to different ports and protocols of the same RIP
brudas at exadel
Dec 11, 2018, 10:37 AM
Post #1 of 2 (398 views)
Permalink
Hello!

I try to do load-balance the service which is listening both TCP and UDP
using same port number *1111*. There are set of real servers, each of them
host few application instances listening on different ports (1111/udp,
1116/udp, 1115/tcp, 1117/tcp)
I used fwmark to group connection by protocol, i.e. 1111/udp have *fwmark 1*,
1111/tcp have *fwmark 2.*
LVS-NAT mode is in use.
What I see is when the client connection coming to 1111/*TCP* the LVS try
to pass it to RTI:1111/*TCP *while the RIP have TCP listeners on ports 1115
and 1117 and LVS was configured accordingly.

Here is test connection info:
# ipvsadm -L -n -c
IPVS connection entries
pro expire state source virtual destination
*TCP* 00:55 SYN_RECV 113.25.79.34:38204 115.5.6.138*:1111 *10.1.2.13
*:1111* *<<<< It should be RIP:1115 or RIP:1117 HERE!*
IP 00:49 NONE 113.25.79.34:0 0.0.0.2:0 10.1.2.13:0
TCP 01:00 SYN_RECV 113.25.79.34:38206 115.5.6.138:1111 10.1.2.13:1111

Below are saved firewall and ipvs configurations:

# Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
*mangle
:PREROUTING ACCEPT [9507:869411]
:INPUT ACCEPT [7637:700414]
:FORWARD ACCEPT [70:10945]
:OUTPUT ACCEPT [8781:1045717]
:POSTROUTING ACCEPT [8851:1056662]
COMMIT
# Completed on Tue Dec 11 17:02:29 2018
# Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [82:11111]
:OUTPUT ACCEPT [16923:2797363]
-A INPUT -p icmp -m comment --comment "000 accept all icmp" -j ACCEPT
-A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j
ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "002 reject local
traffic not on loopback interface" -j REJECT --reject-with
icmp-port-unreachable
-A INPUT -m comment --comment "003 accept related established rules" -m
state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "100 accept
ssh connection" -m state --state NEW -j ACCEPT
-A INPUT -d 224.0.0.0/8 -p vrrp -m comment --comment "104 Accept VRRP
proto" -j ACCEPT
-A INPUT -m comment --comment "999 drop all" -j REJECT --reject-with
icmp-port-unreachable
COMMIT
# Completed on Tue Dec 11 17:02:29 2018
# Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
*nat
:PREROUTING ACCEPT [42:2928]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [4:360]
:POSTROUTING ACCEPT [4:360]
-A PREROUTING -p tcp -m multiport --dports 1111 -j MARK --set-xmark
0x2/0xffffffff
-A PREROUTING -p *udp* -m multiport --dports 1111 -m comment --comment "101
MARK udp 1111 traffic from everywhere" -j MARK --set-xmark *0x1*/0xffffffff
-A PREROUTING -p *tcp* -m multiport --dports 1111 -m comment --comment "101
MARK tcp 1111 traffic from everywhere" -j MARK --set-xmark *0x2*/0xffffffff
COMMIT
# Completed on Tue Dec 11 17:02:29 2018
# Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
*raw
:PREROUTING ACCEPT [17569:3914462]
:OUTPUT ACCEPT [16925:2798776]
COMMIT

# ipvsadm -S
-A -f 1 -s wrr -p 50
-a -f 1 -r 10.1.2.12:1111 -m -w 2
-a -f 1 -r 10.1.2.12:1116 -m -w 2
-a -f 1 -r 10.1.2.13:1111 -m -w 2
-a -f 1 -r 10.1.2.13:1116 -m -w 2
-a -f 1 -r 10.1.2.14:1111 -m -w 2
-a -f 1 -r 10.1.2.14:1116 -m -w 2
-A -f 2 -s wrr -p 50
-a -f 2 -r 10.1.2.12:1115 -m -w 2
-a -f 2 -r 10.1.2.12:1117 -m -w 2
-a -f 2 -r 10.1.2.13:1115 -m -w 2
-a -f 2 -r 10.1.2.13:1117 -m -w 2
-a -f 2 -r 10.1.2.14:1115 -m -w 2
-a -f 2 -r 10.1.2.14:1117 -m -w 2


# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 wrr persistent 50
-> 10.1.2.12:1111 Masq 2 0 0
-> 10.1.2.12:1116 Masq 2 0 0
-> 10.1.2.13:1111 Masq 2 0 0
-> 10.1.2.13:1116 Masq 2 0 0
-> 10.1.2.14:1111 Masq 2 0 0
-> 10.1.2.14:1116 Masq 2 0 0
FWM 2 wrr persistent 50
-> 10.1.2.12:1115 Masq 2 0 0
-> 10.1.2.12:1117 Masq 2 0 0
-> 10.1.2.13:1115 Masq 2 0 0
-> 10.1.2.13:1117 Masq 2 0 1
-> 10.1.2.14:1115 Masq 2 0 0
-> 10.1.2.14:1117 Masq 2 0 0


OS is CentOS 7 with kernel *3.10*

Thank you.



--
Bogdan Rudas
Director of IT Europe
Exadel Inc.
http://www.exadel.com/
E-mail: brudas@exadel.com
Skype ID: bogdan.rudas

--


CONFIDENTIALITY
NOTICE: This email and files attached to it are
confidential. If you
are not the intended recipient you are hereby notified
that using,
copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. If you have
received
this email in error please notify the sender and delete this
email.
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Wrong RIP is used for fwmark-based groups pointing to different ports and protocols of the same RIP [ In reply to ]
ja at ssi
Jan 5, 2019, 1:11 PM
Post #2 of 2 (392 views)
Permalink
Hello,

On Tue, 11 Dec 2018, Bogdan Rudas wrote:

> I try to do load-balance the service which is listening both TCP and UDP
> using same port number *1111*. There are set of real servers, each of them
> host few application instances listening on different ports (1111/udp,
> 1116/udp, 1115/tcp, 1117/tcp)
> I used fwmark to group connection by protocol, i.e. 1111/udp have *fwmark 1*,
> 1111/tcp have *fwmark 2.*
> LVS-NAT mode is in use.
> What I see is when the client connection coming to 1111/*TCP* the LVS try
> to pass it to RTI:1111/*TCP *while the RIP have TCP listeners on ports 1115
> and 1117 and LVS was configured accordingly.

This is a long-standing behaviour to ignore rport for real servers
added to persistent fwmark-based virtual service. What can be changed is
to depend on rport being 0 or not:

# Map services 1:1 (port is preserved)
-r RIP:0 => translate only daddr=VIP->RIP
# Map all vports to same rport
-r RIP:RPORT => translate both address and port

But we risk to change behaviour to existing installations.

> Here is test connection info:
> # ipvsadm -L -n -c
> IPVS connection entries
> pro expire state source virtual destination
> *TCP* 00:55 SYN_RECV 113.25.79.34:38204 115.5.6.138*:1111 *10.1.2.13
> *:1111* *<<<< It should be RIP:1115 or RIP:1117 HERE!*
> IP 00:49 NONE 113.25.79.34:0 0.0.0.2:0 10.1.2.13:0
> TCP 01:00 SYN_RECV 113.25.79.34:38206 115.5.6.138:1111 10.1.2.13:1111
>
> Below are saved firewall and ipvs configurations:
>
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *mangle
> :PREROUTING ACCEPT [9507:869411]
> :INPUT ACCEPT [7637:700414]
> :FORWARD ACCEPT [70:10945]
> :OUTPUT ACCEPT [8781:1045717]
> :POSTROUTING ACCEPT [8851:1056662]
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *filter
> :INPUT ACCEPT [0:0]
> :FORWARD ACCEPT [82:11111]
> :OUTPUT ACCEPT [16923:2797363]
> -A INPUT -p icmp -m comment --comment "000 accept all icmp" -j ACCEPT
> -A INPUT -i lo -m comment --comment "001 accept all to lo interface" -j
> ACCEPT
> -A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "002 reject local
> traffic not on loopback interface" -j REJECT --reject-with
> icmp-port-unreachable
> -A INPUT -m comment --comment "003 accept related established rules" -m
> state --state RELATED,ESTABLISHED -j ACCEPT
> -A INPUT -p tcp -m multiport --dports 22 -m comment --comment "100 accept
> ssh connection" -m state --state NEW -j ACCEPT
> -A INPUT -d 224.0.0.0/8 -p vrrp -m comment --comment "104 Accept VRRP
> proto" -j ACCEPT
> -A INPUT -m comment --comment "999 drop all" -j REJECT --reject-with
> icmp-port-unreachable
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *nat
> :PREROUTING ACCEPT [42:2928]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [4:360]
> :POSTROUTING ACCEPT [4:360]
> -A PREROUTING -p tcp -m multiport --dports 1111 -j MARK --set-xmark
> 0x2/0xffffffff
> -A PREROUTING -p *udp* -m multiport --dports 1111 -m comment --comment "101
> MARK udp 1111 traffic from everywhere" -j MARK --set-xmark *0x1*/0xffffffff
> -A PREROUTING -p *tcp* -m multiport --dports 1111 -m comment --comment "101
> MARK tcp 1111 traffic from everywhere" -j MARK --set-xmark *0x2*/0xffffffff
> COMMIT
> # Completed on Tue Dec 11 17:02:29 2018
> # Generated by iptables-save v1.4.21 on Tue Dec 11 17:02:29 2018
> *raw
> :PREROUTING ACCEPT [17569:3914462]
> :OUTPUT ACCEPT [16925:2798776]
> COMMIT
>
> # ipvsadm -S
> -A -f 1 -s wrr -p 50
> -a -f 1 -r 10.1.2.12:1111 -m -w 2
> -a -f 1 -r 10.1.2.12:1116 -m -w 2

This is not supported: many real services on same real IP
when persistence is used for fwmark. Because: packets to different vport
can be marked with same fwmark to be handled by same virtual service.
As result, the fwmark template <IPPROTO_IP,caddr,0,fwmark,0,daddr,0> does
not use dport from the packet and we can not distinguish the real services
that differ just by rport.

In short, virtual service that can handle traffic to different
vports can not use persistence together with many real services on same
IP.

> -a -f 1 -r 10.1.2.13:1111 -m -w 2
> -a -f 1 -r 10.1.2.13:1116 -m -w 2
> -a -f 1 -r 10.1.2.14:1111 -m -w 2
> -a -f 1 -r 10.1.2.14:1116 -m -w 2
> -A -f 2 -s wrr -p 50
> -a -f 2 -r 10.1.2.12:1115 -m -w 2
> -a -f 2 -r 10.1.2.12:1117 -m -w 2
> -a -f 2 -r 10.1.2.13:1115 -m -w 2
> -a -f 2 -r 10.1.2.13:1117 -m -w 2
> -a -f 2 -r 10.1.2.14:1115 -m -w 2
> -a -f 2 -r 10.1.2.14:1117 -m -w 2
>
>
> # ipvsadm -L -n
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> FWM 1 wrr persistent 50
> -> 10.1.2.12:1111 Masq 2 0 0
> -> 10.1.2.12:1116 Masq 2 0 0
> -> 10.1.2.13:1111 Masq 2 0 0
> -> 10.1.2.13:1116 Masq 2 0 0
> -> 10.1.2.14:1111 Masq 2 0 0
> -> 10.1.2.14:1116 Masq 2 0 0
> FWM 2 wrr persistent 50
> -> 10.1.2.12:1115 Masq 2 0 0
> -> 10.1.2.12:1117 Masq 2 0 0
> -> 10.1.2.13:1115 Masq 2 0 0
> -> 10.1.2.13:1117 Masq 2 0 1
> -> 10.1.2.14:1115 Masq 2 0 0
> -> 10.1.2.14:1117 Masq 2 0 0
>
>
> OS is CentOS 7 with kernel *3.10*
>
> Thank you.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal