Mailing List Archive

[lvs-users] LVS with Piranha in NAT Mode
Hello,
since one week I can't find out why will my LVS not work. It's a setup only
for testing:

The LVS setup after boot up:

[root@lvs1 ~]# ifconfig
eth0 Link encap:Ethernet Hardware Adresse 94:0C:6D:84:2B:3F
inet Adresse:192.168.130.231 Bcast:192.168.130.255
Maske:255.255.255.0
inet6 Adresse: fe80::960c:6dff:fe84:2b3f/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1791 errors:0 dropped:0 overruns:0 frame:0
TX packets:1346 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:171782 (167.7 KiB) TX bytes:225413 (220.1 KiB)

eth0:1 Link encap:Ethernet Hardware Adresse 94:0C:6D:84:2B:3F
inet Adresse:192.168.130.241 Bcast:192.168.130.255
Maske:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1 Link encap:Ethernet Hardware Adresse 00:11:6B:62:C3:C9
inet Adresse:192.168.13.254 Bcast:192.168.13.255
Maske:255.255.255.0
inet6 Adresse: fe80::211:6bff:fe62:c3c9/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2117 errors:0 dropped:0 overruns:0 frame:0
TX packets:2075 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:1213631 (1.1 MiB) TX bytes:138309 (135.0 KiB)

lo Link encap:Lokale Schleife
inet Adresse:127.0.0.1 Maske:255.0.0.0
inet6 Adresse: ::1/128 Gültigkeitsbereich:Maschine
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:9 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:0
RX bytes:540 (540.0 b) TX bytes:540 (540.0 b)

========================================================
SELINUX is disabled

========================================================

Firewall:
[root@lvs1 ~]# service iptables status
Tabelle: nat
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

Tabelle: mangle
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination

Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain FORWARD (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination

Tabelle: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain FORWARD (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination

===================================================
Config:
[root@lvs1 ~]# cat /etc/sysconfig/ha/lvs.cf
serial_no = 34
primary = 192.168.130.231
service = lvs
backup_active = 0
backup = 0.0.0.0
heartbeat = 1
heartbeat_port = 539
keepalive = 6
deadtime = 18
network = nat
nat_router = 192.168.13.254 eth1
nat_nmask = 255.255.255.0
debug_level = NONE
virtual http_intranet {
active = 1
address = 192.168.130.241 eth0:1
vip_nmask = 255.255.255.0
port = 80
send = "GET / HTTP/1.0\r\n\r\n"
expect = "HTTP"
use_regex = 0
load_monitor = none
scheduler = wlc
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 0
server v_182 {
address = 192.168.13.182
active = 1
weight = 100
}
}

=======================================================

LVS Routing Table:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.130.241:80 wlc
-> 192.168.13.182:80 Masq 100 0 0

LVS Processes:
root 1062 0.0 0.0 2408 580 ? Ss 17:16 0:00 pulse
root 1226 0.0 0.0 2400 812 ? Ss 17:16 0:00 /usr/sbin/lvsd --nofork -c
/etc/sysconfig/ha/lvs.cf
root 1230 0.0 0.0 2372 840 ? Ss 17:16 0:00 /usr/sbin/nanny -c -h
192.168.13.182 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x HTTP -a 15 -I
/sbin/ipvsadm -t 6 -w 100 -V 192.168.130.241 -M m -U none --lvs
root 1360 0.4 0.1 5056 1692 tty1 S+ 17:21 0:01 watch ipvsadm

========================================================
My tests from the LVS host:

[root@lvs1 ~]# telnet 192.168.13.182 80
Trying 192.168.13.182...
Connected to 192.168.13.182.
Escape character is '^]'.
GET / HTTP/1.0\r\n\r\n

HTTP/1.1 403 Forbidden
Date: Tue, 08 Jul 2014 13:35:01 GMT
Server: Apache/2.2.15 (CentOS)
Accept-Ranges: bytes
Content-Length: 5039
Connection: close
Content-Type: text/html
(.....) This is the CentOS Apache start page
=> the real server is working, tested from the LVS

But this is not working:

[root@lvs1 ~]# telnet 192.168.130.241 80
Trying 192.168.130.241...
telnet: connect to address 192.168.130.241: Connection timed out

The LVS seems not tranfering the traffic to the real server - but I don't
know why.
Can anybody help please?

best regards
Frank



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] LVS with Piranha in NAT Mode [ In reply to ]
On Tue, Jul 08, 2014 at 03:42:05PM +0200, Frank Kirschner wrote:
> Hello,
> since one week I can't find out why will my LVS not work. It's a setup only
> for testing:
>
> The LVS setup after boot up:
>
> [root@lvs1 ~]# ifconfig
> eth0 Link encap:Ethernet Hardware Adresse 94:0C:6D:84:2B:3F
> inet Adresse:192.168.130.231 Bcast:192.168.130.255
> Maske:255.255.255.0
> inet6 Adresse: fe80::960c:6dff:fe84:2b3f/64
> Gültigkeitsbereich:Verbindung
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:1791 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1346 errors:0 dropped:0 overruns:0 carrier:0
> Kollisionen:0 Sendewarteschlangenlänge:1000
> RX bytes:171782 (167.7 KiB) TX bytes:225413 (220.1 KiB)
>
> eth0:1 Link encap:Ethernet Hardware Adresse 94:0C:6D:84:2B:3F
> inet Adresse:192.168.130.241 Bcast:192.168.130.255
> Maske:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> eth1 Link encap:Ethernet Hardware Adresse 00:11:6B:62:C3:C9
> inet Adresse:192.168.13.254 Bcast:192.168.13.255
> Maske:255.255.255.0
> inet6 Adresse: fe80::211:6bff:fe62:c3c9/64
> Gültigkeitsbereich:Verbindung
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:2117 errors:0 dropped:0 overruns:0 frame:0
> TX packets:2075 errors:0 dropped:0 overruns:0 carrier:0
> Kollisionen:0 Sendewarteschlangenlänge:1000
> RX bytes:1213631 (1.1 MiB) TX bytes:138309 (135.0 KiB)
>
> lo Link encap:Lokale Schleife
> inet Adresse:127.0.0.1 Maske:255.0.0.0
> inet6 Adresse: ::1/128 Gültigkeitsbereich:Maschine
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:9 errors:0 dropped:0 overruns:0 frame:0
> TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
> Kollisionen:0 Sendewarteschlangenlänge:0
> RX bytes:540 (540.0 b) TX bytes:540 (540.0 b)
>
> ========================================================
> SELINUX is disabled
>
> ========================================================
>
> Firewall:
> [root@lvs1 ~]# service iptables status
> Tabelle: nat
> Chain PREROUTING (policy ACCEPT)
> num target prot opt source destination
>
> Chain POSTROUTING (policy ACCEPT)
> num target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> num target prot opt source destination
>
> Tabelle: mangle
> Chain PREROUTING (policy ACCEPT)
> num target prot opt source destination
>
> Chain INPUT (policy ACCEPT)
> num target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> num target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> num target prot opt source destination
>
> Chain POSTROUTING (policy ACCEPT)
> num target prot opt source destination
>
> Tabelle: filter
> Chain INPUT (policy ACCEPT)
> num target prot opt source destination
>
> Chain FORWARD (policy ACCEPT)
> num target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> num target prot opt source destination
>
> ===================================================
> Config:
> [root@lvs1 ~]# cat /etc/sysconfig/ha/lvs.cf
> serial_no = 34
> primary = 192.168.130.231
> service = lvs
> backup_active = 0
> backup = 0.0.0.0
> heartbeat = 1
> heartbeat_port = 539
> keepalive = 6
> deadtime = 18
> network = nat
> nat_router = 192.168.13.254 eth1
> nat_nmask = 255.255.255.0
> debug_level = NONE
> virtual http_intranet {
> active = 1
> address = 192.168.130.241 eth0:1
> vip_nmask = 255.255.255.0
> port = 80
> send = "GET / HTTP/1.0\r\n\r\n"
> expect = "HTTP"
> use_regex = 0
> load_monitor = none
> scheduler = wlc
> protocol = tcp
> timeout = 6
> reentry = 15
> quiesce_server = 0
> server v_182 {
> address = 192.168.13.182
> active = 1
> weight = 100
> }
> }
>
> =======================================================
>
> LVS Routing Table:
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.130.241:80 wlc
> -> 192.168.13.182:80 Masq 100 0 0
>
> LVS Processes:
> root 1062 0.0 0.0 2408 580 ? Ss 17:16 0:00 pulse
> root 1226 0.0 0.0 2400 812 ? Ss 17:16 0:00 /usr/sbin/lvsd --nofork -c
> /etc/sysconfig/ha/lvs.cf
> root 1230 0.0 0.0 2372 840 ? Ss 17:16 0:00 /usr/sbin/nanny -c -h
> 192.168.13.182 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x HTTP -a 15 -I
> /sbin/ipvsadm -t 6 -w 100 -V 192.168.130.241 -M m -U none --lvs
> root 1360 0.4 0.1 5056 1692 tty1 S+ 17:21 0:01 watch ipvsadm
>
> ========================================================
> My tests from the LVS host:
>
> [root@lvs1 ~]# telnet 192.168.13.182 80
> Trying 192.168.13.182...
> Connected to 192.168.13.182.
> Escape character is '^]'.
> GET / HTTP/1.0\r\n\r\n
>
> HTTP/1.1 403 Forbidden
> Date: Tue, 08 Jul 2014 13:35:01 GMT
> Server: Apache/2.2.15 (CentOS)
> Accept-Ranges: bytes
> Content-Length: 5039
> Connection: close
> Content-Type: text/html
> (.....) This is the CentOS Apache start page
> => the real server is working, tested from the LVS
>
> But this is not working:
>
> [root@lvs1 ~]# telnet 192.168.130.241 80
> Trying 192.168.130.241...
> telnet: connect to address 192.168.130.241: Connection timed out
>
> The LVS seems not tranfering the traffic to the real server - but I don't
> know why.
> Can anybody help please?

What is the default route on the real server? It should be your LVS
node.

Ryan


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] LVS with Piranha in NAT Mode [ In reply to ]
> From: Ryan O'Hara [mailto:rohara@redhat.com]
> Sent: Tuesday, July 08, 2014 4:23 PM
> To: frank@celebrate.de; LinuxVirtualServer.org users mailing list.
> Subject: Re: [lvs-users] LVS with Piranha in NAT Mode
>
> On Tue, Jul 08, 2014 at 03:42:05PM +0200, Frank Kirschner wrote:
> > Hello,
> > since one week I can't find out why will my LVS not work.
> It's a setup
> > only for testing:
> >
> > The LVS setup after boot up:
> >
> > [root@lvs1 ~]# ifconfig
> > eth0 Link encap:Ethernet Hardware Adresse 94:0C:6D:84:2B:3F
> > inet Adresse:192.168.130.231 Bcast:192.168.130.255
> > Maske:255.255.255.0
> > inet6 Adresse: fe80::960c:6dff:fe84:2b3f/64
> > Gültigkeitsbereich:Verbindung
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:1791 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:1346 errors:0 dropped:0 overruns:0 carrier:0
> > Kollisionen:0 Sendewarteschlangenlänge:1000
> > RX bytes:171782 (167.7 KiB) TX bytes:225413 (220.1 KiB)
> >
> > eth0:1 Link encap:Ethernet Hardware Adresse 94:0C:6D:84:2B:3F
> > inet Adresse:192.168.130.241 Bcast:192.168.130.255
> > Maske:255.255.255.0
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> >
> > eth1 Link encap:Ethernet Hardware Adresse 00:11:6B:62:C3:C9
> > inet Adresse:192.168.13.254 Bcast:192.168.13.255
> > Maske:255.255.255.0
> > inet6 Adresse: fe80::211:6bff:fe62:c3c9/64
> > Gültigkeitsbereich:Verbindung
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:2117 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:2075 errors:0 dropped:0 overruns:0 carrier:0
> > Kollisionen:0 Sendewarteschlangenlänge:1000
> > RX bytes:1213631 (1.1 MiB) TX bytes:138309 (135.0 KiB)
> >
> > lo Link encap:Lokale Schleife
> > inet Adresse:127.0.0.1 Maske:255.0.0.0
> > inet6 Adresse: ::1/128 Gültigkeitsbereich:Maschine
> > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > RX packets:9 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
> > Kollisionen:0 Sendewarteschlangenlänge:0
> > RX bytes:540 (540.0 b) TX bytes:540 (540.0 b)
> >
> > ========================================================
> > SELINUX is disabled
> >
> > ========================================================
> >
> > Firewall:
> > [root@lvs1 ~]# service iptables status
> > Tabelle: nat
> > Chain PREROUTING (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain POSTROUTING (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain OUTPUT (policy ACCEPT)
> > num target prot opt source destination
> >
> > Tabelle: mangle
> > Chain PREROUTING (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain INPUT (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain FORWARD (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain OUTPUT (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain POSTROUTING (policy ACCEPT)
> > num target prot opt source destination
> >
> > Tabelle: filter
> > Chain INPUT (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain FORWARD (policy ACCEPT)
> > num target prot opt source destination
> >
> > Chain OUTPUT (policy ACCEPT)
> > num target prot opt source destination
> >
> > ===================================================
> > Config:
> > [root@lvs1 ~]# cat /etc/sysconfig/ha/lvs.cf serial_no = 34
> primary =
> > 192.168.130.231 service = lvs backup_active = 0 backup = 0.0.0.0
> > heartbeat = 1 heartbeat_port = 539 keepalive = 6 deadtime =
> 18 network
> > = nat nat_router = 192.168.13.254 eth1 nat_nmask = 255.255.255.0
> > debug_level = NONE virtual http_intranet {
> > active = 1
> > address = 192.168.130.241 eth0:1
> > vip_nmask = 255.255.255.0
> > port = 80
> > send = "GET / HTTP/1.0\r\n\r\n"
> > expect = "HTTP"
> > use_regex = 0
> > load_monitor = none
> > scheduler = wlc
> > protocol = tcp
> > timeout = 6
> > reentry = 15
> > quiesce_server = 0
> > server v_182 {
> > address = 192.168.13.182
> > active = 1
> > weight = 100
> > }
> > }
> >
> > =======================================================
> >
> > LVS Routing Table:
> > IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port
> > Scheduler Flags
> > -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> > TCP 192.168.130.241:80 wlc
> > -> 192.168.13.182:80 Masq 100 0 0
> >
> > LVS Processes:
> > root 1062 0.0 0.0 2408 580 ? Ss 17:16 0:00 pulse root 1226 0.0 0.0
> > 2400 812 ? Ss 17:16 0:00 /usr/sbin/lvsd --nofork -c
> > /etc/sysconfig/ha/lvs.cf root 1230 0.0 0.0 2372 840 ? Ss 17:16 0:00
> > /usr/sbin/nanny -c -h
> > 192.168.13.182 -p 80 -r 80 -s GET / HTTP/1.0\r\n\r\n -x
> HTTP -a 15 -I
> > /sbin/ipvsadm -t 6 -w 100 -V 192.168.130.241 -M m -U none
> --lvs root
> > 1360 0.4 0.1 5056 1692 tty1 S+ 17:21 0:01 watch ipvsadm
> >
> > ========================================================
> > My tests from the LVS host:
> >
> > [root@lvs1 ~]# telnet 192.168.13.182 80 Trying 192.168.13.182...
> > Connected to 192.168.13.182.
> > Escape character is '^]'.
> > GET / HTTP/1.0\r\n\r\n
> >
> > HTTP/1.1 403 Forbidden
> > Date: Tue, 08 Jul 2014 13:35:01 GMT
> > Server: Apache/2.2.15 (CentOS)
> > Accept-Ranges: bytes
> > Content-Length: 5039
> > Connection: close
> > Content-Type: text/html
> > (.....) This is the CentOS Apache start page => the real server is
> > working, tested from the LVS
> >
> > But this is not working:
> >
> > [root@lvs1 ~]# telnet 192.168.130.241 80 Trying 192.168.130.241...
> > telnet: connect to address 192.168.130.241: Connection timed out
> >
> > The LVS seems not tranfering the traffic to the real server - but I
> > don't know why.
> > Can anybody help please?
>
> What is the default route on the real server? It should be
> your LVS node.
>
Oh yes, this is the right direction. Now it will be difficult:
The real server is a virtual container of OpenVZ on a RedHat hardware node.
Routing table of the hardware node:

192.168.13.182 0.0.0.0 255.255.255.255 UH 0 0 0
venet0
192.168.130.182 0.0.0.0 255.255.255.255 UH 0 0 0
venet0
192.168.130.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.13.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.110.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 eth1
0.0.0.0 192.168.130.254 0.0.0.0 UG 0 0 0 eth0

Explanation of the used subnets on the hardware node:
192.168.130.0/24 with 192.168.130.254 as gateway to ISP => local LAN /
intranet
192.168.110.0/24 => the SAN where three GlusterFS nodes are providing
the document root for apache
192.168.13.0/24 => the subnet for testing LVS

192.168.13.254 is the nat_router ip of the LVS. If I would change the
default gateway of the hardware node,
I will loose the NAT (port forwarding) of the 192.168.130.254 ISP gateway to
the other containers inside the hardware node.

What's the routing solution for this?
Thanks for your help.

Frank





_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] LVS with Piranha in NAT Mode [ In reply to ]
2014-07-09 7:17 GMT+02:00 Frank Kirschner <frank@celebrate.de>:
> 192.168.13.254 is the nat_router ip of the LVS. If I would change the
> default gateway of the hardware node,
> I will loose the NAT (port forwarding) of the 192.168.130.254 ISP gateway to
> the other containers inside the hardware node.
>
> What's the routing solution for this?

Policy routing (ip rule) on the real server, keyed on local source
192.168.13.0/24, e.g.
ip route add 192.168.13.0/24 table 1234
ip route add default via 192.168.13.254 table 1234
ip rule add from 192.168.13.0/24 table 1234

If you can make it so, best also separate the LVS traffic in a
different VLAN. You still need that policy routing, but the resulting
setup is simpler to debug and analyze and has some operational
advantages that I won't go into now :)

best regards
Patrick

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users