Mailing List Archive

Stephen,

Make sure that your real servers (backend servers) have the default gateway
set as the load balancer.
Do a quick couple of tests from a client on the external network (not the
internal one).
Then quickly do an :
ipvsadm -Lnc
To see if LVS is processing the connections.
Check you don't have anything firewalled etc.






On 18 June 2014 21:39, Stephen Carville <c1d62208@opayq.com> wrote:

>
> I set up a CentOS 6.5 box to test ipvsadm. So far I have been unable to
> get it to forward connections. When I try to connect, it doesn't write
> anything in /var/log/messages to tell me what is happening. Netstat
> doesn't see anything listening on the interface IP (I read elsewhere
> that is normal) and tshark sees the incoming SYN but there is either a
> timeout or a RST.
>
> Rules right now:
>
> $ ipvsadm -L
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 10.212.160.40:4172 lc persistent 360
> -> 10.212.170.162:4172 Route 1 0 0
> -> 10.212.170.163:4172 Route 1 0 0
>
> IP forwarding is turned on:
>
> $ sysctl net.ipv4.ip_forward
> net.ipv4.ip_forward = 1
>
> Addresses all seem to be there.
>
> $ ip addr list eth0
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 00:50:56:84:64:8d brd ff:ff:ff:ff:ff:ff
> inet 10.212.160.40/24 brd 10.212.160.255 scope global eth0
> inet 10.212.160.43/32 brd 10.212.160.43 scope global eth0
> inet6 fe80::250:56ff:fe84:648d/64 scope link
> valid_lft forever preferred_lft forever
>
> $ ip addr list eth1
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 00:50:56:84:4c:4a brd ff:ff:ff:ff:ff:ff
> inet 10.212.170.13/24 brd 10.212.170.255 scope global eth1
> inet6 fe80::250:56ff:fe84:4c4a/64 scope link
> valid_lft forever preferred_lft forever
>
> I obviously have done something wrong but damned if I can figure it out.
>
> --
> Stephen
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Firewall was the first thing I thought of :)

$ iptables-save

*nat
:PREROUTING ACCEPT [42817:7374591]
:POSTROUTING ACCEPT [1290:88312]
:OUTPUT ACCEPT [704:51857]
COMMIT
*filter
:INPUT ACCEPT [141174:20581389]
:FORWARD ACCEPT [72032:53381828]
:OUTPUT ACCEPT [27693:3275147]
COMMIT

Anyways, I set the default route on the backend servers to point back to
the lvs interface. I also had to change the forwarding to masquerade.
It seems to work now.

Thank you.

On 06/19/2014 05:11 AM, Malcolm Turnbull [Masked] wrote:
>
> Stephen,
>
> Make sure that your real servers (backend servers) have the default gateway
> set as the load balancer.
> Do a quick couple of tests from a client on the external network (not the
> internal one).
> Then quickly do an :
> ipvsadm -Lnc
> To see if LVS is processing the connections.
> Check you don't have anything firewalled etc.

--
Stephen


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On 18.06.2014, Stephen Carville wrote:
> I set up a CentOS 6.5 box to test ipvsadm. So far I have been unable to
> get it to forward connections. When I try to connect, it doesn't write
> anything in /var/log/messages to tell me what is happening. Netstat
> doesn't see anything listening on the interface IP (I read elsewhere
> that is normal) and tshark sees the incoming SYN but there is either a
> timeout or a RST.
>
> Rules right now:
>
> $ ipvsadm -L
>
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 10.212.160.40:4172 lc persistent 360
> -> 10.212.170.162:4172 Route 1 0 0
> -> 10.212.170.163:4172 Route 1 0 0
>
> IP forwarding is turned on:
>
> $ sysctl net.ipv4.ip_forward
> net.ipv4.ip_forward = 1

Short answer: switch to kernel 3.6 or newer, turn off rp_filter for the interface receiving the reply packet, and replace rp_filter functionality by more accurate and flexible iptables rules in the FORWARD chain.

Long:

When you're using direct routing/gatewaying or tunneling, your replies will have the VIP address as the source address of the packet. They're routed via your balancer, who also knows the same IP address to be a local address.

Depending on /proc/sys/net/ipv4/conf/*/rp_filter or kernel version, you need to either change a little bit of configuration or apply special kernel patches (forward_shared).

If you're running a linux kernel <3.6, your kernel will always drop those packets as being spoofed. If you're turning on net/ipv4/conf/*/log_martians, you'll also see kernel log messages on this.

>From the stone age of IPVS, there is a kernel patch introducing a sysctl switch (forward_shared) to turn off this behaviour for selected interfaces. If misconfigured (e.g. "turn it on for all interfaces"), this patch will introduce an option for third parties to spoof packets onto your network (which is something you don't want to do).

http://www.ssi.bg/~ja/ does offer recent patches for about any major linux kernel source from 2.4.19 to 3.15, just in case you're interested.



Linux Kernel 3.6 did somehow obsolete the forward_shared-patch by changing the behaviour of rp_filter for the interface receiving the spoofed packet:
-if rp_filter is turned on and ip_forward=1, the kernel will most likely drop your reply packets, as they're most likely received by the 'wrong' interface: an interface without a matching entry in the routing table for this IP address. Given the point many linux distros do enable rp_filter per default, you most likely hit that wall.
-if rp_filter is turned off (=0) and ip_forward=1, the kernel will gladly forward your "spoofed" reply packets.

The exact use of rp_filter on a dedicated load balancer is fairly limited, if at all. It's much better to replace its functionality by dedicated iptables rules and restricting the FORWARD chain.

For your configuration, this results in the following configuration changes:

$ iptables -P FORWARD DROP
$ iptables -A FORWARD -i eth1 -o eth0 --source 10.212.160.40 -j ACCEPT
$ sysctl -w proc/sys/net/ipv4/conf/eth1/rp_filter=0

If you're keen on extra-tight security, you may also copy the permissive line and add "-m --mac-source $realserver-mac" for every realserver. This ensures only your real servers may send "spoofed" traffic via your balancer.



Best,

Anders
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On 23.06.2014 11:57, Anders Henke wrote:
> On 18.06.2014, Stephen Carville wrote:
>> I set up a CentOS 6.5 box to test ipvsadm. So far I have been unable to
>> get it to forward connections. When I try to connect, it doesn't write
>> anything in /var/log/messages to tell me what is happening. Netstat
>> doesn't see anything listening on the interface IP (I read elsewhere
>> that is normal) and tshark sees the incoming SYN but there is either a
>> timeout or a RST.
>>
>> Rules right now:
>>
>> $ ipvsadm -L
>>
>> IP Virtual Server version 1.2.1 (size=4096)
>> Prot LocalAddress:Port Scheduler Flags
>> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>> TCP 10.212.160.40:4172 lc persistent 360
>> -> 10.212.170.162:4172 Route 1 0 0
>> -> 10.212.170.163:4172 Route 1 0 0
>>
>> IP forwarding is turned on:
>>
>> $ sysctl net.ipv4.ip_forward
>> net.ipv4.ip_forward = 1
>
> Short answer: switch to kernel 3.6 or newer, turn off rp_filter for the interface receiving the reply packet, and replace rp_filter functionality by more accurate and flexible iptables rules in the FORWARD chain.

Since he is running CentOS 6.5 he can simply set
/proc/sys/net/ipv4/conf/<interface>/accept_local to 1 to prevent packets
from being dropped as martians. This was introduced in 2.6.33 but
backported to recent RHEL/CentOS kernels so no need to go to 3.6 or newer.
You still have to set the rp_filter though since this is a different
isssue than the martian packet one.

Regards,
Dennis


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

On 23.06.2014, Dennis Jacobfeuerborn wrote:
> On 23.06.2014 11:57, Anders Henke wrote:
> > On 18.06.2014, Stephen Carville wrote:
> >> I set up a CentOS 6.5 box to test ipvsadm. So far I have been unable to
> >> get it to forward connections. When I try to connect, it doesn't write
> >> anything in /var/log/messages to tell me what is happening. Netstat
> >> doesn't see anything listening on the interface IP (I read elsewhere
> >> that is normal) and tshark sees the incoming SYN but there is either a
> >> timeout or a RST.
> >>
> >> Rules right now:
> >>
> >> $ ipvsadm -L
> >>
> >> IP Virtual Server version 1.2.1 (size=4096)
> >> Prot LocalAddress:Port Scheduler Flags
> >> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> >> TCP 10.212.160.40:4172 lc persistent 360
> >> -> 10.212.170.162:4172 Route 1 0 0
> >> -> 10.212.170.163:4172 Route 1 0 0
> >>
> >> IP forwarding is turned on:
> >>
> >> $ sysctl net.ipv4.ip_forward
> >> net.ipv4.ip_forward = 1
> >
> > Short answer: switch to kernel 3.6 or newer, turn off rp_filter for the interface receiving the reply packet, and replace rp_filter functionality by more accurate and flexible iptables rules in the FORWARD chain.
>
> Since he is running CentOS 6.5 he can simply set
> /proc/sys/net/ipv4/conf/<interface>/accept_local to 1 to prevent packets
> from being dropped as martians. This was introduced in 2.6.33 but
> backported to recent RHEL/CentOS kernels so no need to go to 3.6 or newer.
> You still have to set the rp_filter though since this is a different
> isssue than the martian packet one.

It's a little bit more complicated :-)

In this setup, the VIP is configured on eth0, but the realserver is contacted via eth1. The realserver will most likely to reply via eth1 as well.

Upstream network equipment will route incoming traffic for the VIP to eth0, and by seperating the realservers onto a dedicated network, we're removing the need for handling the ARP problem with realserver configuration. As long as no realservers are connected to eth0, the realservers don't need to disable ARP replies at all.

Regarding kernel configuration, accept_local is only revelant if rp_filter is set.

-if eth1 has both rp_filter=1 and accept_local=1, the kernel
checks if the IP address is routed and local to eth1, otherwise it is being
dropped. The VIP is configured on eth0, so the packet is being discarded.
-If eth1 has rp_filter=2 (loose mode) and accept_local=1, the packet
is accepted.

Loose mode doesn't check if the incoming interface matches the routing
decision, just if "any" route on the host, including a default route,
does know where to send this packet may be sent to.

So whenever your system does have a default route configured,
rp_filter=2 for any interface other than your default interface
is almost equivalent to rp_filter=0.
The only difference is the handling of addresses local to the balancer,
and by configuring accept_local=1 for that interface, one is also removing
that single difference.
In the end, you don't have any kind of reverse path filtering for
that specific interface, but you've just added some check to
your kernel which will always return true, but merely "looks" like
some added level of security.

That's part of the reason why in such a situation, I'm advocating to
de-configure rp_filter for that specific interface: you don't gain
security, you're just adding a useless check. If you do want to
improve security, you need to add the expected or aimed level of
security by dedicated iptables forwarding rules.

=== Alternative option

Another option is to configure the VIP to be hosted on eth1,
configure all interfaces with rp_filter=1 (default on CentOS 6.5)
and additionally configure eth1 with accept_local=1:

$ ip address del 10.212.160.40/32 dev eth0
$ ip address add 10.212.160.40/32 dev eth1
$ sysctl -w net.ipv4.conf.all.rp_filter = 1
$ sysctl -w net.ipv4.conf.eth1.accept_local = 1

Due to Linux' weak host model, packets for 10.212.160.40 will be accepted
on any interface and sent to the realserver. The realserver replies are
received at an interface hosting that IP address, so accept_local will match
and forward replies as intended.

Downside: most network engineers looking at this configuration instinctively
will assume a misconfiguration. It's not that intuitive, as it's relying on
the weak host model. However, it's consistent with most realserver
configurations and does have the benefits of strict mode reverse path
filtering.

As there are no forwarding restrictions per default other than the just
configured reverse path filtering and we're running on ip_forward=1, one
should still take care of the exact forwarding configuration, e.g. by
adding iptables rules to the forwarding chain. Otherwise, a malicious host
on eth0 may access any host running behind eth1 just by adding the
balancer as a router.

Specific iptables forwarding rules or selectively enabling forwarding only for eth1 does remove that risk. However, the later one is very specific to the exact realserver network configuration (you need to enable forwarding on eth0 to forward traffic from eth0 to eth1, if your balancer acts as the only router for your realservers).

So if one is adding fine-granular forwarding rules for a specific interface anyway, which are tighter than rp_filter for that interface - why care about rp_filter for eth1?



Best,

Anders
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users