Mailing List Archive: LSV with FreeBSD

LSV with FreeBSD

Jan 17, 2001, 1:29 AM

Post #1 of 3 (969 views)

Hello,

I am setting up a LVS/DR cluster with 2 nodes(FreeBSD), but It doesn't work.

Here is my network configuration;

Internet (203.231.63.70 is Virtual IP)
|
|
Router (203.231.63.0/24 network)
|
| <----- eth0 : 203.231.63.74
LVS(Linux)
| <----- eth1 : 203.231.63.70 (VIP)
|
------------------
| |
SVR1 SVR2 <-- Real Servers are FreeBSD 4.2-RELEASE

fxp0 : 203.231.63.70 203.231.63.70 (VIP)
fxp1 : 203.231.63.71 203.231.63.72 (Real IP)

----] in Load Valancing Server(203.231.63.74);

[root@ha1 log]# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:10:5A:80:D7:FF
inet addr:203.231.63.74 Bcast:203.231.63.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth1 Link encap:Ethernet HWaddr 00:10:5A:76:02:49
inet addr:203.231.63.70 Bcast:203.231.63.70 Mask:255.255.255.255
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:3924 Metric:1

[root@LVS /]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
203.231.63.70 0.0.0.0 255.255.255.255 UH 0 0 0 eth1
203.231.63.74 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
203.231.63.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 203.231.63.254 0.0.0.0 UG 0 0 0 eth0

[root@LVS /]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_always_defrag = 0
kernel.sysrq = 0

[root@LVS /]# vi /etc/ha.d/conf/ldirectord.cf
timeout=3
checkinterval=5
autoreload=no
fallback=127.0.0.1:80
virtual=203.231.63.70:80
real=203.231.63.71:80 gate 1
real=203.231.63.72:80 gate 1
service=http
request="index.html"
receive="Test Page"
scheduler=rr
protocol=tcp

----] in Real Server(231.63.71,72);

SVR1# ifconfig -a
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 203.231.63.70 netmask 0xffffffff broadcast 203.231.63.70
fxp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 203.231.63.72 netmask 0xffffff00 broadcast 203.231.63.255

SVR2 in the same way..

******* Problem:

1. LVS dosen't forward http request packet to the real server..
2. I don't know how FreeBSD(real server) can avoid arp request..

About first problem:

Client try to connect 203.231.63.70:80, but LVS doesn't reply to that SYN packet..

There are 2 things that seems odd..

First, When I start up ldirectord, output is like this..

[root@LVS /]# /etc/rc.d/init.d/ldirectord start
Starting ldirectord [ OK ]
[root@LVS /]# vi /var/log/ldirectord.log
......
[Tue Jan 16 13:47:48 2001..] Starting Linux Director Daemon
[Tue Jan 16 13:47:48 2001..] Adding virtual server: 203.231.63.70:80
[Tue Jan 16 13:47:48 2001..] Starting fallback server for: 203.231.63.70:80
[Tue Jan 16 13:47:49 2001..] Adding real server: 203.231.63.71:80 (1*203.231.63.70:80)
[Tue Jan 16 13:47:49 2001..] Turning off fallback server for: 203.231.63.70:80
[Tue Jan 16 13:47:49 2001..] system(/sbin/ipvsadm -a -t 203.231.63.70:80 -R 203.231.63.72:80 -g -w 1) failed
[Tue Jan 16 13:47:49 2001..] Adding real server: 203.231.63.72:80 (2*203.231.63.70:80)
......

system(/sbin/ipvsadm -a -t 203.231.63.70:80 -R 203.231.63.72:80 -g -w 1) failed
**Why this error occured?? What should I do to eliminate this error message??

Second, Here's my ipvsadm output:
[root@LVS /]# ipvsadm -L -n
IP Virtual Server version 0.9.7 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 203.231.63.70:www rr
-> 255.255.255.255:52199 Masq 4194304 0 0

Last output line seems wrong,, I think It should look like this.. right?

TCP 203.231.63.70:www rr
-> 203.231.63.71:80 route 1 0 0
-> 203.231.63.72:80 route 1 0 0

**How can I fix this thing??

Second problem:

As you know.. in LVS cluster, real servers should not reply to arp request that
asks VIP's MAC address.. Only LVS should reply to that arp request..
I have an idea about it.. Let the real server reply to client's arp request(for VIP)
with LVS's hardware address.. then all client's packet that towards VIP go to the LVS..
That's a good idea..
so I commanded like this..

arp -s 203.231.63.70 00:10:5A:76:02:49 pub

203.231.63.70 is VIP and 00:10:5A:76:02:49 is LVS's MAC address..

I got a meessage :
set: proxy entry exists for non 802 device

without 'pub' flag :
set: can only proxy for 203.231.63.70

***What is this message means??
***ARP reply with another MAC address is impossible??
***How can I solve this arp problem??

Thanks in advance..

Re: LSV with FreeBSD [ In reply to ]

mack.joseph at epa

Jan 17, 2001, 4:56 AM

Post #2 of 3 (998 views)

Permalink

> I am setting up a LVS/DR cluster with 2 nodes(FreeBSD), but It doesn't work.

> 1. LVS dosen't forward http request packet to the real server..
> 2. I don't know how FreeBSD(real server) can avoid arp request..

> Second, Here's my ipvsadm output:
> [root@LVS /]# ipvsadm -L -n
> IP Virtual Server version 0.9.7 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 203.231.63.70:www rr
> -> 255.255.255.255:52199 Masq 4194304 0 0
>
> Last output line seems wrong,, I think It should look like this.. right?
>
> TCP 203.231.63.70:www rr
> -> 203.231.63.71:80 route 1 0 0
> -> 203.231.63.72:80 route 1 0 0
>
> **How can I fix this thing??

you are probably using a version of ipvsadm that doesn't match the ipvs.

> Second problem:
>
> As you know.. in LVS cluster, real servers should not reply to arp request that
> asks VIP's MAC address.. Only LVS should reply to that arp request..

there are many ways of handing the arp problem. Arranging for the real-servers
to not reply
to arp requests is one of them, but there are others (see the HOWTO).

> I have an idea about it.. Let the real server reply to client's arp request(for VIP)
> with LVS's hardware address.. then all client's packet that towards VIP go to the LVS..
> That's a good idea..
> so I commanded like this..
>
> arp -s 203.231.63.70 00:10:5A:76:02:49 pub
>
> 203.231.63.70 is VIP and 00:10:5A:76:02:49 is LVS's MAC address..
>
> I got a meessage :
> set: proxy entry exists for non 802 device
>
> without 'pub' flag :
> set: can only proxy for 203.231.63.70

on Linux, the 'pub' flag fails too, but without it's fine. I don't know what
your messages are about.

What you are doing is putting an entry into the real-server's arp table telling
it how to find the ethernet card with the VIP. The real-server doesn't need to
know where the VIP is. It already knows that it's local.

The machine that needs to know the MAC address of the VIP is the router. You can
put
an arp entry for the VIP into the router's arp table instead. This should fix
the problem.

AFAIK ifconfig on BSD has a -arp option, which should turn off arp replies from
your
real-servers.

Another way then to handle the arp problem would be to try a command like

SVR1# ifconfig lo0 alias $VIP netmask 0xffffffff -arp up

(this is from ratz, who figured all these things out, but we haven't had a lot
of people test them yet.)

When you get it to work, can you send me the commands you used to set up your
lvs?
Thanks

Joe
--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Re: LSV with FreeBSD [ In reply to ]

ratz at tac

Jan 17, 2001, 5:13 AM

Post #3 of 3 (938 views)

Permalink

Hi,

First of all, this is a very nice problem description. We will be releasing
a draft for problem description soon and it'll look like that one.

cwmin@simplexi.com wrote:
>
> Hello,
>
> I am setting up a LVS/DR cluster with 2 nodes(FreeBSD), but It doesn't work.
>
> Here is my network configuration;
>
> Internet (203.231.63.70 is Virtual IP)
> |
> |
> Router (203.231.63.0/24 network)
> |
> | <----- eth0 : 203.231.63.74
> LVS(Linux)
> | <----- eth1 : 203.231.63.70 (VIP)

Wow, here you might have a routing decision problem. Where should a
packet with 203.231.63.71 go for example? Take out one NIC and set
the VIP on eth0:0 and take care of the arp problem (Setup and arp
problem description is all in the HOWTO). If you really want to use
the LVS with DR and in a router like setup, which is rather a not
so intelligent design since incoming/outgoing packets must be processed
be the LVS and therefore we have LVS-NAT, you must use a private net
and do some simple routing. However I recommend you my first proposal.

> |
> ------------------
> | |
> SVR1 SVR2 <-- Real Servers are FreeBSD 4.2-RELEASE
>
> fxp0 : 203.231.63.70 203.231.63.70 (VIP)
> fxp1 : 203.231.63.71 203.231.63.72 (Real IP)

This is ok.

> ----] in Load Valancing Server(203.231.63.74);
>
> [root@ha1 log]# ifconfig -a
> eth0 Link encap:Ethernet HWaddr 00:10:5A:80:D7:FF
> inet addr:203.231.63.74 Bcast:203.231.63.255 Mask:255.255.255.0
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
>
> eth1 Link encap:Ethernet HWaddr 00:10:5A:76:02:49
> inet addr:203.231.63.70 Bcast:203.231.63.70 Mask:255.255.255.255
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

Ok, now routing decision problem, but still, with LVS-DR you only need
one NIC normally. Have a look at the HOWTO under section 2.1.2.2ff the
first following picture shows you the concept of Direct Routing.

> [root@LVS /]# route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use Iface
> 203.231.63.70 0.0.0.0 255.255.255.255 UH 0 0 0 eth1
> 203.231.63.74 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
> 203.231.63.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

Well, see, here you had to add manually a route which is not needed.
Again, please change your setup and it will work.

> 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
> 0.0.0.0 203.231.63.254 0.0.0.0 UG 0 0 0 eth0
>
> [root@LVS /]# sysctl -p
> net.ipv4.ip_forward = 1
> net.ipv4.conf.all.rp_filter = 1
> net.ipv4.ip_always_defrag = 0
> kernel.sysrq = 0

Perfect, you might even set rp_filter to 2.

> [root@LVS /]# vi /etc/ha.d/conf/ldirectord.cf
> timeout=3
> checkinterval=5
> autoreload=no
> fallback=127.0.0.1:80
> virtual=203.231.63.70:80
> real=203.231.63.71:80 gate 1
> real=203.231.63.72:80 gate 1
> service=http
> request="index.html"
> receive="Test Page"
> scheduler=rr
> protocol=tcp

I don't know ldirectord but it looks ok for me.

> ----] in Real Server(231.63.71,72);
>
> SVR1# ifconfig -a
> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> inet 203.231.63.70 netmask 0xffffffff broadcast 203.231.63.70

You forgot to set the NOARP flag. Oh wait, again two NICs? You just need
one NIC and a lo that doesn't arp. See explaination below or read section
8.3.1.2 about Other Unices.

> fxp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> inet 203.231.63.72 netmask 0xffffff00 broadcast 203.231.63.255
>
> SVR2 in the same way..
>
> ******* Problem:
>
> 1. LVS dosen't forward http request packet to the real server..

...see description above why. If you tcpdump, remeber the signature and the
pattern of the dump to avoid future problems like this.

> 2. I don't know how FreeBSD(real server) can avoid arp request..

ifconfig fxp0 netmask 0xffffff00 broadcast 203.231.63.255 up
ifconfig lo0 alias 203.231.63.70 netmask 0xffffffff -arp up

> About first problem:
>
> Client try to connect 203.231.63.70:80, but LVS doesn't reply to that SYN packet..

You send the SYN, the client sends a SYN/ACK and this is not directly seen by
the LVS in DR mode.

> system(/sbin/ipvsadm -a -t 203.231.63.70:80 -R 203.231.63.72:80 -g -w 1) failed
> **Why this error occured?? What should I do to eliminate this error message??

> Second, Here's my ipvsadm output:
> [root@LVS /]# ipvsadm -L -n
> IP Virtual Server version 0.9.7 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 203.231.63.70:www rr
> -> 255.255.255.255:52199 Masq 4194304 0 0

Could you please patch your kernel with a more recent LVS-patch level?
And do compile the correct ipvsadm tool which is in the package as well.

> Last output line seems wrong,, I think It should look like this.. right?
>
> TCP 203.231.63.70:www rr
> -> 203.231.63.71:80 route 1 0 0
> -> 203.231.63.72:80 route 1 0 0

Yes, it should look like this, except the www will be 80 because you told
above that you use -n :)

> **How can I fix this thing??

See above.

> Second problem:
>
> As you know.. in LVS cluster, real servers should not reply to arp request that
> asks VIP's MAC address.. Only LVS should reply to that arp request..

Correct.

> I have an idea about it.. Let the real server reply to client's arp request(for VIP)
> with LVS's hardware address.. then all client's packet that towards VIP go to the LVS..
> That's a good idea..
> so I commanded like this..
>
> arp -s 203.231.63.70 00:10:5A:76:02:49 pub

See the arp section in the HOWTO about solutions like yours.

> 203.231.63.70 is VIP and 00:10:5A:76:02:49 is LVS's MAC address..
>
> I got a meessage :
> set: proxy entry exists for non 802 device

Haven't seen it yet but the 802 indicates very low level stuff.
Its CSMA/CD and you can get more infos at:
http://www.cis.ohio-state.edu/~gurari/course/cis677/ethernet.html

> without 'pub' flag :
> set: can only proxy for 203.231.63.70
>
> ***What is this message means??
> ***ARP reply with another MAC address is impossible??

IMHO yes but I don't know if I understood you here. But if so, then
you would have twice the same MAC on different nodes but in the same
physical net you end up with a mess!

> ***How can I solve this arp problem??

See the HOWTO

> Thanks in advance..

no problem, as long as you directly address the mailinglist ;)
Best regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`