Mailing List Archive

Matthias Weidle wrote:
>
> hi!
>
> i would like to use lvs to load balance several firewall/vpn boxes. after
> reading all the excellent documentation on this project i have learnt that
> in most of the cases lvs is used to do load sharing between a bunch of
> servers, so you usually have only one lvs director (or 2 for failover) in
> front of your server cluster.
>
> doing load balancing between firewalls/vpn boxes would require a slightly
> different setup: one director in front of the cluster and one behind. ok,
> lets have some ascii art to illustrate this:
>
> ________
> | |
> | host A |
> |________| on the internet
> |
> |
> (router)
> |
> |
> __________
> | |
> | LVS-DR 1 |
> |__________|
> |
> |
> -------------------------------------
> | | |
> | | |
> ____________ ____________ ____________
> | | | | | |
> | fw/vpn box | | fw/vpn box | | fw/vpn box |
> |____________| |____________| |____________|
>
> | | |
> | | |
> -------------------------------------
> |
> |
> __________
> | |
> | LVS-DR 2 |
> |__________|
> |
> | internal network
> -------------------------------------
> | | |
> | | |
> ________
> | |
> | host B |
> |________|
>
> lets say host A on the internet wants to talk to host B on the internal
> LAN. in case of a new connection the connection setup traffic would arrive
> on DR1 which has no table entry for host A yet and hence decides according
> to the choosen strategy which fw/vpn box is next to use. after the packet
> has been processed on that fw/vpn box it would arrive on DR 2 which in turn
> forwards the packet to the internal lan where it reaches host B. now B
> wants to acknowledge the packet of host A.

I'm OK to here.

the response would pass DR2 who
> thinks that this packet is for a new ip flow and therefor choses one of the
> boxes according to the configured strategy.

I don't understand this sentence. What does "pass DR2" mean?

The connect request packet from A will go through DR-1, fw, DR-2 and arrive
on host B. The response from B will go to its default gw (which it seems
you understand).



and this is definitly _not_
> what you want to happen here! the expected behaviour for this setup would
> be to forward the packet from host B to the box where the traffic from host
> A arrived in the first place.

I assume this means fw-1, fw-2, fw-3?

I'm assuming that the fw boxes (after doing their checking) as just passing
packets
transparently.

When the connect request arrives at B, there is no evidence of the route it took
to get there. It only has the destination IP and the source IP (the client's IP)
to identify it. If the fw started a proxy session with its own IP's and sent
them to DR-2 then you could put 3 VIPs on DR-2.

> is there any way to tell DR2 to remember where the initial packets came
> from?

From what I understand of this setup at the moment, no.

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Matthias Weidle wrote:
>
> now B
> wants to acknowledge the packet of host A. the response would pass DR2 who
> thinks that this packet is for a new ip flow

you may have to explain your setup to me some more. Does the
fw box start up
a new session, with new source IPs on the packets?

and therefor choses one of the
> boxes according to the configured strategy. and this is definitly _not_
> what you want to happen here! the expected behaviour for this setup would
> be to forward the packet from host B to the box where the traffic from host
> A arrived in the first place.

what happens if there is no 2nd director (ie is was replaced
with a switch)?
Where would the packets go?

Joe


--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer
Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

>> the response would pass DR2 who
>> thinks that this packet is for a new ip flow and therefor choses one of
the
>> boxes according to the configured strategy.

>I don't understand this sentence. What does "pass DR2" mean?
>
>The connect request packet from A will go through DR-1, fw, DR-2 and arrive
>on host B. The response from B will go to its default gw (which it seems
>you understand).

all the hosts on the internal LAN have DR2 as their default GW. hence the
response from B to A will go through DR2 which in turn has to make a
decision which fw box has to be used.

for the lvs config that means that we have to configure one VIP on each lvs
director: the first VIP for the fw cluster seen from the outside and one
VIP for the cluster as seen from the inside. (to be exactly: the internal
fw VIP is the default GW address for the internal hosts)

>> and this is definitly _not_
>> what you want to happen here! the expected behaviour for this setup would
>> be to forward the packet from host B to the box where the traffic from
host
>> A arrived in the first place.
>
>I assume this means fw-1, fw-2, fw-3?

exactly. if the connection setup packet goes through dr1-fw1-dr2 i want the
response packet to go the same way backwards ...

>I'm assuming that the fw boxes (after doing their checking) as just passing
>packets transparently.

well, more or less: yes.
source and destination ip's won't be changed by the firewall.

the main reason that i need persistant connections in both ways is the vpn
software installed on the fw boxes:
if A wants to talk to B it first has to establish a secure tunnel to one of
the fw boxes. then the secure traffic can be passed through that very
tunnel to the internal host B.
since the fw boxes don't share any keys i have to go sure that the backward
traffic from B to A goes through that box too!


>When the connect request arrives at B, there is no evidence of the route
it took
>to get there. It only has the destination IP and the source IP (the
client's IP)
>to identify it. If the fw started a proxy session with its own IP's and
sent
>them to DR-2 then you could put 3 VIPs on DR-2.

no, its no proxy session. you still have the src=A, dst=B when the packet
arrives on host B. therefor the only point to remember the path (i.e. which
fw box) the incoming packet to host B took is on DR2.

here are my thoughts how this could be done by the lvs code:

if a packet arrives on DR2 the lvs code has to inspect if the src MAC
address of this packet is one of the known fw boxes. if that is the case it
doesn't have to do any load balancing decision, it only has to remember the
src and dst ip of that packet together with the MAC address of the fw box
in some internal table.
if the src MAC address of the incoming packet is not any of the configured
fw boxes, then the packet came from the attached internal LAN and a load
balancing decision has to be made if there is NOT yet an entry in the
connection tracking table. if we already got a matching entry (same src and
dst ips) we have to use the already associated fw box for this connection
(means forwarding the packet for further processing to the MAC of the fw
box).

of course the same logic has to be applied to DR1 in case of connection
setup from the internal LAN to the outside internet :)


i hope i could bring some more light in that issue! :)

best regards,
-- matt.


PS: since i am subscribed in digest mode to this mailing list ... could you
please do a CC to my mail address when answering? would be easier to make
replies :))


>> is there any way to tell DR2 to remember where the initial packets came
>> from?

From what I understand of this setup at the moment, no.
>
>Joe

Matthias Weidle wrote:

>
> for the lvs config that means that we have to configure one VIP on each lvs
> director: the first VIP for the fw cluster seen from the outside and one
> VIP for the cluster as seen from the inside. (to be exactly: the internal
> fw VIP is the default GW address for the internal hosts)

the default gw for the fw you mean?

> the main reason that i need persistant connections in both ways is the vpn
> software installed on the fw boxes:
> if A wants to talk to B it first has to establish a secure tunnel to one of
> the fw boxes. then the secure traffic can be passed through that very
> tunnel to the internal host B.
> since the fw boxes don't share any keys i have to go sure that the backward
> traffic from B to A goes through that box too!

got it.

> here are my thoughts how this could be done by the lvs code:
>
> if a packet arrives on DR2 the lvs code has to inspect if the src MAC
> address of this packet is one of the known fw boxes. if that is the case it
> doesn't have to do any load balancing decision, it only has to remember the
> src and dst ip of that packet together with the MAC address of the fw box
> in some internal table.

hmm, might be possible. there are lots of tools which fiddle with packets, eg
fwmarks, the new routing stuff.

(This isn't neccessarily a good solution, but it's a first step).
Howabout: each fw box also does NAT. Packets coming out of each fw box
have the IP on the inside of the fw box as the source addr.
(These fw boxes are going to be fairly busy CPU wise).
You could then use a regular (non martian) VS-DR director for DR-2 and the
real-servers
for LVS-2 (eg host B) would only have 3 host routes (to the 3 fws, which are
the only machines they know about, since they are only getting packets with
src_addr of the fw machines).

I assume changing the IP's on the packets is going to mess up the key exchange a
bit. ;-\


> if the src MAC address of the incoming packet is not any of the configured
> fw boxes, then the packet came from the attached internal LAN and a load
> balancing decision has to be made if there is NOT yet an entry in the
> connection tracking table. if we already got a matching entry (same src and
> dst ips) we have to use the already associated fw box for this connection
> (means forwarding the packet for further processing to the MAC of the fw
> box).

are you saying -
packets from the local LAN are processed by the LVS in the normal way
(ie without consideration of fw boxes etc)


> PS: since i am subscribed in digest mode to this mailing list ... could you
> please do a CC to my mail address when answering? would be easier to make
> replies :))

OK this time, for your next posting, get yourself on the list. I have
a RFC non-compliant mail client (netscape) and when I "reply to all", I only
send to the list.

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

--On Monday, January 29, 2001 11:06:19 AM -0500 Joseph Mack
<mack.joseph@epa.gov> wrote:

>
> (This isn't neccessarily a good solution, but it's a first step).
> Howabout: each fw box also does NAT. Packets coming out of each fw box
> have the IP on the inside of the fw box as the source addr.
> (These fw boxes are going to be fairly busy CPU wise).
> You could then use a regular (non martian) VS-DR director for DR-2 and the
> real-servers
> for LVS-2 (eg host B) would only have 3 host routes (to the 3 fws, which
> are the only machines they know about, since they are only getting
> packets with src_addr of the fw machines).
>
> I assume changing the IP's on the packets is going to mess up the key
> exchange a bit. ;-\

yes, absolutely. doing NAT is no option at all since it would break the VPN
stuff.

>> if the src MAC address of the incoming packet is not any of the
>> configured fw boxes, then the packet came from the attached internal LAN
>> and a load balancing decision has to be made if there is NOT yet an
>> entry in the connection tracking table. if we already got a matching
>> entry (same src and dst ips) we have to use the already associated fw
>> box for this connection (means forwarding the packet for further
>> processing to the MAC of the fw box).
>
> are you saying -
> packets from the local LAN are processed by the LVS in the normal way
> (ie without consideration of fw boxes etc)

no, i think you got me wrong on that.

1. for the communicating hosts A and B the number of involved fw boxes is
completely transparent. they don't know that there is a load balancing
mechanism between them. host B on the local LAN only knows its default GW
for sending packets to the outside: this is the VIP of the fw cluster (i.e.
the VIP on DR2 = VIP2). host A on the other hand knows that there is
something between A and B whis has to be used to establish the secure
tunnel: this is VIP1 (the VIP of the fw cluster on DR1).

2. since the fw boxes don't share any key information we have to go sure
that all packets for a particular connection between A and B only travel
over exactly _one_ fw box (this includes of course both directions A->B and
B->A).

3. the only instances which know about the involved fw box for the
communication between A and B are the two LVS directors DR1 and DR2. for
both DR's i would chose the persistant mode, so that a load balancing
decision (i.e. which fw box to use) has to be made only for the very first
packet from A to B (this happens on DR1). all the following packets will be
just forwarded to the choosen fw box. DR2 doesn't do any load balancing for
incoming traffic from A to B - it only has to remember which fw box the
packet came from (i.e. which SRC MAC the packet had), so that the response
traffic from B to A can flow over the same fw box.
for connections initiated on host B the mechanism works the other way
round. that means DR2 makes a load balancing decision and DR1 has to
remember the fw box the packet arrived from.



as i understand the LVS for now there is not any mechanism already included
which can handle the response traffic the way i do need it in my specific
application.
so what i do need to know is whether there are already similar mechanisms
(maybe for LVS-NAT) which do inspect those return packets and how much
effort you guys think it would take to include such behaviour for that
special case of LVS-DR (if at all possible).


best regards,
-- matt.

Matthias Weidle wrote:
>

> > I assume changing the IP's on the packets is going to mess up the key
> > exchange a bit. ;-\
>
> yes, absolutely. doing NAT is no option at all since it would break the VPN
> stuff.

hmm, I don't know a whole lot about vpn. which pairs of machines are exchanging
keys?
Host B and the fw machines?

> >> if the src MAC address of the incoming packet is not any of the
> >> configured fw boxes, then the packet came from the attached internal LAN
> >> and a load balancing decision has to be made if there is NOT yet an
> >> entry in the connection tracking table. if we already got a matching
> >> entry (same src and dst ips) we have to use the already associated fw
> >> box for this connection (means forwarding the packet for further
> >> processing to the MAC of the fw box).
> >
> > are you saying -
> > packets from the local LAN are processed by the LVS in the normal way
> > (ie without consideration of fw boxes etc)
>
> no, i think you got me wrong on that.
>
> 1. for the communicating hosts A and B the number of involved fw boxes is
> completely transparent. they don't know that there is a load balancing
> mechanism between them. host B on the local LAN only knows its default GW
> for sending packets to the outside: this is the VIP of the fw cluster (i.e.
> the VIP on DR2 = VIP2).

If host B is a real-server in LVS-2, then its default gw can't be VIP2. It
has to be an IP on the director (for VS-NAT) or the router on the outside
of director-2 (if VS-DR/VS-Tun).

host A on the other hand knows that there is
> something between A and B whis has to be used to establish the secure
> tunnel: this is VIP1 (the VIP of the fw cluster on DR1).

again I don't know enough about vpn here. I would assume that host A is just
sending packets to VIP1 and has no idea about vpn/fw or what happens to the
packet after it reaches director1.

> 2. since the fw boxes don't share any key information we have to go sure
> that all packets for a particular connection between A and B only travel
> over exactly _one_ fw box (this includes of course both directions A->B and
> B->A).

got that.

> 3. the only instances which know about the involved fw box for the
> communication between A and B are the two LVS directors DR1 and DR2.

they have to know about the fws to make reply packets go back along
the path that the incoming packets traversed.

for
> both DR's i would chose the persistant mode, so that a load balancing
> decision (i.e. which fw box to use) has to be made only for the very first
> packet from A to B (this happens on DR1). all the following packets will be
> just forwarded to the choosen fw box. DR2 doesn't do any load balancing for
> incoming traffic from A to B - it only has to remember which fw box the
> packet came from (i.e. which SRC MAC the packet had), so that the response
> traffic from B to A can flow over the same fw box.
> for connections initiated on host B the mechanism works the other way
> round. that means DR2 makes a load balancing decision and DR1 has to
> remember the fw box the packet arrived from.
>
> as i understand the LVS for now there is not any mechanism already included
> which can handle the response traffic the way i do need it in my specific
> application.

yes

> so what i do need to know is whether there are already similar mechanisms
> (maybe for LVS-NAT) which do inspect those return packets and how much
> effort you guys think it would take to include such behaviour for that
> special case of LVS-DR (if at all possible).

LVS is all layer-4 (at the moment). So you could make your scheme work
if the packets coming out of the fw had different IP's on them. The NAT
scheme for doing this you've already discounted.

Your suggestion was differentiating by MAC address (which I think is layer-3,
tcpip layers don't map onto the ISO-7 layers real well). I'm sure it's doable.
How much work it would be I don't know. You'd have to familiarise yourself
with the LVS code and start programming.

I had a think about this problem yesterday afternoon, and couldn't come up with
any other ideas.

Joe


--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Joseph Mack wrote:
>
> hmm, I don't know a whole lot about vpn. which pairs of machines are exchanging
> keys?

been thinking some more while I'm waiting for your reply.

Do the vpn machines hold session management info (ie do they
check that the ingoing packets get a reply). If they don't,
then you could use VS-DR for both directors and send the
reply from host B to the router in front of the first director

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA