Mailing List Archive

Hello,

On Tue, 30 Jan 2001, Joseph Mack wrote:

> In a VS-NAT LVS, the incoming packets are NAT'ed (or
> whatever it's called) to the real-servers.
>
> The returning packets are NAT'ed (in the opposite sense)
> on the director by rules like
>
> ipchains -A forward -p tcp -j MASQ -s realserver1 telnet -d 0.0.0.0/0
>
> If there are several IP's on the outside of the director eg
>
> 192.168.2.1 eth0
> 192.168.2.110 eth0:1 (the VIP)
>
> how does the director know to make the src_addr = VIP
> for the outgoing packet?
>
> In general for a NAT box, how is the src_addr chosen for masqueraded
> packets, when there are several IPs on the outside of the NAT box.

The rule is simple: only the routing decides when outgoing
connections are created initiated from the NAT-ed servers. When LVS
creates connections to NAT-ed real servers the VIP is set as the
masquerading address.

How one can select specific source addresses for the
masquerading:

ip route add 10.0.0.0/24 brd + dev eth0 via uplink1 src SRCIP1
ip route add 10.0.1.0/24 brd + dev eth0 via uplink2 src SRCIP2

Source routing can't be used to select the maddr for the
connections initiated from the internal hosts. It will be ignored.

In this example, the SRCIP1 and SRCIP2 will be selected
according to the destination. This is for connections initiated from
the internal servers. LVS always assigns VIP as maddr.

So, the rule when using masquerading with many addresses
is to setup correctly the source addresses for each route. The
default value is usually the first interface address:

ip route

> Joe

Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:

>
> The rule is simple: only the routing decides when outgoing
> connections are created initiated from the NAT-ed servers. When LVS
> creates connections to NAT-ed real servers the VIP is set as the
> masquerading address.

with an VS_NAT setup, when I run

> ipchains -A forward -p tcp -j MASQ -s realserver1 telnet -d 0.0.0.0/0

on the director, so that telnet packets can get back from the real-server
to the client, LVS steps in and makes sure that the VIP becomes the
src_addr, whereas if there had been no ipvsadm commands run, then
the routing would have determined the src_addr?

Does LVS fiddle with the ipchains tables to do this?

Joe


--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Julian Anastasov wrote:
>

I need to clarify some nomenclature here.
In an LVS, which direction is the masquerading,
the inward or outward direction?
what's the other direction called? reverse masquerading?

> > Does LVS fiddle with the ipchains tables to do this?
>
> No, ipchains only delivers packets to the masquerading code.
> Nobody is interested how the packets are selected in the ipchains
> rule.

OK something else determines what happens to the packet.

>
> How the things work:
>
> - the masquerading address is assigned when the first packet is seen
>
> - LVS sees the first packet in the LOCAL_IN chain when it comes from
> the client. LVS assigns the VIP as maddr

OK

> - the MASQ code sees the first packet in the FORWARD chain when
> there is a -j MASQ target in the ipchains rule.
> The routing selects the maddr.

this is the outward going packet right?
this is for LVS or a machine behind a regular NAT box or both?

>If the connection already exists the packets are masqueraded.

what about a machine behind a NAT box initiating a telnet session?
Doesn't its first outward packet (the telnet connect request) have to be
masqueraded?

> - the LVS can see packets in the FORWARD chain but they are for already
> created connections, so no maddr is assigned and the packets are
> masqueraded with the address saved in the connections structure (the
> VIP) when it was created.

OK

from the earlier posting

>
> How one can select specific source addresses for the
> masquerading:
>
> ip route add 10.0.0.0/24 brd + dev eth0 via uplink1 src SRCIP1
> ip route add 10.0.1.0/24 brd + dev eth0 via uplink2 src SRCIP2
>
> Source routing can't be used to select the maddr for the
> connections initiated from the internal hosts. It will be ignored.

I don't understand this last statement. You look like you're showing
an example which depends on the src IP, but then you say the source
will be ignored.

> In this example, the SRCIP1 and SRCIP2 will be selected
> according to the destination. This is for connections initiated from
> the internal servers. LVS always assigns VIP as maddr.

eth0 has SRCIP1 and SRCIP2 on it (presumably one is an alias?)

> So, the rule when using masquerading with many addresses
> is to setup correctly the source addresses for each route. The
> default value is usually the first interface address:

thanks

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Wed, 31 Jan 2001, Joseph Mack wrote:

> with an VS_NAT setup, when I run
>
> > ipchains -A forward -p tcp -j MASQ -s realserver1 telnet -d 0.0.0.0/0
>
> on the director, so that telnet packets can get back from the real-server
> to the client, LVS steps in and makes sure that the VIP becomes the
> src_addr, whereas if there had been no ipvsadm commands run, then
> the routing would have determined the src_addr?

Right.

> Does LVS fiddle with the ipchains tables to do this?

No, ipchains only delivers packets to the masquerading code.
Nobody is interested how the packets are selected in the ipchains
rule.

How the things work:

- the masquerading address is assigned when the first packet is seen

- LVS sees the first packet in the LOCAL_IN chain when it comes from
the client. LVS assigns the VIP as maddr

- the MASQ code sees the first packet in the FORWARD chain when
there is a -j MASQ target in the ipchains rule. The routing selects
the maddr. If the connection already exists the packets are masqueraded.

- the LVS can see packets in the FORWARD chain but they are for already
created connections, so no maddr is assigned and the packets are
masqueraded with the address saved in the connections structure (the
VIP) when it was created.

> Joe


Regards

--
Julian Anastasov <ja@ssi.bg>

Hello,

On Wed, 31 Jan 2001, Joseph Mack wrote:

> Julian Anastasov wrote:
> >
>
> I need to clarify some nomenclature here.
> In an LVS, which direction is the masquerading,
> the inward or outward direction?

from client to NAT-ed real server we call it demasquerading
from NAT-ed real server to client is masquerading
from client to DR real server is ?forwarding?
from client to TUN real server is ?tunneling?

> what's the other direction called? reverse masquerading?

Yes, reverse or de-masquerading

> > - the MASQ code sees the first packet in the FORWARD chain when
> > there is a -j MASQ target in the ipchains rule.
> > The routing selects the maddr.

In fact, the masq code calls the routing to select maddr.

> this is the outward going packet right?
> this is for LVS or a machine behind a regular NAT box or both?

This is for normal NAT. LVS does not create connections
in the outward direction.

> >If the connection already exists the packets are masqueraded.
>
> what about a machine behind a NAT box initiating a telnet session?
> Doesn't its first outward packet (the telnet connect request) have to be
> masqueraded?

Yes, these packets are always masqueraded if the ipchains rule
delivers these packets to the masquerading. And this can be done only
in the FORWARD chain. The demasqueraded packets walk the forwarding but
not the forwarding firewall chain. The forwarding firewall chain is
traversed only for outward (masqueraded) packets.

> > ip route add 10.0.0.0/24 brd + dev eth0 via uplink1 src SRCIP1
> > ip route add 10.0.1.0/24 brd + dev eth0 via uplink2 src SRCIP2
> >
> > Source routing can't be used to select the maddr for the
> > connections initiated from the internal hosts. It will be ignored.
>
> I don't understand this last statement. You look like you're showing
> an example which depends on the src IP, but then you say the source
> will be ignored.

Source routing is when we use "from SOURCE" in the ip
rule. By this way we match the packet source address. The "src"
option defines the default source address for the route. Here is
one example:

ip rule add prio 100 from 192.168.0.1 table 100
ip route add table 100 10.0.0.1 via 192.168.1.2 dev eth0 src 192.168.1.3

> > In this example, the SRCIP1 and SRCIP2 will be selected
> > according to the destination. This is for connections initiated from
> > the internal servers. LVS always assigns VIP as maddr.
>
> eth0 has SRCIP1 and SRCIP2 on it (presumably one is an alias?)

The "alias" in 2.2+ is only an ifconfig magic. In the "ip"
command they are called "labels". In fact, they are only names.
We can attach many addresses to one interface. We can even attach
addresses with different priorities (ip addr add 10.0.0.1/24 dev lo
scope 1) and by this way to reorder them by this priority and not
by the order based on the creation time.

> thanks
>
> Joe


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:

> Here is one example:
>
> ip rule add prio 100 from 192.168.0.1 table 100
^^^^^^^
I assume this should be 10.0 ?
> ip route add table 100 10.0.0.1 via 192.168.1.2 dev eth0 src 192.168.1.3

I have a VS-DR LVS. One of the real-servers is telnet'ing
to the outside world with NAT, using your instructions of 25 Sep 2000.
I haven't tested, but I presume these NAT'ed packets
arrive with s_addr = the IP (on eth0) of the director. I would like them
to arrive with s_addr = VIP (on eth0:1)

the realserver is 192.168.1.12. The VIP is 192.168.2.110 (eth0:1
on the director), the director's NIC facing the outside world is 192.168.2.1.
All IPs are pingable from the director (only the realserver's IP is not
on the director.)

To get telnet packets from the real-server to emerge with s_addr = VIP,
I tried

ip rule add prio 100 from 192.168.1.12 table 100
ip route add table 100 192.168.1.12 via 192.168.2.1 dev eth0 src 192.168.1.110

the last command gives the error

RTNETLINK answers: Network is unreachable.

Do you know what's going on?

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Julian Anastasov wrote:

> ip route add default dev eth0 via uplink src VIP

the iproute2 HOWTO says that uplink should be the IP of
the machine being NAT'ed (here 192.168.1.12).

my command was

ip route add default dev eth0 via 192.168.1.12 src 192.168.2.110
| |
RIP VIP


I tried the IP of the director and the client, but got errors
running the command (network not available or something like that).

> Do I know this setup? What IP
> addresses and routes are defined in the director?

client
(eth0 192.168.2.254)
|
(eth0 192.168.2.1 = director)
(eth0:1 192.168.2.110 = VIP)
director
(eth1:192.168.1.9)
|
(eth0 192.168.1.11)
(lo:0 192.168.2.110 = VIP)
real-server

director has ip_forwarding on, is just NAT'ing at the moment
(have run ipvsadm -C)

tcpdump on client shows telnet packets arrive with src=director

If I can get sessions initiating on the real-server to emerge from
the director with s_addr=VIP, I want to use this to get the connect
requests from authd on the real-servers out to the client. The
authd connect requests from the real-server have s_addr = VIP.
Presumably I can NAT this on the real-server to the RIP
before it is sent to the director?

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Thu, 1 Feb 2001, Joseph Mack wrote:

> Julian Anastasov wrote:
>
> > Here is one example:
> >
> > ip rule add prio 100 from 192.168.0.1 table 100
> ^^^^^^^
> I assume this should be 10.0 ?
> > ip route add table 100 10.0.0.1 via 192.168.1.2 dev eth0 src 192.168.1.3
>
> I have a VS-DR LVS. One of the real-servers is telnet'ing
> to the outside world with NAT, using your instructions of 25 Sep 2000.
> I haven't tested, but I presume these NAT'ed packets
> arrive with s_addr = the IP (on eth0) of the director. I would like them
> to arrive with s_addr = VIP (on eth0:1)

It seems this is not possible with -j MASQ. When a maddr is
selected the only argument used in the decision is the destination
address (where you try to connect to). The source address 192.168.1.12
is not considered. May be this is possible with the dumb nat, i.e.
ip route ... nat .... But it is stateless and collisions can occur
with the normally masqueraded connections.

So, if we define:

ip route add default dev eth0 via uplink src VIP

the VIP will be always selected as maddr. The primary address will not
be selected. And this is a problem when we want different VIPs to be
selected. I'm not sure why the source address is not involved in the
decision. I have to investigate it and we can patch it. This fix will
allow for the different NAT-ed internal hosts different masquerade
addresses to be selected. I'm not sure whether this will work for all
places where such address is selected.

> the realserver is 192.168.1.12. The VIP is 192.168.2.110 (eth0:1
> on the director), the director's NIC facing the outside world is 192.168.2.1.
> All IPs are pingable from the director (only the realserver's IP is not
> on the director.)
>
> To get telnet packets from the real-server to emerge with s_addr = VIP,
> I tried
>
> ip rule add prio 100 from 192.168.1.12 table 100
> ip route add table 100 192.168.1.12 via 192.168.2.1 dev eth0 src 192.168.1.110
^^^^^^^^^^^^
this is destination, not a source address

> the last command gives the error
>
> RTNETLINK answers: Network is unreachable.

May be you don't have route to 192.168.2.1 or 192.168.1.11
is not local address defined on eth0? Do I know this setup? What IP
addresses and routes are defined in the director?

> Do you know what's going on?
>
> Joe


Regards

--
Julian Anastasov <ja@ssi.bg>

Hello,

On Thu, 1 Feb 2001, Joseph Mack wrote:

> > ip route add default dev eth0 via uplink src VIP
>
> the iproute2 HOWTO says that uplink should be the IP of
> the machine being NAT'ed (here 192.168.1.12).
>
> my command was
>
> ip route add default dev eth0 via 192.168.1.12 src 192.168.2.110
> | |
> RIP VIP

No :) this is the default route. It points to the uplink
router. Hm, my mail from Sep 25 is for setting source routing in the
real server. By this way we can redirect the outgoing traffic to
different uplink gateways: to the director (when NAT-ed) or to
another uplink router when for DR method.

I assume in current thread we try to solve the problem in the
director where we want to select different VIPs as masquerading
addresses. This is for the NAT-ed connections initiated from the
internal hosts and this is not related to the LVS operation. We
can remove the words LVS here. We just need to select different
masquerade addresses in the NAT box according to the internal server
IP address. This is in the FORWARD chain. For LVS we know that this
process (maddr selection) is performed in the LOCAL_IN chain.

> I tried the IP of the director and the client, but got errors
> running the command (network not available or something like that).
>
> > Do I know this setup? What IP
> > addresses and routes are defined in the director?
>
> client
> (eth0 192.168.2.254)
> |
> (eth0 192.168.2.1 = director)
> (eth0:1 192.168.2.110 = VIP)
> director
> (eth1:192.168.1.9)
> |
> (eth0 192.168.1.11)
> (lo:0 192.168.2.110 = VIP)
> real-server
>
> director has ip_forwarding on, is just NAT'ing at the moment
> (have run ipvsadm -C)
>
> tcpdump on client shows telnet packets arrive with src=director
>
> If I can get sessions initiating on the real-server to emerge from
> the director with s_addr=VIP, I want to use this to get the connect
> requests from authd on the real-servers out to the client. The
> authd connect requests from the real-server have s_addr = VIP.
> Presumably I can NAT this on the real-server to the RIP
> before it is sent to the director?

Yes, I was sure we talk for this problem. I'm still thinking on
this problem. For now, I'm sure the packets must exit from the real
server with saddr!=any_director_ip and they must be NAT-ed in the
director when forwarded to the client with maddr=VIP. So, we must
convert them in the real server:

- when there is a packet from VIP to 0.0.0.0/0 113 then convert the
source address (VIP) to RIP2 and then send it to the director.

- the director must be patched to lookup for maddr in this way:

when packet comes from RIP2 and goes to 0.0.0.0/0 then use maddr=VIP

Currently, the NAT box think in this way: when packets goes to
0.0.0.0/0 then use maddr=route_src. I.e. the saddr in the packets is
not considered.

So, we need a way to nat the outgoing packets in the real
server but only when we access the client's authd. There is no
route by fwmark for locally generated packet in 2.2, so I'm not
sure how we will distinguish port 113 from port 1024 (the client).

> Joe


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:

> So, we need a way to nat the outgoing packets in the real
> server but only when we access the client's authd.

The packets from the real-server to the client's authd
come from the VIP on the real-server and not the RIP.
We fiddle with packets on the real-server
that come from VIP with ports 1025:65535.

> There is no
> route by fwmark for locally generated packet in 2.2, so I'm not
> sure how we will distinguish port 113 from port 1024 (the client).

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Fri, 2 Feb 2001, Joseph Mack wrote:

> > So, we need a way to nat the outgoing packets in the real
> > server but only when we access the client's authd.
>
> The packets from the real-server to the client's authd
> come from the VIP on the real-server and not the RIP.

But we need they to leave from the real server with saddr=RIP2.
RIP2 will be used only to connect to remote authd through the director.
We need one RIP2 for each VIP - this will be used in the director
to select the right VIP as maddr for the different RIP2 saddr.

If the director receives packets from the real server with
saddr=VIP they will be masqueraded (martian patch) but when the client
replies the replies will not reach the real server because the
demasquerading will select the VIP as next hop when sending the
packet to the real server and will deliver the packet locally.
This is the reason we prefer these packets to be NAT-ed in the real
server at the moment when they are generated. For the director such
packets (with saddr=RIP2) will be treated as they come from NAT-ed box.
When patched the director will select the right VIP as maddr.

> We fiddle with packets on the real-server
> that come from VIP with ports 1025:65535.
>
> Joe
>
> --
> Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
> contractor to the National Environmental Supercomputer Center,
> mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:
>
> Hello,
>
> On Fri, 2 Feb 2001, Joseph Mack wrote:
>
> > > So, we need a way to nat the outgoing packets in the real
> > > server but only when we access the client's authd.
> >
> > The packets from the real-server to the client's authd
> > come from the VIP on the real-server and not the RIP.
>
> But we need they to leave from the real server with saddr=RIP2.
> RIP2 will be used only to connect to remote authd through the director.
> We need one RIP2 for each VIP - this will be used in the director
> to select the right VIP as maddr for the different RIP2 saddr.

hmm, diagram for clarification of what I'm thinking
(I only have one VIP, not sure why you have several).
(There is no LVS installed, just showing NAT parts,
but IPs are for a VS-DR LVS)

client
(eth0 192.168.2.254)
|
(eth0 192.168.2.1 = DIP)
(eth0:1 192.168.2.110 = VIP)
director
(eth1:192.168.1.9)
|
(eth0 192.168.1.12 = RIP)
(lo:0 192.168.2.110 = VIP)
real-server

the authd client on the real-server makes its call

VIP:high_port -> client:authd

the replies don't get back to the real-server.

my scheme to get it to work

on real-server

VIP:high_port -> client:authd (high_port is say 1025)

have NAT running on real-server, the
packet emerges from the real-server

RIP:higher_port -> client:authd (higher_port is say 60001)

this packet arrives at director where it is NAT'ed
again and emerges from the LVS as

VIP:even_higher_port -> client:authd (even_higher_port is ?)

Something I hadn't thought of... Can NAT accept a packet
with a port number that's already been NAT'ed?

Joe


--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Fri, 2 Feb 2001, Joseph Mack wrote:

> Julian Anastasov wrote:
> >
> > Hello,
> >
> > On Fri, 2 Feb 2001, Joseph Mack wrote:
> >
> > > > So, we need a way to nat the outgoing packets in the real
> > > > server but only when we access the client's authd.
> > >
> > > The packets from the real-server to the client's authd
> > > come from the VIP on the real-server and not the RIP.
> >
> > But we need they to leave from the real server with saddr=RIP2.
> > RIP2 will be used only to connect to remote authd through the director.
> > We need one RIP2 for each VIP - this will be used in the director
> > to select the right VIP as maddr for the different RIP2 saddr.
>
> hmm, diagram for clarification of what I'm thinking
> (I only have one VIP, not sure why you have several).

We have several RIPs:

RIP: the main RIP
RIP1 is used to translate packets VIP1:1024 -> client1:113 to
RIP1:1024 -> client1:113
RIP2 is used to translate packets VIP2:1024 -> client2:113 to
RIP2:1024 -> client2:113

and so on. This is the case where we must select the right VIP
in the director when talking to client*:113. For each VIP* in the real
server we need one unused RIP* for these cases. If we map all VIPs
to one RIP before forwarding this packet through the director we always
will select one VIP as maddr there and this breaks the ident protocol.
We need one unused RIP for each VIP that can use ident.

Then in the patched director we will use:

ip rule ... from RIP1 table 100
ip route add table 100 ... src VIP1

ip rule ... from RIP2 table 101
ip route add table 101 ... src VIP2

and will hope the patched masquerading to select the right
maddr (oh, all these troubles for such protocol).

> (There is no LVS installed, just showing NAT parts,
> but IPs are for a VS-DR LVS)
>
> client
> (eth0 192.168.2.254)
> |
> (eth0 192.168.2.1 = DIP)
> (eth0:1 192.168.2.110 = VIP)
> director
> (eth1:192.168.1.9)
> |
> (eth0 192.168.1.12 = RIP)
> (lo:0 192.168.2.110 = VIP)
> real-server
>
> the authd client on the real-server makes its call
>
> VIP:high_port -> client:authd
>
> the replies don't get back to the real-server.
>
> my scheme to get it to work
>
> on real-server
>
> VIP:high_port -> client:authd (high_port is say 1025)
>
> have NAT running on real-server, the
> packet emerges from the real-server
>
> RIP:higher_port -> client:authd (higher_port is say 60001)

No, higher_port is still equal to high_port. At least if
dumb nat is used (you can't change the port with dumb nat).

> this packet arrives at director where it is NAT'ed
> again and emerges from the LVS as
>
> VIP:even_higher_port -> client:authd (even_higher_port is ?)

Right. I was talking about the same case. So, may be we can do
it with dumb nat in Linux 2.4 but not in 2.2. We need to distinguish
the both cases:

VIP:25 -> client:1024
VIP:1024 -> client:authd

Only in 2.4 we can use fwmark in the LOCAL_OUT chain to
mark the packets when they are locally generated. So, we have to check
whether dumb nat can do the desired translation when the packet leaves
the real server and then when the replies from client:authd are received.

> Something I hadn't thought of... Can NAT accept a packet
> with a port number that's already been NAT'ed?

Yes, but we can't NAT the port for locally generated packets
in the real server.

> Joe
>
>
> --
> Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
> contractor to the National Environmental Supercomputer Center,
> mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:

> and will hope the patched masquerading to select the right
> maddr (oh, all these troubles for such protocol).

this is starting to look like a lot of trouble. I think I'll find
something else to do for a while.

thanks
Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA