Mailing List Archive

Pat O'Rourke wrote:
>
> We're trying to set up a NAT config using a 2.4.0 based director, but are
> having some difficulty in getting it working.

There are problems with 2.4 and NAT.
Julian has talked about this a bit. You might want to look for his postings
on the matter first.

the archive is at

http://marc.theaimsgroup.com/?l=linux-virtual-server&r=1&w=2

Joe
--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Pat O'Rourke wrote:

>
> I did, but I somehow got the impression that one could make LVS-NAT work on
> 2.4 using the ipchains compatability module.

I haven't kept up with this. You need to talk to Julian for the latest.
Sorry

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

> There are problems with 2.4 and NAT.
> Julian has talked about this a bit. You might want to look for his postings
> on the matter first.

I did, but I somehow got the impression that one could make LVS-NAT work on
2.4 using the ipchains compatability module. The changelog also makes several
references to VS/NAT fixes, so I assumed it would work. Are you saying that
it does not work? If this is the case, could the README of the ipvs code be
amended to state that a NAT configure is unsupported for now?

Thanks for your reply.

Pat

--
Patrick O'Rourke
orourke@missioncriticallinux.com

Hello,

On Wed, 17 Jan 2001, Pat O'Rourke wrote:

>
> > There are problems with 2.4 and NAT.
> > Julian has talked about this a bit. You might want to look for his postings
> > on the matter first.
>
> I did, but I somehow got the impression that one could make LVS-NAT work on
> 2.4 using the ipchains compatability module. The changelog also makes several
> references to VS/NAT fixes, so I assumed it would work. Are you saying that
> it does not work? If this is the case, could the README of the ipvs code be
> amended to state that a NAT configure is unsupported for now?

Yes, it is really unsupported. The tests with ipchains.o were
long time ago. It seems LVS can work with ipchains.o only for firewalling.
LVS hooks in FORWARD after the FILTER but it seems ipchains.o makes
firewalling and masquerading in one hook. So, the priorities don't play
here. Patching is required to place LVS between the FORWARD firewall and
the outgoing masquerade. But this is my first impression from the code in
ip_fw_compat_masq.c:do_masquerade() and ip_fw_compat.c:fw_in().

If you don't create any other outgoing masquerading connections
from the internal servers you can just delete the ipchains rule. The LVS
will maintain its connections without the help from ipchains. Even for
the related ICMP messages. Use ipchains rules only for filtering in this
case.

In your tests the SYN+ACK replies hit the ipchains masquerading
and the packet don't reach the LVS ip_vs_out() function. The netfilter
treats this as a new connection and masquerades the packets before
reaching LVS in the same chain.

The other problem in ip_fw_compat_masq.c is that inet_select_addr
is still involved in the routing decisions. This is wrong. The result
is that always the primary address from the outgoing device is selected
as masquerading address instead of using rt->rt_src as maddr.

Wensong! IMO, we should make some fixes to support the ipchains
compatibility mode.

> Thanks for your reply.
>
> Pat


Regards

--
Julian Anastasov <ja@ssi.bg>

On Wed, 17 Jan 2001, Julian Anastasov wrote:

>
> Yes, it is really unsupported. The tests with ipchains.o were
> long time ago. It seems LVS can work with ipchains.o only for firewalling.
> LVS hooks in FORWARD after the FILTER but it seems ipchains.o makes
> firewalling and masquerading in one hook. So, the priorities don't play
> here. Patching is required to place LVS between the FORWARD firewall and
> the outgoing masquerade. But this is my first impression from the code in
> ip_fw_compat_masq.c:do_masquerade() and ip_fw_compat.c:fw_in().
>
> If you don't create any other outgoing masquerading connections
> from the internal servers you can just delete the ipchains rule. The LVS
> will maintain its connections without the help from ipchains. Even for
> the related ICMP messages. Use ipchains rules only for filtering in this
> case.
>

Hi Pat,

Julian is right. The VS/NAT in 2.4 should work when there is no
masquerading rule specified. If your real servers don't need to access
the outside through the director (issue new connections), you can remove
the ip_conntrack module for performance.

> In your tests the SYN+ACK replies hit the ipchains masquerading
> and the packet don't reach the LVS ip_vs_out() function. The netfilter
> treats this as a new connection and masquerades the packets before
> reaching LVS in the same chain.
>
> The other problem in ip_fw_compat_masq.c is that inet_select_addr
> is still involved in the routing decisions. This is wrong. The result
> is that always the primary address from the outgoing device is selected
> as masquerading address instead of using rt->rt_src as maddr.
>
> Wensong! IMO, we should make some fixes to support the ipchains
> compatibility mode.
>

Yeah, we should do it long time ago. But it is not easy to do a good fix
on this, because conntrack tracks every incoming packet. :)

Just thought a dirty fix to support this kind of compatibility. We can
add several lines of checking code in the tcp_packet() of the
ip_conntrack_proto_tcp.c as follows:
if (oldtcpstate == TCP_CONNTRACK_NONE
&& tcph->syn && tcph->ack) {
/* release conntrack entry immediately*/
...

return -1;
}

Add some code in the do_masquerade() as follows:
if (!ct && iph->protocol == IPPROTO_ICMP) {
struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph+iph->ihl);
if (tcph->syn && tcph->ack) {
/* let it continue traversal */
return NF_ACCEPT;
}
}

The first change lets tcp_packet delete the conntrack entry when the syn
and ack flag of the packet are set up and it is in NONE state. The
second change lets this kind of packet continue traversal, so that it
can be caught by the ip_vs_out().

I haven't tested it, and don't know if it really works. I just came back
to my parent's home for the Spring Festival (Chinese lunar new year),
don't have hardware to test it now. Anyway, I will try to borrow some
machines to build a development environment at home (because I will stay
here for two weeks).

Thanks,

Wensong

> > Thanks for your reply.
> >
> > Pat
>
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>

Hello,

I will make some coding and testing this weekend too.

On Fri, 19 Jan 2001, Wensong Zhang wrote:

> > In your tests the SYN+ACK replies hit the ipchains masquerading
> > and the packet don't reach the LVS ip_vs_out() function. The netfilter
> > treats this as a new connection and masquerades the packets before
> > reaching LVS in the same chain.
> >
> > The other problem in ip_fw_compat_masq.c is that inet_select_addr
> > is still involved in the routing decisions. This is wrong. The result
> > is that always the primary address from the outgoing device is selected
> > as masquerading address instead of using rt->rt_src as maddr.
> >
> > Wensong! IMO, we should make some fixes to support the ipchains
> > compatibility mode.
> >
>
> Yeah, we should do it long time ago. But it is not easy to do a good fix
> on this, because conntrack tracks every incoming packet. :)
>
> Just thought a dirty fix to support this kind of compatibility. We can
> add several lines of checking code in the tcp_packet() of the
> ip_conntrack_proto_tcp.c as follows:
> if (oldtcpstate == TCP_CONNTRACK_NONE
> && tcph->syn && tcph->ack) {
> /* release conntrack entry immediately*/
> ...
>
> return -1;
> }

May be yes. May be we can't avoid the conntrack to be created
from the pre routing but we can try to release the netfilter
connection as soon as possible.

> Add some code in the do_masquerade() as follows:
> if (!ct && iph->protocol == IPPROTO_ICMP) {
> struct tcphdr *tcph = (struct tcphdr *)((u_int32_t *)iph+iph->ihl);
> if (tcph->syn && tcph->ack) {
> /* let it continue traversal */
> return NF_ACCEPT;
> }
> }
>
> The first change lets tcp_packet delete the conntrack entry when the syn
> and ack flag of the packet are set up and it is in NONE state. The
> second change lets this kind of packet continue traversal, so that it
> can be caught by the ip_vs_out().

Yes, we have two choices:

1. to add code in do_masquerade

2. to add code in the caller, i.e. in fw_in()

May be I prefer the second case.

In either case we will be called twice from FORWARD:

- fw_in (FORWARD:0)

if ipchains.o is inserted

- ip_vs_out (FORWARD:100)

in any case:
1. no ipchains and no iptable_nat/ip_conntrack
2. ipchains only
3. iptable_nat/ip_conntrack

So, may be we can return from fw_in with NF_STOLEN/NF_DROP
after masquerading the LVS related connections. The problem is
how to avoid the second lookup in FORWARD:100 when the connection
lookup fails in FORWARD:0. May be in ip_vs_out we will see that
the connection is created from the netfilter. So, may be we can
call ip_vs_out from many places but may be we will need to mark
the packets as already handled. May be will need some code in
ip_nat_fn for the POST_ROUTING. The PRE_ROUTING comes too in
ip_nat_fn but we can't do anything here. We can't waste CPU for
LVS before LOCAL_IN or FORWARD. So, if the iptable_nat and the
ip_conntrack code is inserted we will have ip_conntrack_in calls
and we have to release this structure as soon as possible if
the iptables rules don't like this packet and leave it to LVS.
The main goals is LVS to work with:

- firewall rules: with ipchains or iptables
- QoS
- masquerading: LVS-only (only for NAT-ed real servers), iptable_nat
rules, ipchains rules

- We must be before any netfilter nat code (mostly in the FORWARD chain).
- The packets after ip_route_output() not to be dropped in the
post_routing from ip_nat_fn

We must make all this working :)

> I haven't tested it, and don't know if it really works. I just came back
> to my parent's home for the Spring Festival (Chinese lunar new year),
> don't have hardware to test it now. Anyway, I will try to borrow some
> machines to build a development environment at home (because I will stay
> here for two weeks).

I don't like to patch but it looks as the only solution. Let's
make something working and then to optimize it with help from the
netfilter gurus.

> Thanks,
>
> Wensong


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:
>

>
> If you don't create any other outgoing masquerading connections
> from the internal servers you can just delete the ipchains rule. The LVS
> will maintain its connections without the help from ipchains. Even for
> the related ICMP messages. Use ipchains rules only for filtering in this
> case.

in 2.4.x VS-NAT, are all ports from the real-server masqueraded or only
the ports for the services that LVS is controlling?

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Thu, 1 Mar 2001, Joseph Mack wrote:

> > If you don't create any other outgoing masquerading connections
> > from the internal servers you can just delete the ipchains rule. The LVS
> > will maintain its connections without the help from ipchains. Even for
> > the related ICMP messages. Use ipchains rules only for filtering in this
> > case.
>
> in 2.4.x VS-NAT, are all ports from the real-server masqueraded or only
> the ports for the services that LVS is controlling?

LVS masquerades only its connections, even in 2.2.

> Joe
>
> --
> Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
> contractor to the National Environmental Supercomputer Center,
> mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:

> > in 2.4.x VS-NAT, are all ports from the real-server masqueraded or only
> > the ports for the services that LVS is controlling?
>
> LVS masquerades only its connections, even in 2.2.


In 2.2.x, I setup VS-NAT by running ipvsadm commands for a service
and then a complementary ipchains command like

ipchains -A forward -p tcp -j MASQ -s realserver_name service_name -d 0.0.0.0/0

to demasquerade the service.

Are you saying this wasn't neccessary in 2.2.x?

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Thu, 1 Mar 2001, Joseph Mack wrote:

> Julian Anastasov wrote:
>
> > > in 2.4.x VS-NAT, are all ports from the real-server masqueraded or only
> > > the ports for the services that LVS is controlling?
> >
> > LVS masquerades only its connections, even in 2.2.
>
>
> In 2.2.x, I setup VS-NAT by running ipvsadm commands for a service
> and then a complementary ipchains command like
>
> ipchains -A forward -p tcp -j MASQ -s realserver_name service_name -d 0.0.0.0/0
>
> to demasquerade the service.

The above rule is used to masquerade and not to demasquerade.
So, it is needed only when NAT-ed real servers are used. For DR you don't
need it.

> Are you saying this wasn't neccessary in 2.2.x?

It is neccessary in 2.2. But that does not mean LVS masquerades
other connections. The ipchains rule in 2.2 simply feeds the LVS and
the MASQ code with packets (it is in the FORWARD chain) while in 2.4
LVS hooks in the FORWARDing to check the packets and eventually to
masquerade them. The other packets are not masqueraded. For 2.4
netfilter NAT rules are needed to masquerade other connections,
not related to LVS.

> Joe
>
> --
> Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
> contractor to the National Environmental Supercomputer Center,
> mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA


Regards

--
Julian Anastasov <ja@ssi.bg>

Julian Anastasov wrote:
>
> Hello,
>
> On Thu, 1 Mar 2001, Joseph Mack wrote:
>
> > Julian Anastasov wrote:
> >
> > > > in 2.4.x VS-NAT, are all ports from the real-server masqueraded or only
> > > > the ports for the services that LVS is controlling?
> > >
> > > LVS masquerades only its connections, even in 2.2.
> >
> >
> > In 2.2.x, I setup VS-NAT by running ipvsadm commands for a service
> > and then a complementary ipchains command like
> >
> > ipchains -A forward -p tcp -j MASQ -s realserver_name service_name -d 0.0.0.0/0
> >
> > to demasquerade the service.
>
> The above rule is used to masquerade and not to demasquerade.
> So, it is needed only when NAT-ed real servers are used. For DR you don't
> need it.

yes understand

In 2.2.x if I was VS-NAT'ing http, then I would also run

Julian Anastasov wrote:
>
> Hello,
>
> On Thu, 1 Mar 2001, Joseph Mack wrote:
>
> > Julian Anastasov wrote:
> >
> > > > in 2.4.x VS-NAT, are all ports from the real-server masqueraded or only
> > > > the ports for the services that LVS is controlling?
> > >
> > > LVS masquerades only its connections, even in 2.2.
> >
> >
> > In 2.2.x, I setup VS-NAT by running ipvsadm commands for a service
> > and then a complementary ipchains command like
> >
> > ipchains -A forward -p tcp -j MASQ -s realserver_name service_name -d 0.0.0.0/0
> >
> > to demasquerade the service.
>
> The above rule is used to masquerade and not to demasquerade.
> So, it is needed only when NAT-ed real servers are used. For DR you don't
> need it.
>
> > Are you saying this wasn't neccessary in 2.2.x?
>
> It is neccessary in 2.2. But that does not mean LVS masquerades
> other connections. The ipchains rule in 2.2 simply feeds the LVS and
> the MASQ code with packets (it is in the FORWARD chain) while in 2.4
> LVS hooks in the FORWARDing to check the packets and eventually to
> masquerade them. The other packets are not masqueraded. For 2.4
> netfilter NAT rules are needed to masquerade other connections,
> not related to LVS.
>
> > Joe
> >
> > --
> > Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
> > contractor to the National Environmental Supercomputer Center,
> > mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
ipchains -A forward -p tcp -j MASQ -s realserver_name http -d 0.0.0.0/0

to masquerade the http replies.

With this arrangement, telnet initiated from the real-server would go
out without being masqueraded.

> > Are you saying this wasn't neccessary in 2.2.x?
>
> It is neccessary in 2.2. But that does not mean LVS masquerades
> other connections. The ipchains rule in 2.2 simply feeds the LVS and
> the MASQ code with packets (it is in the FORWARD chain) while in 2.4
> LVS hooks in the FORWARDing to check the packets and eventually to
> masquerade them. The other packets are not masqueraded.

Are you saying that in 2.4.x if I setup VS-NAT for http, then
http will be masqueraded from the real-server without having
to run ipchains commands, but that other services are not affected.
ie telnet from the real-server will go out without masquerading?

> For 2.4
> netfilter NAT rules are needed to masquerade other connections,
> not related to LVS.

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Thu, 1 Mar 2001, Joseph Mack wrote:

> > The above rule is used to masquerade and not to demasquerade.
> > So, it is needed only when NAT-ed real servers are used. For DR you don't
> > need it.
>
> yes understand
>
> In 2.2.x if I was VS-NAT'ing http, then I would also run
>
> ipchains -A forward -p tcp -j MASQ -s realserver_name http -d 0.0.0.0/0
>
> to masquerade the http replies.

Right.

> With this arrangement, telnet initiated from the real-server would go
> out without being masqueraded.

Yes.

> > > Are you saying this wasn't neccessary in 2.2.x?
> >
> > It is neccessary in 2.2. But that does not mean LVS masquerades
> > other connections. The ipchains rule in 2.2 simply feeds the LVS and
> > the MASQ code with packets (it is in the FORWARD chain) while in 2.4
> > LVS hooks in the FORWARDing to check the packets and eventually to
> > masquerade them. The other packets are not masqueraded.
>
> Are you saying that in 2.4.x if I setup VS-NAT for http, then
> http will be masqueraded from the real-server without having
> to run ipchains commands, but that other services are not affected.
> ie telnet from the real-server will go out without masquerading?

Nice feature :) LVS can work without ipchains -j MASQ or
iptables -t nat ... We need these NAT rules only to NAT another
traffic, in your case the telnet which is not a LVS service.

> Joe

Regards

--
Julian Anastasov <ja@ssi.bg>