Mailing List Archive: ip

ip_vs & NAT

Feb 24, 2001, 9:28 AM

Post #1 of 7 (981 views)

I have a few machines running apache & mysql behind a router running ipvs.

The router masquerades the connections, like so;

TCP 172.24.51.1:www lc
-> evilwillow.sunnydale.antefacto.com:www Masq 1 0 0
-> goodwillow.sunnydale.antefacto.com:www Masq 1 0 0
TCP 172.24.51.6:mysql lc
-> evilwillow.sunnydale.antefacto.com:mysql Masq 1 0 0
-> goodwillow.sunnydale.antefacto.com:mysql Masq 1 0 0

It works fine. External apps can get to these machines. However,
the router and the two machines above can't get to 172.24.51.6:mysql - the
connection hangs. Like wise for apache. The machines are all on a switch -
not a hub, if that matters.

I telnetted to 172.24.51.1:www from "evilwillow", and did a tcp dump on
the ipvs machine, and saw;

User level filter, protocol ALL, datagram packet socket
tcpdump: listening on all devices
16:25:20.567319 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: S [ECN-Echo,CWR] 3633840634:3633840634(0) win 5840 <mss 1460,sackOK,timestamp 17852294 0,nop,wscale 0> (DF)
16:25:20.567775 eth0 > evilwillow.sunnydale.antefacto.com.www > goodwillow.sunnydale.antefacto.com.1926: S [ECN-Echo] 3634986324:3634986324(0) ack 3633840635 win 5792 <mss 1460,sackOK,timestamp 17521934 17852294,nop,wscale 0> (DF)
16:25:20.567890 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: R 3633840635:3633840635(0) win 0 (DF)
16:25:23.564060 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: S [ECN-Echo,CWR] 3633840634:3633840634(0) win 5840 <mss 1460,sackOK,timestamp 17852594 0,nop,wscale 0> (DF)
16:25:23.564139 eth0 > evilwillow.sunnydale.antefacto.com.www > goodwillow.sunnydale.antefacto.com.1926: S [ECN-Echo] 3637982691:3637982691(0) ack 3633840635 win 5792 <mss 1460,sackOK,timestamp 17522234 17852594,nop,wscale 0> (DF)
16:25:23.564229 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: R 3633840635:3633840635(0) win 0 (DF)

That looks like the machines are talking. But, I keep getting "connection
refused". Is there something special you need to do when both machines
from outside a cluster and inside a cluster have to access HA services ?

Kate

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software

Re: ip_vs & NAT [ In reply to ]

ja at ssi

Feb 24, 2001, 6:03 PM

Post #2 of 7 (976 views)

Permalink

Hello,

On Sat, 24 Feb 2001, John P . Looney wrote:

> I have a few machines running apache & mysql behind a router running ipvs.
>
> The router masquerades the connections, like so;
>
> TCP 172.24.51.1:www lc
> -> evilwillow.sunnydale.antefacto.com:www Masq 1 0 0
> -> goodwillow.sunnydale.antefacto.com:www Masq 1 0 0
> TCP 172.24.51.6:mysql lc
> -> evilwillow.sunnydale.antefacto.com:mysql Masq 1 0 0
> -> goodwillow.sunnydale.antefacto.com:mysql Masq 1 0 0
>
> It works fine. External apps can get to these machines. However,
> the router and the two machines above can't get to 172.24.51.6:mysql - the
> connection hangs. Like wise for apache. The machines are all on a switch -
> not a hub, if that matters.

The clients can't run in the director. If they run in NAT-ed
real servers then they can't work. Only for DR and TUN method you
can run clients in the real servers but they connect to the local host
and not to the director.

> I telnetted to 172.24.51.1:www from "evilwillow", and did a tcp dump on
> the ipvs machine, and saw;
>
> User level filter, protocol ALL, datagram packet socket
> tcpdump: listening on all devices
> 16:25:20.567319 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: S [ECN-Echo,CWR] 3633840634:3633840634(0) win 5840 <mss 1460,sackOK,timestamp 17852294 0,nop,wscale 0> (DF)
> 16:25:20.567775 eth0 > evilwillow.sunnydale.antefacto.com.www > goodwillow.sunnydale.antefacto.com.1926: S [ECN-Echo] 3634986324:3634986324(0) ack 3633840635 win 5792 <mss 1460,sackOK,timestamp 17521934 17852294,nop,wscale 0> (DF)
> 16:25:20.567890 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: R 3633840635:3633840635(0) win 0 (DF)
> 16:25:23.564060 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: S [ECN-Echo,CWR] 3633840634:3633840634(0) win 5840 <mss 1460,sackOK,timestamp 17852594 0,nop,wscale 0> (DF)
> 16:25:23.564139 eth0 > evilwillow.sunnydale.antefacto.com.www > goodwillow.sunnydale.antefacto.com.1926: S [ECN-Echo] 3637982691:3637982691(0) ack 3633840635 win 5792 <mss 1460,sackOK,timestamp 17522234 17852594,nop,wscale 0> (DF)
> 16:25:23.564229 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: R 3633840635:3633840635(0) win 0 (DF)

Hm, I don't understand this output. What is the LVS version?

> That looks like the machines are talking. But, I keep getting "connection
> refused". Is there something special you need to do when both machines
> from outside a cluster and inside a cluster have to access HA services ?

To run direct routing instead of NAT.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: ip_vs & NAT [ In reply to ]

john at antefacto

Feb 25, 2001, 4:51 AM

Post #3 of 7 (976 views)

Permalink

On Sun, Feb 25, 2001 at 01:03:52AM +0000, Julian Anastasov mentioned:
> > It works fine. External apps can get to these machines. However,
> > the router and the two machines above can't get to 172.24.51.6:mysql - the
> > connection hangs. Like wise for apache. The machines are all on a switch -
> > not a hub, if that matters.
> The clients can't run in the director. If they run in NAT-ed
> real servers then they can't work. Only for DR and TUN method you
> can run clients in the real servers but they connect to the local host
> and not to the director.

So, I can't run monitoring software on the director machine, to connect
to the Virtual Services ?

> Hm, I don't understand this output. What is the LVS version?

Version 1.0.5.

> > That looks like the machines are talking. But, I keep getting "connection
> > refused". Is there something special you need to do when both machines
> > from outside a cluster and inside a cluster have to access HA services ?
> To run direct routing instead of NAT.

I tryed this;

I setup some new virtual IPs, and used directo routing. It didn't work
either - I saw the same problem, where the clients were sending out
traffic, and ipvsadm said that the connections were made, but "inactive".

Is there any sort of debug mode I can put ip_vs into, so it logs all
connection attempts ?

John

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software

Re: ip_vs & NAT [ In reply to ]

ja at ssi

Feb 25, 2001, 7:31 AM

Post #4 of 7 (973 views)

Permalink

Hello,

On Sun, 25 Feb 2001, John P . Looney wrote:

> On Sun, Feb 25, 2001 at 01:03:52AM +0000, Julian Anastasov mentioned:
> > > It works fine. External apps can get to these machines. However,
> > > the router and the two machines above can't get to 172.24.51.6:mysql - the
> > > connection hangs. Like wise for apache. The machines are all on a switch -
> > > not a hub, if that matters.
> > The clients can't run in the director. If they run in NAT-ed
> > real servers then they can't work. Only for DR and TUN method you
> > can run clients in the real servers but they connect to the local host
> > and not to the director.
>
> So, I can't run monitoring software on the director machine, to connect
> to the Virtual Services ?

Not possible. But if you use direct routing you can run the
client on another host that is not real server. This is for service
checks directly checking the virtual service. You still can run
service checks for NAT setups that connect to the real services, i.e.
to RIP:RPORT instead of VIP:VPORT. Virtual services with NAT-ed
real services are still possible to check when the client is on
different logical network and when the route to the internal
real server addresses points to the director (the client can't
connect directly to the real server for NAT setups, the director needs
to NAT the addresses in both directions). So, again not from real server.

> I tryed this;
>
> I setup some new virtual IPs, and used directo routing. It didn't work
> either - I saw the same problem, where the clients were sending out
> traffic, and ipvsadm said that the connections were made, but "inactive".

No, it is not working in theory.

> Is there any sort of debug mode I can put ip_vs into, so it logs all
> connection attempts ?

The syslog debugging is enabled setting 111 to
/proc/sys/net/ipv4/vs/debug_level when CONFIG_IP_VS_DEBUG is defined.

> John

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: ip_vs & NAT [ In reply to ]

ratz at tac

Feb 26, 2001, 1:37 AM

Post #5 of 7 (975 views)

Permalink

> > Is there any sort of debug mode I can put ip_vs into, so it logs all
> > connection attempts ?
>
> The syslog debugging is enabled setting 111 to
> /proc/sys/net/ipv4/vs/debug_level when CONFIG_IP_VS_DEBUG is defined.

:) Julian, you're a nasty boy !!! I vote for 666 to enable intesive logging.

ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`

Re: ip_vs & NAT [ In reply to ]

ratz at tac

Feb 26, 2001, 1:52 AM

Post #6 of 7 (979 views)

Permalink

Hi,

Just some more comments, although Julian already said it.

"John P . Looney" wrote:
>
> I have a few machines running apache & mysql behind a router running ipvs.
>
> The router masquerades the connections, like so;
>
> TCP 172.24.51.1:www lc
> -> evilwillow.sunnydale.antefacto.com:www Masq 1 0 0
> -> goodwillow.sunnydale.antefacto.com:www Masq 1 0 0
> TCP 172.24.51.6:mysql lc
> -> evilwillow.sunnydale.antefacto.com:mysql Masq 1 0 0
> -> goodwillow.sunnydale.antefacto.com:mysql Masq 1 0 0
>
> It works fine. External apps can get to these machines. However,
> the router and the two machines above can't get to 172.24.51.6:mysql - the
> connection hangs. Like wise for apache. The machines are all on a switch -
> not a hub, if that matters.
>
> I telnetted to 172.24.51.1:www from "evilwillow", and did a tcp dump on
> the ipvs machine, and saw;

Could you please next time send such outputs in readable form, e.g.
numeric? ipvsadm -L -n and tcpdump -n are much more preferred then
if I first have to run the mail through a sed to get it in a readable
form. And if possible also all versions and configs (kernel, ipvsadm,
forwarding method)

As you can see, the counter in the active_conns is 1 and I bet it would
stay there for quite a long time in your tests. This is typical for:

o not handling the arp-problem properly
o trying to connect to the VIP from one of the nodes inside the LVS-cluster

You have to accept that due to the policy the packets get forwarded
in a LVS-cluster you cannot, never ever, connect to the VIP from a
node inside a LVS-cluster (normally defined as RS and LB itself).
This is in the HOWTO but I can't find it right now ;)

> User level filter, protocol ALL, datagram packet socket
> tcpdump: listening on all devices

Wow, did you compile the new tcpdump? This must be a new feature,
listening on all devices.

> 16:25:20.567319 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: S [ECN-Echo,CWR] 3633840634:3633840634(0) win 5840 <mss

Hmmm, you stated that you're using LVS-NAT with LVS patch 1.0.5.
How the hell does this ECN flag come into your stream ?? Are your
RS running kernel 2.4.x?

1460,sackOK,timestamp 17852294 0,nop,wscale 0> (DF)
> 16:25:20.567775 eth0 > evilwillow.sunnydale.antefacto.com.www > goodwillow.sunnydale.antefacto.com.1926: S [ECN-Echo] 3634986324:3634986324(0) ack 3633840635 win 5792 <mss 1460,sackOK,timestamp 17521934 17852294,nop,wscale 0> (DF)
> 16:25:20.567890 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: R 3633840635:3633840635(0) win 0 (DF)

Oh yes, this RST says it all: you're connecting to the VIP fom
a node inside the LVS-cluster.

> 16:25:23.564060 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: S [ECN-Echo,CWR] 3633840634:3633840634(0) win 5840 <mss 1460,sackOK,timestamp 17852594 0,nop,wscale 0> (DF)

Oups, we're going back in time? Is this a feature of the new
tcpdump listening on all interfaces?

> 16:25:23.564139 eth0 > evilwillow.sunnydale.antefacto.com.www > goodwillow.sunnydale.antefacto.com.1926: S [ECN-Echo] 3637982691:3637982691(0) ack 3633840635 win 5792 <mss 1460,sackOK,timestamp 17522234 17852594,nop,wscale 0> (DF)
> 16:25:23.564229 eth0 < goodwillow.sunnydale.antefacto.com.1926 > evilwillow.sunnydale.antefacto.com.www: R 3633840635:3633840635(0) win 0 (DF)
>
> That looks like the machines are talking. But, I keep getting "connection
> refused". Is there something special you need to do when both machines

Yes, this is because of the RST and this is because the packet look
forged.

> from outside a cluster and inside a cluster have to access HA services ?

Yes, don't do it from inside ;)

Best regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`

Re: ip_vs & NAT [ In reply to ]

john at antefacto

Feb 26, 2001, 2:26 AM

Post #7 of 7 (975 views)

Permalink

On Mon, Feb 26, 2001 at 09:52:04AM +0100, Roberto Nibali mentioned:
> Could you please next time send such outputs in readable form, e.g.
> numeric? ipvsadm -L -n and tcpdump -n are much more preferred then
> if I first have to run the mail through a sed to get it in a readable
> form. And if possible also all versions and configs (kernel, ipvsadm,
> forwarding method)

Will do.

> As you can see, the counter in the active_conns is 1 and I bet it would
> stay there for quite a long time in your tests. This is typical for:
>
> o not handling the arp-problem properly
> o trying to connect to the VIP from one of the nodes inside the LVS-cluster
>
> You have to accept that due to the policy the packets get forwarded
> in a LVS-cluster you cannot, never ever, connect to the VIP from a
> node inside a LVS-cluster (normally defined as RS and LB itself).
> This is in the HOWTO but I can't find it right now ;)

Smeg. And there is no way around this ? I can't setup separate IPs & the
like on the LB, and use direct routing for those, and masq for IP's used
outside the cluster ?

I'm really really curious why this doesn't work. So curious, that if
someone could give me a good reason, I'd devote a lot of time to making it
work, if possible.

Kate

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software