Mailing List Archive: [lvs-users] Sorry, it's pretty unusable!

[lvs-users] Sorry, it's pretty unusable!

Oct 16, 2013, 11:48 PM

Post #1 of 5 (1249 views)

Hi!

I'm not subscribed to the list, so I hope someone will receive it anyway:

I could pretty well use LVS for a load-balance, high-availability scenario like distributing SMTP requests to different servers, but the setup seems so complicated that I won't do. Reading the documentation, I felt that the NAT (masq) mechanism would be the most elegant for my requirements. However as it tuned out it did not work (as for many others). The reason is simple: LVS rewrites the destination TSAP (IP address and port), but it leaves the source TSAP unchanged. So any replies from a real server go to the original sender, instead of the LVS host.

The proposed solution is to set the LVS host as default gateway on any real server. This has several problems:
1) You create a SPoF on the LVS host
2) You create a network bottleneck on the LVS host (_all_ traffic from a real goes to the LVS host which must be a router)
3) If LVS host and real server are not in the same subnet, you cannot route from the real server to the LVS directly
4) You cannot have two different LVS hosts that use different services on the same real host

I reall wonder why you don't rewrite the source TSAP (in addition to the destination TSAP) as well so that the sender of the packet seems to be the LVS host. On a second rewrite the LVS destination TSAP would be rewritten to the original requester. I feel this would work like a charm:
1) The real server will reply to the LVS host automatically
2) Only LVS traffic needs to go through LVS host
3) LVS host does not need to be a router (after rewriting the destination, I think)
4) LVS host and real server can be in different subnets
5) You can use one real server from different LVS hosts

Did I overlook something that makes this impossible or impractical?

Regards,
Ulrich

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Sorry, it's pretty unusable! [ In reply to ]

graeme at graemef

Oct 17, 2013, 5:51 AM

Post #2 of 5 (1215 views)

Permalink

Hi

On 17 Oct 2013, at 07:48, "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de> wrote:
> I'm not subscribed to the list, so I hope someone will receive it anyway:

Yes we did, but if people reply to the list then you won't see it unless you're watching an archiver somewhere...

> I could pretty well use LVS for a load-balance, high-availability scenario like distributing SMTP requests to different servers, but the setup seems so complicated that I won't do. Reading the documentation, I felt that the NAT (masq) mechanism would be the most elegant for my requirements. However as it tuned out it did not work (as for many others). The reason is simple: LVS rewrites the destination TSAP (IP address and port), but it leaves the source TSAP unchanged. So any replies from a real server go to the original sender, instead of the LVS host

That's right. In NAT mode, the realservers don't talk directly to anything but the director.

> The proposed solution is to set the LVS host as default gateway on any real server. This has several problems:
> 1) You create a SPoF on the LVS host
> 2) You create a network bottleneck on the LVS host (_all_ traffic from a real goes to the LVS host which must be a router)
> 3) If LVS host and real server are not in the same subnet, you cannot route from the real server to the LVS directly
> 4) You cannot have two different LVS hosts that use different services on the same real host

That's NAT mode for you.

> I reall wonder why you don't rewrite the source TSAP (in addition to the destination TSAP) as well so that the sender of the packet seems to be the LVS host. On a second rewrite the LVS destination TSAP would be rewritten to the original requester. I feel this would work like a charm:
> 1) The real server will reply to the LVS host automatically
> 2) Only LVS traffic needs to go through LVS host
> 3) LVS host does not need to be a router (after rewriting the destination, I think)
> 4) LVS host and real server can be in different subnets
> 5) You can use one real server from different LVS hosts
>
> Did I overlook something that makes this impossible or impractical?

Yes. You've sort of described both DR and TUN modes here, except for the source IP being rewritten. LVS/IPVS is *not a proxy*, it's a fancy router. If you want to do this with source rewriting, use a system such as haproxy.

NAT mode is most useful where the realservers don't require any special configuration apart from their default gateway.

DR and TUN modes require extra configuration on the realservers, but do away with the SPOF and bottleneck in the director.

Graeme
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Sorry, it's pretty unusable! [ In reply to ]

michael at schwartzkopff

Oct 17, 2013, 6:21 AM

Post #3 of 5 (1216 views)

Permalink

Am Donnerstag, 17. Oktober 2013, 08:48:23 schrieb Ulrich Windl:
> Hi!
>
> I'm not subscribed to the list, so I hope someone will receive it anyway:
>
> I could pretty well use LVS for a load-balance, high-availability scenario
> like distributing SMTP requests to different servers, but the setup seems
> so complicated that I won't do.

OK. <Dear reader: please insert your own thoughts here>

> Reading the documentation, I felt that the
> NAT (masq) mechanism would be the most elegant for my requirements. However
> as it tuned out it did not work (as for many others).

How did you get the implression that it does not work for many others?

> The reason is simple:

The reason, it does not work for _you_.

> LVS rewrites the destination TSAP (IP address and port), but it leaves the
> source TSAP unchanged. So any replies from a real server go to the original
> sender, instead of the LVS host.

Yes. This behaviour is well documented.

> The proposed solution is to set the LVS host as default gateway on any real
> server. This has several problems: 1) You create a SPoF on the LVS host
> 2) You create a network bottleneck on the LVS host (_all_ traffic from a
> real goes to the LVS host which must be a router) 3) If LVS host and real
> server are not in the same subnet, you cannot route from the real server to
> the LVS directly 4) You cannot have two different LVS hosts that use
> different services on the same real host

You hit the nail on the head.

> I reall wonder why you don't rewrite the source TSAP (in addition to the
> destination TSAP) as well so that the sender of the packet seems to be the
> LVS host.

This feature is well documented here:

http://blog.loadbalancer.org/enabling-snat-in-lvs-xt_ipvs-and-iptables/

I think this is included already in later kernel versions. But google a little
bit.

> On a second rewrite the LVS destination TSAP would be rewritten
> to the original requester. I feel this would work like a charm: 1) The real
> server will reply to the LVS host automatically
> 2) Only LVS traffic needs to go through LVS host
> 3) LVS host does not need to be a router (after rewriting the destination, I
> think) 4) LVS host and real server can be in different subnets
> 5) You can use one real server from different LVS hosts
>
> Did I overlook something that makes this impossible or impractical?

No. That is why people implemented it already.

--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0162) 1650044
Fax: (089) 620 304 13

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Sorry, it's pretty unusable! [ In reply to ]

tom.van.leeuwen at saasplaza

Oct 17, 2013, 6:22 AM

Post #4 of 5 (1223 views)

Permalink

So he may not be receiving your reply.

Ulrich if you want source nat on your LVS box, you'll have to do:

1) Create a loadbalancer:
ipvsadm -A -t 192.168.200.100:443
# add real servers

2) Create the source nat rule for this loadbalancer:
iptables -t nat -A POSTROUTING -m ipvs --vaddr 192.168.200.100/32
--vport 443 -j MASQUERADE

3) Set sysctl setting to allow conntrack on vs
sysctl net.ipv4.vs.conntrack=1

So it seems what you want is possible and not difficult at all...

Regards,
Tom

On 10/17/2013 02:51 PM, Graeme Fowler wrote:
> Hi
>
> On 17 Oct 2013, at 07:48, "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de> wrote:
>> I'm not subscribed to the list, so I hope someone will receive it anyway:
> Yes we did, but if people reply to the list then you won't see it unless you're watching an archiver somewhere...
>
>> I could pretty well use LVS for a load-balance, high-availability scenario like distributing SMTP requests to different servers, but the setup seems so complicated that I won't do. Reading the documentation, I felt that the NAT (masq) mechanism would be the most elegant for my requirements. However as it tuned out it did not work (as for many others). The reason is simple: LVS rewrites the destination TSAP (IP address and port), but it leaves the source TSAP unchanged. So any replies from a real server go to the original sender, instead of the LVS host
> That's right. In NAT mode, the realservers don't talk directly to anything but the director.
>
>> The proposed solution is to set the LVS host as default gateway on any real server. This has several problems:
>> 1) You create a SPoF on the LVS host
>> 2) You create a network bottleneck on the LVS host (_all_ traffic from a real goes to the LVS host which must be a router)
>> 3) If LVS host and real server are not in the same subnet, you cannot route from the real server to the LVS directly
>> 4) You cannot have two different LVS hosts that use different services on the same real host
> That's NAT mode for you.
>
>> I reall wonder why you don't rewrite the source TSAP (in addition to the destination TSAP) as well so that the sender of the packet seems to be the LVS host. On a second rewrite the LVS destination TSAP would be rewritten to the original requester. I feel this would work like a charm:
>> 1) The real server will reply to the LVS host automatically
>> 2) Only LVS traffic needs to go through LVS host
>> 3) LVS host does not need to be a router (after rewriting the destination, I think)
>> 4) LVS host and real server can be in different subnets
>> 5) You can use one real server from different LVS hosts
>>
>> Did I overlook something that makes this impossible or impractical?
> Yes. You've sort of described both DR and TUN modes here, except for the source IP being rewritten. LVS/IPVS is *not a proxy*, it's a fancy router. If you want to do this with source rewriting, use a system such as haproxy.
>
> NAT mode is most useful where the realservers don't require any special configuration apart from their default gateway.
>
> DR and TUN modes require extra configuration on the realservers, but do away with the SPOF and bottleneck in the director.
>
> Graeme
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] Sorry, it's pretty unusable! [ In reply to ]

dennisml at conversis

Oct 17, 2013, 9:33 AM

Post #5 of 5 (1218 views)

Permalink

On 17.10.2013 14:51, Graeme Fowler wrote:
> Hi
>
> On 17 Oct 2013, at 07:48, "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de> wrote:
>> I'm not subscribed to the list, so I hope someone will receive it anyway:
>
> Yes we did, but if people reply to the list then you won't see it unless you're watching an archiver somewhere...
>
>> I could pretty well use LVS for a load-balance, high-availability scenario like distributing SMTP requests to different servers, but the setup seems so complicated that I won't do. Reading the documentation, I felt that the NAT (masq) mechanism would be the most elegant for my requirements. However as it tuned out it did not work (as for many others). The reason is simple: LVS rewrites the destination TSAP (IP address and port), but it leaves the source TSAP unchanged. So any replies from a real server go to the original sender, instead of the LVS host
>
> That's right. In NAT mode, the realservers don't talk directly to anything but the director.
>
>> The proposed solution is to set the LVS host as default gateway on any real server. This has several problems:
>> 1) You create a SPoF on the LVS host
>> 2) You create a network bottleneck on the LVS host (_all_ traffic from a real goes to the LVS host which must be a router)
>> 3) If LVS host and real server are not in the same subnet, you cannot route from the real server to the LVS directly
>> 4) You cannot have two different LVS hosts that use different services on the same real host
>
> That's NAT mode for you.
>
>> I reall wonder why you don't rewrite the source TSAP (in addition to the destination TSAP) as well so that the sender of the packet seems to be the LVS host. On a second rewrite the LVS destination TSAP would be rewritten to the original requester. I feel this would work like a charm:
>> 1) The real server will reply to the LVS host automatically
>> 2) Only LVS traffic needs to go through LVS host
>> 3) LVS host does not need to be a router (after rewriting the destination, I think)
>> 4) LVS host and real server can be in different subnets
>> 5) You can use one real server from different LVS hosts
>>
>> Did I overlook something that makes this impossible or impractical?
>
> Yes. You've sort of described both DR and TUN modes here, except for the source IP being rewritten. LVS/IPVS is *not a proxy*, it's a fancy router. If you want to do this with source rewriting, use a system such as haproxy.
>
> NAT mode is most useful where the realservers don't require any special configuration apart from their default gateway.
>
> DR and TUN modes require extra configuration on the realservers, but do away with the SPOF and bottleneck in the director.

I'm not sure I understand how DR does away with the SPOF in a way that
NAT doesn't. AFAICT the both behave the same way and require to be made
redundant to not be a SPOF no?

With overhead are you referring to the NAT translation bit or something
else.

Also nowadays it is possible to do DR without the need to have the
director run as a separate machine i.e. on the default gateway which
sounds like a fairly compact and efficient setup. Are there any
downsides to this other that that you still have to modify the
realservers slightly?

Regards,
Dennis

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users