Mailing List Archive

[lvs-users] lvs. Nat
In nat mode,does director do dnat and snat packets for all request packets?


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
On 29.04.2014, net.study.sea@gmail.com wrote:
> In nat mode,does director do dnat and snat packets for all request packets?

In NAT mode, the director does perform destination nat on request packets,
so your realserver does still see the correct "source" ip address.

However, replies then need to be sent via the director as well, so the
director can reverse the changed destination IP address and the client
does see the reply packet from the "expected" IP address/port.


Anders
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?

ÔÚ 2014-4-29£¬22:43£¬Anders Henke <anders.henke@1und1.de> дµÀ£º

> On 29.04.2014, net.study.sea@gmail.com wrote:
>> In nat mode,does director do dnat and snat packets for all request packets?
>
> In NAT mode, the director does perform destination nat on request packets,
> so your realserver does still see the correct "source" ip address.
>
> However, replies then need to be sent via the director as well, so the
> director can reverse the changed destination IP address and the client
> does see the reply packet from the "expected" IP address/port.
>
>
> Anders
> --
> 1&1 Internet AG Expert Systems Architect (IT Operations)
> Brauerstrasse 50 v://49.721.91374.0
> D-76135 Karlsruhe f://49.721.91374.225
>
> Amtsgericht Montabaur HRB 6484
> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
> Jan Oetjen, Christian W¨¹rst
> Aufsichtsratsvorsitzender: Michael Scheeren
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
There are multiple issues with this.
-IP-address changes
-possible port changes


The client initiates a connection from Client-IP A to Director-VIP B:

A->B, using a random source port number to destination port 80.

The director rewrites the IP packet as if it had been sent
from Client A to Realserver C and sends it to the realserver:
A->C, using the client's source port number to destination port 80.

If the realserver would reply to those director-NATed packet,
the reply would look like this:

C->A, from port 80 to client's source port

The client in turn knows it did create a connection from A to B,
but doesn't know about a connection from A to C and so can't
match those reply packets with "the other" connection. Well,
their destination port is known to be in use with a different connection,
so why should A assume this to be valid?

The reply packets are being dropped as being invalid.



Depending on your configuration, your director may also change
the destination port, e.g. from 80 to 8080:
A->C, using the client's source port number to destination port 8080

... and so the realserver's reply would look like this:

C->A, from port 8080 to client's source port

The client in turn knows it did create a connection from A to B.
Any replies from C won't match that connection and so are being dropped.
Neither IP address nor port number do even closely match a known connection,
there's no way for the client to match them.

In both cases, any firewall protecting A will perform similar
checks and drop the reply packets, as they are not related to a known
outgoing connection.

By routing and "un-NAT'ing" the reply packet via the director, the
director will rewrite the packet from 'C->A' to 'B->A' and rewrite
any port changes as well to meet the expectations of A (or any
firewall in between director and A).


In Direct Routing mode, the IP packet isn't changed at all, just
the ethernet destination MAC address ("outside" of the IP packet)
is changed. In Tunneling mode, the incoming IP packet is encapsulated and
isn't changed as well.

That's why in Direct Routing and Tunneling mode, any replies will be sent
bypassing your director and your director will only see "incoming" packets.

And that's also the reason why in NAT mode, both incoming and outgoing
bandwith are limited by the director, while in DR/TUN mode, the director
only limits the incoming bandwith. Beside this, the extra complexity of
performing NAT for incoming and outgoing packets does also affect the
overall performance of the director.


Best,

Anders



On 30.04.2014, net.study.sea@gmail.com wrote:
> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
>
> 在 2014-4-29,22:43,Anders Henke <anders.henke@1und1.de> 写道:
>
> > On 29.04.2014, net.study.sea@gmail.com wrote:
> >> In nat mode,does director do dnat and snat packets for all request packets?
> >
> > In NAT mode, the director does perform destination nat on request packets,
> > so your realserver does still see the correct "source" ip address.
> >
> > However, replies then need to be sent via the director as well, so the
> > director can reverse the changed destination IP address and the client
> > does see the reply packet from the "expected" IP address/port.
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
ÔÚ 2014-4-30£¬19:56£¬Anders Henke <anders.henke@1und1.de> дµÀ£º

> There are multiple issues with this.
> -IP-address changes
> -possible port changes
>
>
> The client initiates a connection from Client-IP A to Director-VIP B:
>
> A->B, using a random source port number to destination port 80.
>
> The director rewrites the IP packet as if it had been sent
> from Client A to Realserver C and sends it to the realserver:
> A->C, using the client's source port number to destination port 80.
>
> If the realserver would reply to those director-NATed packet,
> the reply would look like this:
>
> C->A, from port 80 to client's source port

At this time,if C could reply directly to A
,not through director ,then what is the result?


After I configure well Lvs in NAT mode,
I find it deal with client's request so slowly, can not find the reason.

> The client in turn knows it did create a connection from A to B,
> but doesn't know about a connection from A to C and so can't
> match those reply packets with "the other" connection. Well,
> their destination port is known to be in use with a different connection,
> so why should A assume this to be valid?
>
> The reply packets are being dropped as being invalid.
>
>
>
> Depending on your configuration, your director may also change
> the destination port, e.g. from 80 to 8080:
> A->C, using the client's source port number to destination port 8080
>
> ... and so the realserver's reply would look like this:
>
> C->A, from port 8080 to client's source port
>
> The client in turn knows it did create a connection from A to B.
> Any replies from C won't match that connection and so are being dropped.
> Neither IP address nor port number do even closely match a known connection,
> there's no way for the client to match them.
>
> In both cases, any firewall protecting A will perform similar
> checks and drop the reply packets, as they are not related to a known
> outgoing connection.
>
> By routing and "un-NAT'ing" the reply packet via the director, the
> director will rewrite the packet from 'C->A' to 'B->A' and rewrite
> any port changes as well to meet the expectations of A (or any
> firewall in between director and A).
>
>
> In Direct Routing mode, the IP packet isn't changed at all, just
> the ethernet destination MAC address ("outside" of the IP packet)
> is changed. In Tunneling mode, the incoming IP packet is encapsulated and
> isn't changed as well.
>
> That's why in Direct Routing and Tunneling mode, any replies will be sent
> bypassing your director and your director will only see "incoming" packets.
>
> And that's also the reason why in NAT mode, both incoming and outgoing
> bandwith are limited by the director, while in DR/TUN mode, the director
> only limits the incoming bandwith. Beside this, the extra complexity of
> performing NAT for incoming and outgoing packets does also affect the
> overall performance of the director.
>
>
> Best,
>
> Anders
>
>
>
> On 30.04.2014, net.study.sea@gmail.com wrote:
>> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
>>
>> ÔÚ 2014-4-29£¬22:43£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>
>>> On 29.04.2014, net.study.sea@gmail.com wrote:
>>>> In nat mode,does director do dnat and snat packets for all request packets?
>>>
>>> In NAT mode, the director does perform destination nat on request packets,
>>> so your realserver does still see the correct "source" ip address.
>>>
>>> However, replies then need to be sent via the director as well, so the
>>> director can reverse the changed destination IP address and the client
>>> does see the reply packet from the "expected" IP address/port.
> --
> 1&1 Internet AG Expert Systems Architect (IT Operations)
> Brauerstrasse 50 v://49.721.91374.0
> D-76135 Karlsruhe f://49.721.91374.225
>
> Amtsgericht Montabaur HRB 6484
> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
> Jan Oetjen, Christian W¨¹rst
> Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
On 30.04.2014, net.study.sea@gmail.com wrote:
> 在 2014-4-30,19:56,Anders Henke <anders.henke@1und1.de> 写道:
>
> > There are multiple issues with this.
> > -IP-address changes
> > -possible port changes
> >
> >
> > The client initiates a connection from Client-IP A to Director-VIP B:
> >
> > A->B, using a random source port number to destination port 80.
> >
> > The director rewrites the IP packet as if it had been sent
> > from Client A to Realserver C and sends it to the realserver:
> > A->C, using the client's source port number to destination port 80.
> >
> > If the realserver would reply to those director-NATed packet,
> > the reply would look like this:
> >
> > C->A, from port 80 to client's source port
>
> At this time,if C could reply directly to A
> ,not through director ,then what is the result?
>
>
> After I configure well Lvs in NAT mode,
> I find it deal with client's request so slowly, can not find the reason.

If you're balancing only a single tcp port, the director will reply
to any other tcp ports and protocols.

So "ping" (icmp) will be answered from the director with the
correct IP address, and the attempt to access a port without
a listener on the director will result in "connection refused".

When you're accessing the balanced tcp port, the realserver will reply
from an unexpected IP address and the client will drop the packet as
being invalid.

After some timeout, the client will retry sending the packet,
which will also fail. So it's not "slow" for your client to access the balanced service, it's "impossible" to establish a working connection.


If you do want to bypass the director, ask yourself why you'd like to do so:
- you're expecting more outgoing bandwith than the director will
be able to handle: you need to use either direct routing or tunneling.

- you're not comfortable with the idea of making your director also
your host's default gateway.

By choosing tunneling or direct routing, you'll gain that possibility,
but maybe you don't want to care about the ARP problem (a minor
configuration on every realserver - if you've forgotten one,
that realserver may attract all traffic to be balanced) or
about monitoring service X at IP address Y (the balanced IP
is not directly accessible for monitoring, so all checks need
to be performed at a different IP).

If it's just a mindset issue - you may see it that way: if your director
is not available, your service is not available and it doesn't matter
that you can't access your hosts. In order to re-establish service,
you need to make your director available anyway, and after you've done so,
you may need to take care about your hosts.




Anders


> > The client in turn knows it did create a connection from A to B,
> > but doesn't know about a connection from A to C and so can't
> > match those reply packets with "the other" connection. Well,
> > their destination port is known to be in use with a different connection,
> > so why should A assume this to be valid?
> >
> > The reply packets are being dropped as being invalid.
> >
> >
> >
> > Depending on your configuration, your director may also change
> > the destination port, e.g. from 80 to 8080:
> > A->C, using the client's source port number to destination port 8080
> >
> > ... and so the realserver's reply would look like this:
> >
> > C->A, from port 8080 to client's source port
> >
> > The client in turn knows it did create a connection from A to B.
> > Any replies from C won't match that connection and so are being dropped.
> > Neither IP address nor port number do even closely match a known connection,
> > there's no way for the client to match them.
> >
> > In both cases, any firewall protecting A will perform similar
> > checks and drop the reply packets, as they are not related to a known
> > outgoing connection.
> >
> > By routing and "un-NAT'ing" the reply packet via the director, the
> > director will rewrite the packet from 'C->A' to 'B->A' and rewrite
> > any port changes as well to meet the expectations of A (or any
> > firewall in between director and A).
> >
> >
> > In Direct Routing mode, the IP packet isn't changed at all, just
> > the ethernet destination MAC address ("outside" of the IP packet)
> > is changed. In Tunneling mode, the incoming IP packet is encapsulated and
> > isn't changed as well.
> >
> > That's why in Direct Routing and Tunneling mode, any replies will be sent
> > bypassing your director and your director will only see "incoming" packets.
> >
> > And that's also the reason why in NAT mode, both incoming and outgoing
> > bandwith are limited by the director, while in DR/TUN mode, the director
> > only limits the incoming bandwith. Beside this, the extra complexity of
> > performing NAT for incoming and outgoing packets does also affect the
> > overall performance of the director.
> >
> >
> > Best,
> >
> > Anders
> >
> >
> >
> > On 30.04.2014, net.study.sea@gmail.com wrote:
> >> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
> >>
> >> 在 2014-4-29,22:43,Anders Henke <anders.henke@1und1.de> 写道:
> >>
> >>> On 29.04.2014, net.study.sea@gmail.com wrote:
> >>>> In nat mode,does director do dnat and snat packets for all request packets?
> >>>
> >>> In NAT mode, the director does perform destination nat on request packets,
> >>> so your realserver does still see the correct "source" ip address.
> >>>
> >>> However, replies then need to be sent via the director as well, so the
> >>> director can reverse the changed destination IP address and the client
> >>> does see the reply packet from the "expected" IP address/port.
> > --
> > 1&1 Internet AG Expert Systems Architect (IT Operations)
> > Brauerstrasse 50 v://49.721.91374.0
> > D-76135 Karlsruhe f://49.721.91374.225
> >
> > Amtsgericht Montabaur HRB 6484
> > Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
> > Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
> > Jan Oetjen, Christian Würst
> > Aufsichtsratsvorsitzender: Michael Scheeren
--
1&1 Internet AG Expert Systems Architect (IT Operations)
Brauerstrasse 50 v://49.721.91374.0
D-76135 Karlsruhe f://49.721.91374.225

Amtsgericht Montabaur HRB 6484
Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
Jan Oetjen, Christian Würst
Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
Thanks for reply!

I have configured the reply going through director ,but the reply is very slowly.

ÔÚ 2014-4-30£¬22:36£¬Anders Henke <anders.henke@1und1.de> дµÀ£º

> On 30.04.2014, net.study.sea@gmail.com wrote:
>> ÔÚ 2014-4-30£¬19:56£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>
>>> There are multiple issues with this.
>>> -IP-address changes
>>> -possible port changes
>>>
>>>
>>> The client initiates a connection from Client-IP A to Director-VIP B:
>>>
>>> A->B, using a random source port number to destination port 80.
>>>
>>> The director rewrites the IP packet as if it had been sent
>>> from Client A to Realserver C and sends it to the realserver:
>>> A->C, using the client's source port number to destination port 80.
>>>
>>> If the realserver would reply to those director-NATed packet,
>>> the reply would look like this:
>>>
>>> C->A, from port 80 to client's source port
>>
>> At this time,if C could reply directly to A
>> ,not through director ,then what is the result?
>>
>>
>> After I configure well Lvs in NAT mode,
>> I find it deal with client's request so slowly, can not find the reason.
>
> If you're balancing only a single tcp port, the director will reply
> to any other tcp ports and protocols.
>
> So "ping" (icmp) will be answered from the director with the
> correct IP address, and the attempt to access a port without
> a listener on the director will result in "connection refused".
>
> When you're accessing the balanced tcp port, the realserver will reply
> from an unexpected IP address and the client will drop the packet as
> being invalid.
>
> After some timeout, the client will retry sending the packet,
> which will also fail. So it's not "slow" for your client to access the balanced service, it's "impossible" to establish a working connection.
>
>
> If you do want to bypass the director, ask yourself why you'd like to do so:
> - you're expecting more outgoing bandwith than the director will
> be able to handle: you need to use either direct routing or tunneling.
>
> - you're not comfortable with the idea of making your director also
> your host's default gateway.
>
> By choosing tunneling or direct routing, you'll gain that possibility,
> but maybe you don't want to care about the ARP problem (a minor
> configuration on every realserver - if you've forgotten one,
> that realserver may attract all traffic to be balanced) or
> about monitoring service X at IP address Y (the balanced IP
> is not directly accessible for monitoring, so all checks need
> to be performed at a different IP).
>
> If it's just a mindset issue - you may see it that way: if your director
> is not available, your service is not available and it doesn't matter
> that you can't access your hosts. In order to re-establish service,
> you need to make your director available anyway, and after you've done so,
> you may need to take care about your hosts.
>
>
>
>
> Anders
>
>
>>> The client in turn knows it did create a connection from A to B,
>>> but doesn't know about a connection from A to C and so can't
>>> match those reply packets with "the other" connection. Well,
>>> their destination port is known to be in use with a different connection,
>>> so why should A assume this to be valid?
>>>
>>> The reply packets are being dropped as being invalid.
>>>
>>>
>>>
>>> Depending on your configuration, your director may also change
>>> the destination port, e.g. from 80 to 8080:
>>> A->C, using the client's source port number to destination port 8080
>>>
>>> ... and so the realserver's reply would look like this:
>>>
>>> C->A, from port 8080 to client's source port
>>>
>>> The client in turn knows it did create a connection from A to B.
>>> Any replies from C won't match that connection and so are being dropped.
>>> Neither IP address nor port number do even closely match a known connection,
>>> there's no way for the client to match them.
>>>
>>> In both cases, any firewall protecting A will perform similar
>>> checks and drop the reply packets, as they are not related to a known
>>> outgoing connection.
>>>
>>> By routing and "un-NAT'ing" the reply packet via the director, the
>>> director will rewrite the packet from 'C->A' to 'B->A' and rewrite
>>> any port changes as well to meet the expectations of A (or any
>>> firewall in between director and A).
>>>
>>>
>>> In Direct Routing mode, the IP packet isn't changed at all, just
>>> the ethernet destination MAC address ("outside" of the IP packet)
>>> is changed. In Tunneling mode, the incoming IP packet is encapsulated and
>>> isn't changed as well.
>>>
>>> That's why in Direct Routing and Tunneling mode, any replies will be sent
>>> bypassing your director and your director will only see "incoming" packets.
>>>
>>> And that's also the reason why in NAT mode, both incoming and outgoing
>>> bandwith are limited by the director, while in DR/TUN mode, the director
>>> only limits the incoming bandwith. Beside this, the extra complexity of
>>> performing NAT for incoming and outgoing packets does also affect the
>>> overall performance of the director.
>>>
>>>
>>> Best,
>>>
>>> Anders
>>>
>>>
>>>
>>> On 30.04.2014, net.study.sea@gmail.com wrote:
>>>> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
>>>>
>>>> ÔÚ 2014-4-29£¬22:43£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>>>
>>>>> On 29.04.2014, net.study.sea@gmail.com wrote:
>>>>>> In nat mode,does director do dnat and snat packets for all request packets?
>>>>>
>>>>> In NAT mode, the director does perform destination nat on request packets,
>>>>> so your realserver does still see the correct "source" ip address.
>>>>>
>>>>> However, replies then need to be sent via the director as well, so the
>>>>> director can reverse the changed destination IP address and the client
>>>>> does see the reply packet from the "expected" IP address/port.
>>> --
>>> 1&1 Internet AG Expert Systems Architect (IT Operations)
>>> Brauerstrasse 50 v://49.721.91374.0
>>> D-76135 Karlsruhe f://49.721.91374.225
>>>
>>> Amtsgericht Montabaur HRB 6484
>>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>>> Jan Oetjen, Christian W¨¹rst
>>> Aufsichtsratsvorsitzender: Michael Scheeren
> --
> 1&1 Internet AG Expert Systems Architect (IT Operations)
> Brauerstrasse 50 v://49.721.91374.0
> D-76135 Karlsruhe f://49.721.91374.225
>
> Amtsgericht Montabaur HRB 6484
> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
> Jan Oetjen, Christian W¨¹rst
> Aufsichtsratsvorsitzender: Michael Scheeren

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] lvs. Nat [ In reply to ]
Hello,

Are you using a broadcomm NIC? Perhaps you are hitting the issue with GRO
enabled causing brutal performance. If so, you could try disabling GRO
and see if things are speedy.

ethtool -K eth0 gro off
ethtool -K eth1 gro off

https://bugzilla.redhat.com/show_bug.cgi?id=854066

daryl

On Thu, 1 May 2014, net.study.sea@gmail.com wrote:

>
> Thanks for reply!
>
> I have configured the reply going through director ,but the reply is very slowly.
>
> ÔÚ 2014-4-30£¬22:36£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>
>> On 30.04.2014, net.study.sea@gmail.com wrote:
>>> ÔÚ 2014-4-30£¬19:56£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>>
>>>> There are multiple issues with this.
>>>> -IP-address changes
>>>> -possible port changes
>>>>
>>>>
>>>> The client initiates a connection from Client-IP A to Director-VIP B:
>>>>
>>>> A->B, using a random source port number to destination port 80.
>>>>
>>>> The director rewrites the IP packet as if it had been sent
>>>> from Client A to Realserver C and sends it to the realserver:
>>>> A->C, using the client's source port number to destination port 80.
>>>>
>>>> If the realserver would reply to those director-NATed packet,
>>>> the reply would look like this:
>>>>
>>>> C->A, from port 80 to client's source port
>>>
>>> At this time,if C could reply directly to A
>>> ,not through director ,then what is the result?
>>>
>>>
>>> After I configure well Lvs in NAT mode,
>>> I find it deal with client's request so slowly, can not find the reason.
>>
>> If you're balancing only a single tcp port, the director will reply
>> to any other tcp ports and protocols.
>>
>> So "ping" (icmp) will be answered from the director with the
>> correct IP address, and the attempt to access a port without
>> a listener on the director will result in "connection refused".
>>
>> When you're accessing the balanced tcp port, the realserver will reply
>> from an unexpected IP address and the client will drop the packet as
>> being invalid.
>>
>> After some timeout, the client will retry sending the packet,
>> which will also fail. So it's not "slow" for your client to access the balanced service, it's "impossible" to establish a working connection.
>>
>>
>> If you do want to bypass the director, ask yourself why you'd like to do so:
>> - you're expecting more outgoing bandwith than the director will
>> be able to handle: you need to use either direct routing or tunneling.
>>
>> - you're not comfortable with the idea of making your director also
>> your host's default gateway.
>>
>> By choosing tunneling or direct routing, you'll gain that possibility,
>> but maybe you don't want to care about the ARP problem (a minor
>> configuration on every realserver - if you've forgotten one,
>> that realserver may attract all traffic to be balanced) or
>> about monitoring service X at IP address Y (the balanced IP
>> is not directly accessible for monitoring, so all checks need
>> to be performed at a different IP).
>>
>> If it's just a mindset issue - you may see it that way: if your director
>> is not available, your service is not available and it doesn't matter
>> that you can't access your hosts. In order to re-establish service,
>> you need to make your director available anyway, and after you've done so,
>> you may need to take care about your hosts.
>>
>>
>>
>>
>> Anders
>>
>>
>>>> The client in turn knows it did create a connection from A to B,
>>>> but doesn't know about a connection from A to C and so can't
>>>> match those reply packets with "the other" connection. Well,
>>>> their destination port is known to be in use with a different connection,
>>>> so why should A assume this to be valid?
>>>>
>>>> The reply packets are being dropped as being invalid.
>>>>
>>>>
>>>>
>>>> Depending on your configuration, your director may also change
>>>> the destination port, e.g. from 80 to 8080:
>>>> A->C, using the client's source port number to destination port 8080
>>>>
>>>> ... and so the realserver's reply would look like this:
>>>>
>>>> C->A, from port 8080 to client's source port
>>>>
>>>> The client in turn knows it did create a connection from A to B.
>>>> Any replies from C won't match that connection and so are being dropped.
>>>> Neither IP address nor port number do even closely match a known connection,
>>>> there's no way for the client to match them.
>>>>
>>>> In both cases, any firewall protecting A will perform similar
>>>> checks and drop the reply packets, as they are not related to a known
>>>> outgoing connection.
>>>>
>>>> By routing and "un-NAT'ing" the reply packet via the director, the
>>>> director will rewrite the packet from 'C->A' to 'B->A' and rewrite
>>>> any port changes as well to meet the expectations of A (or any
>>>> firewall in between director and A).
>>>>
>>>>
>>>> In Direct Routing mode, the IP packet isn't changed at all, just
>>>> the ethernet destination MAC address ("outside" of the IP packet)
>>>> is changed. In Tunneling mode, the incoming IP packet is encapsulated and
>>>> isn't changed as well.
>>>>
>>>> That's why in Direct Routing and Tunneling mode, any replies will be sent
>>>> bypassing your director and your director will only see "incoming" packets.
>>>>
>>>> And that's also the reason why in NAT mode, both incoming and outgoing
>>>> bandwith are limited by the director, while in DR/TUN mode, the director
>>>> only limits the incoming bandwith. Beside this, the extra complexity of
>>>> performing NAT for incoming and outgoing packets does also affect the
>>>> overall performance of the director.
>>>>
>>>>
>>>> Best,
>>>>
>>>> Anders
>>>>
>>>>
>>>>
>>>> On 30.04.2014, net.study.sea@gmail.com wrote:
>>>>> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
>>>>>
>>>>> ÔÚ 2014-4-29£¬22:43£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>>>>
>>>>>> On 29.04.2014, net.study.sea@gmail.com wrote:
>>>>>>> In nat mode,does director do dnat and snat packets for all request packets?
>>>>>>
>>>>>> In NAT mode, the director does perform destination nat on request packets,
>>>>>> so your realserver does still see the correct "source" ip address.
>>>>>>
>>>>>> However, replies then need to be sent via the director as well, so the
>>>>>> director can reverse the changed destination IP address and the client
>>>>>> does see the reply packet from the "expected" IP address/port.
>>>> --
>>>> 1&1 Internet AG Expert Systems Architect (IT Operations)
>>>> Brauerstrasse 50 v://49.721.91374.0
>>>> D-76135 Karlsruhe f://49.721.91374.225
>>>>
>>>> Amtsgericht Montabaur HRB 6484
>>>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>>>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>>>> Jan Oetjen, Christian W¨¹rst
>>>> Aufsichtsratsvorsitzender: Michael Scheeren
>> --
>> 1&1 Internet AG Expert Systems Architect (IT Operations)
>> Brauerstrasse 50 v://49.721.91374.0
>> D-76135 Karlsruhe f://49.721.91374.225
>>
>> Amtsgericht Montabaur HRB 6484
>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>> Jan Oetjen, Christian W¨¹rst
>> Aufsichtsratsvorsitzender: Michael Scheeren
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

--
/**
* Daryl Herzmann
* Assistant Scientist -- Iowa Environmental Mesonet
* http://mesonet.agron.iastate.edu
*/
Re: [lvs-users] lvs. Nat [ In reply to ]
No , it is not Broadcomm nic

ÔÚ 2014-5-1£¬21:02£¬daryl herzmann <akrherz@iastate.edu> дµÀ£º

> Hello,
>
> Are you using a broadcomm NIC? Perhaps you are hitting the issue with GRO enabled causing brutal performance. If so, you could try disabling GRO and see if things are speedy.
>
> ethtool -K eth0 gro off
> ethtool -K eth1 gro off
>
> https://bugzilla.redhat.com/show_bug.cgi?id=854066
>
> daryl
>
> On Thu, 1 May 2014, net.study.sea@gmail.com wrote:
>
>>
>> Thanks for reply!
>>
>> I have configured the reply going through director ,but the reply is very slowly.
>>
>> ÔÚ 2014-4-30£¬22:36£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>
>>> On 30.04.2014, net.study.sea@gmail.com wrote:
>>>> ÔÚ 2014-4-30£¬19:56£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>>>
>>>>> There are multiple issues with this.
>>>>> -IP-address changes
>>>>> -possible port changes
>>>>>
>>>>>
>>>>> The client initiates a connection from Client-IP A to Director-VIP B:
>>>>>
>>>>> A->B, using a random source port number to destination port 80.
>>>>>
>>>>> The director rewrites the IP packet as if it had been sent
>>>>> from Client A to Realserver C and sends it to the realserver:
>>>>> A->C, using the client's source port number to destination port 80.
>>>>>
>>>>> If the realserver would reply to those director-NATed packet,
>>>>> the reply would look like this:
>>>>>
>>>>> C->A, from port 80 to client's source port
>>>>
>>>> At this time,if C could reply directly to A
>>>> ,not through director ,then what is the result?
>>>>
>>>>
>>>> After I configure well Lvs in NAT mode,
>>>> I find it deal with client's request so slowly, can not find the reason.
>>>
>>> If you're balancing only a single tcp port, the director will reply
>>> to any other tcp ports and protocols.
>>>
>>> So "ping" (icmp) will be answered from the director with the
>>> correct IP address, and the attempt to access a port without
>>> a listener on the director will result in "connection refused".
>>>
>>> When you're accessing the balanced tcp port, the realserver will reply
>>> from an unexpected IP address and the client will drop the packet as
>>> being invalid.
>>>
>>> After some timeout, the client will retry sending the packet,
>>> which will also fail. So it's not "slow" for your client to access the balanced service, it's "impossible" to establish a working connection.
>>>
>>>
>>> If you do want to bypass the director, ask yourself why you'd like to do so:
>>> - you're expecting more outgoing bandwith than the director will
>>> be able to handle: you need to use either direct routing or tunneling.
>>>
>>> - you're not comfortable with the idea of making your director also
>>> your host's default gateway.
>>>
>>> By choosing tunneling or direct routing, you'll gain that possibility,
>>> but maybe you don't want to care about the ARP problem (a minor
>>> configuration on every realserver - if you've forgotten one,
>>> that realserver may attract all traffic to be balanced) or
>>> about monitoring service X at IP address Y (the balanced IP
>>> is not directly accessible for monitoring, so all checks need
>>> to be performed at a different IP).
>>>
>>> If it's just a mindset issue - you may see it that way: if your director
>>> is not available, your service is not available and it doesn't matter
>>> that you can't access your hosts. In order to re-establish service,
>>> you need to make your director available anyway, and after you've done so,
>>> you may need to take care about your hosts.
>>>
>>>
>>>
>>>
>>> Anders
>>>
>>>
>>>>> The client in turn knows it did create a connection from A to B,
>>>>> but doesn't know about a connection from A to C and so can't
>>>>> match those reply packets with "the other" connection. Well,
>>>>> their destination port is known to be in use with a different connection,
>>>>> so why should A assume this to be valid?
>>>>>
>>>>> The reply packets are being dropped as being invalid.
>>>>>
>>>>>
>>>>>
>>>>> Depending on your configuration, your director may also change
>>>>> the destination port, e.g. from 80 to 8080:
>>>>> A->C, using the client's source port number to destination port 8080
>>>>>
>>>>> ... and so the realserver's reply would look like this:
>>>>>
>>>>> C->A, from port 8080 to client's source port
>>>>>
>>>>> The client in turn knows it did create a connection from A to B.
>>>>> Any replies from C won't match that connection and so are being dropped.
>>>>> Neither IP address nor port number do even closely match a known connection,
>>>>> there's no way for the client to match them.
>>>>>
>>>>> In both cases, any firewall protecting A will perform similar
>>>>> checks and drop the reply packets, as they are not related to a known
>>>>> outgoing connection.
>>>>>
>>>>> By routing and "un-NAT'ing" the reply packet via the director, the
>>>>> director will rewrite the packet from 'C->A' to 'B->A' and rewrite
>>>>> any port changes as well to meet the expectations of A (or any
>>>>> firewall in between director and A).
>>>>>
>>>>>
>>>>> In Direct Routing mode, the IP packet isn't changed at all, just
>>>>> the ethernet destination MAC address ("outside" of the IP packet)
>>>>> is changed. In Tunneling mode, the incoming IP packet is encapsulated and
>>>>> isn't changed as well.
>>>>>
>>>>> That's why in Direct Routing and Tunneling mode, any replies will be sent
>>>>> bypassing your director and your director will only see "incoming" packets.
>>>>>
>>>>> And that's also the reason why in NAT mode, both incoming and outgoing
>>>>> bandwith are limited by the director, while in DR/TUN mode, the director
>>>>> only limits the incoming bandwith. Beside this, the extra complexity of
>>>>> performing NAT for incoming and outgoing packets does also affect the
>>>>> overall performance of the director.
>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>> Anders
>>>>>
>>>>>
>>>>>
>>>>> On 30.04.2014, net.study.sea@gmail.com wrote:
>>>>>> The real server received packet's source ip is client ip , why not it reply directly to client if there is router available route?
>>>>>>
>>>>>> ÔÚ 2014-4-29£¬22:43£¬Anders Henke <anders.henke@1und1.de> дµÀ£º
>>>>>>
>>>>>>> On 29.04.2014, net.study.sea@gmail.com wrote:
>>>>>>>> In nat mode,does director do dnat and snat packets for all request packets?
>>>>>>>
>>>>>>> In NAT mode, the director does perform destination nat on request packets,
>>>>>>> so your realserver does still see the correct "source" ip address.
>>>>>>>
>>>>>>> However, replies then need to be sent via the director as well, so the
>>>>>>> director can reverse the changed destination IP address and the client
>>>>>>> does see the reply packet from the "expected" IP address/port.
>>>>> --
>>>>> 1&1 Internet AG Expert Systems Architect (IT Operations)
>>>>> Brauerstrasse 50 v://49.721.91374.0
>>>>> D-76135 Karlsruhe f://49.721.91374.225
>>>>>
>>>>> Amtsgericht Montabaur HRB 6484
>>>>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>>>>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>>>>> Jan Oetjen, Christian W¨¹rst
>>>>> Aufsichtsratsvorsitzender: Michael Scheeren
>>> --
>>> 1&1 Internet AG Expert Systems Architect (IT Operations)
>>> Brauerstrasse 50 v://49.721.91374.0
>>> D-76135 Karlsruhe f://49.721.91374.225
>>>
>>> Amtsgericht Montabaur HRB 6484
>>> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
>>> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
>>> Jan Oetjen, Christian W¨¹rst
>>> Aufsichtsratsvorsitzender: Michael Scheeren
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> --
> /**
> * Daryl Herzmann
> * Assistant Scientist -- Iowa Environmental Mesonet
> * http://mesonet.agron.iastate.edu
> */
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users