Mailing List Archive

On Sun, Jan 28, 2001 at 12:38:36AM +0100, Alois Treindl wrote:
> Hi
>
> I will be setting up a LVS with one directore, 100 mbit Internet
> connection, and four web servers.
>
> I want to use the director also as a packet filtering firewall, with two
> interfaces. I have not enough funds and see no real need for an extra
> "real" firewall.
>
> As far as I understand it, DR would not have a performance advantage in
> that case over NAT, because all outgoing packets of from the realservers
> stil have to pass the director/firewall.

[snip]

My (informed) oppinion is that the main performance advantage of using DR
over NAT is derived from return traffic not having to return through the
box. If you are using 100Mb/s networking then NAT sould easily be
able to cope with this. You will probably incur a _slight_ latency penalty
but I doubt that this will be a problem. If I was you I would use
NAT as it is going to be a lot easier to set up and should run more
than fast enough for your needs.

If you are really worried about performance you should look into:

* Gigabit NICs
* Using 64bit/66MHz PCI bus (instead of 32bit/33MHz)
* Using the 2.4 kernel instead of 2.2

However, you won't see any real performance gains unless you are worried
about more than 100Mb/s of _sustained_ traffic.

DR is a nice way to get more than 100Mb/s of traffic on a network that is
only 100Mb/s but has more external bandwidth than that. If all your traffic
is going through a firewall with 2 100Mb/s NICs then this doesn't apply to
your configuration. Hence, you are better off using NAT.

--
Horms

Horms wrote:
> If all your traffic
> is going through a firewall with 2 100Mb/s NICs then this doesn't apply to
> your configuration. Hence, you are better off using NAT.

Thank you for the clear statement.
I am much impressed by the quality of information in this LVS group, the
quality
of the documentation and the friendly tone. It looks like his is really a
great
project, very much needed in that cut-throat market of 'content handling'
and security
devices.

Alois
--
|| Alois Treindl, Astrodienst AG, mailto:alois@astro.com
|| Zollikon/Zurich, Switzerland

On Sun, 28 Jan 2001, Alois Treindl wrote:

> I have read the (excellent!!!) HOWTO and mini-HOWTO, but they are not clear
> on this
> point. The potential performance problem with NAT is discussed, but not for
> DR in a configuration with a dual-interface directory. It is mentioned that
> a DR cluster will
> also have performance penalties when the return path goes through the
> director, but no
> comparison with NAT performance is made.

There is a detailed comparison of VS-NAT and VS-DR (and others)
performance on the Documents page.

Joe

--
Joseph Mack mack@ncifcrf.gov

Horms wrote:

> My (informed) oppinion is that the main performance advantage of using DR
> over NAT is derived from return traffic not having to return through the
> box.

When I compared DR (using Julian's martian patch, which allowed the director
to be the default gw for the real-servers) and NAT, at the same packet
throughput, the load average was 5 on the NAT director and the keyboard
and mouse weren't responding anymore, while the DR director had low load
average (<0.1 I think) and the mouse and keyboard responded just fine. I assume
the rewriting of packets in NAT is the main load on the director. The same
CPU can push the VS-DR packets through without any apparent effort.

I used top to watch the load average. There have been comments here that
top doesn't report CPU usage by the kernel and may not be a useful tool
for comparing load for kernel bound CPU usage against user land usage,
but the difference in mouse/keyboard response was clear. (With top
I thought "system" displayed kernel usage so I'm not clear on the problem
with top).

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

On Mon, 29 Jan 2001, Joseph Mack wrote:

> When I compared DR (using Julian's martian patch, which allowed the director
> to be the default gw for the real-servers) and NAT, at the same packet
> throughput, the load average was 5 on the NAT director and the keyboard
> and mouse weren't responding anymore, while the DR director had low load
> average (<0.1 I think) and the mouse and keyboard responded just fine. I assume
> the rewriting of packets in NAT is the main load on the director. The same
> CPU can push the VS-DR packets through without any apparent effort.

Do I understand this correct?

The director was in both cases in a two-NIC configuration, so it
would also have to pass the return packets from the internal NIC to
the external NIC, even in DR mode?

I am surprised that there should be a big laod difference, whether those
packets are re-written or just passed.

Also, if I use the director as a firewall with ipchains and packet
filters, will it not anyway have to inspect each outgoing packet header,
independent whether it runs a NAT or DR configuration? Is there a reason
why I should see much difference in load levels then?

|| Alois Treindl, Astrodienst AG, mailto:alois@astro.com
|| Zollikon/Zurich, Switzerland

Alois Treindl wrote:
>
> On Mon, 29 Jan 2001, Joseph Mack wrote:
>
> > When I compared DR (using Julian's martian patch, which allowed the director
> > to be the default gw for the real-servers) and NAT, at the same packet
> > throughput, the load average was 5 on the NAT director and the keyboard
> > and mouse weren't responding anymore, while the DR director had low load
> > average (<0.1 I think) and the mouse and keyboard responded just fine. I assume
> > the rewriting of packets in NAT is the main load on the director. The same
> > CPU can push the VS-DR packets through without any apparent effort.
>
> Do I understand this correct?
>
> The director was in both cases in a two-NIC configuration, so it
> would also have to pass the return packets from the internal NIC to
> the external NIC, even in DR mode?

yes

> I am surprised that there should be a big laod difference, whether those
> packets are re-written or just passed.

I didn't know enough to make a prediction. I made up hand
waving explanations
after the fact but I don't know if they're correct.

> Also, if I use the director as a firewall with ipchains and packet
> filters, will it not anyway have to inspect each outgoing packet header,
> independent whether it runs a NAT or DR configuration? Is there a reason
> why I should see much difference in load levels then?

I understand that smart (and expensive) ethernet cards on
non-linux systems can inspect
the source and destination of a packet and forward the
packet without reading
the rest of the packet and without intervention of the CPU.
There appears
to be kernel code in linux (I think called "fast copy") that
works with
tulip cards that sounds a bit like this. However the cheap
NICs I have
are not supposed to be able to do any of this.

I found that high packet throughput on VS-DR director with
Julian's
martian modification put little load on the director and
that VS-NAT
did. I explain it by saying that the rewritting of the
packet causes
the load, but I don;t really know. I expect that routing
tables
derived from rules using only the source/dest addresses will
also
not load the CPU much.

I don't know why though.

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer
Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

[snip]
> As far as I understand it, DR would not have a performance advantage in that
> case over NAT, because all outgoing packets of from the realservers stil
> have to pass the director/firewall.
[snip]
> I have read the (excellent!!!) HOWTO and mini-HOWTO, but they are not
> clear on this point. The potential performance problem with NAT is
> discussed, but not for DR in a configuration with a dual-interface
> directory. It is mentioned that a DR cluster will also have
> performance penalties when the return path goes through the director,
> but no comparison with NAT performance is made.

It depends on what performance you're talking about, really. Compared to
it's 'normal', asynchronous self, you do lose performance when routing
packets back through the DR director. You no longer have an async route
to the upstream. Consider the following:

---------------------------------
+ Switch +------ 1000SX -----+ Router
---------------------------------
|100TX |100TX |100TX
Director Server Server



Under normal DR, traffic follows the following path:
1. Incoming to router (Unknown)
2. Router to director (100TX / 1000SX)
3. Director to server (100TX)
4. Server to router (100TX / 1000SX)

If you go back through the director:
1. Incoming to router (Unknown)
2. Router to director (100TX / 1000SX)
3. Director to server (100TX)
4. Server to director (100TX)
5. Director to router (100TX)

Okay, so with the second option, you lose the higher return throughput,
since in the first, your limit is 100 times the number of real servers, to
a maximum of 100MBit; on the second, it's 100, since that's all the
director can handle.

Now, NAT v. DR traffic path looks something like this:

DR:
1. World to router
2. Router to director
a. Director checks IP payload.
b. Select destination.
c. Mangle layer 2 frame address
d. Recalculate frame checksum
3. Director sends frame to server
4. Server handles traffic as normal.
Optionally, if server reroutes through director:
5. Server sends to director
6. Director 'routes' packet, or mangles it as in step 2.
(I'm not sure which, I'm assuming mangles --
anyone know for sure?)

NAT:
1. World to router
2. Router to director
a. Director checks IP payload.
b. Director mangles IP payload.
c. Director recalculates IP checksum.
d. Director creates new frame to send IP packet in
1. (Calculates frame checksum -- usually in
hardware)
3. Send to frame+packet to server.
4. Server replies through director.
a. Director checks IP payload
b. Director mangles IP payload
c. Director recalculates IP checksum
1. NOT IN HARDWARE.
d. Director creates new frame to send packet in
1. (Caculates frame checksum -- usually in
hardware)
5. Director sends packet to router.


DR has fewer steps, and most of it's checksumming can be done in hardware.
NAT has that step where it mangles the IP packet, so it has to do software
checksumming of that packet :) So it costs a _lot_ more CPU wise to do
NAT.

So, to answer your original question -- DR ALWAYS has a performance
advantage. NAT is more expensive. But, it also has the potential for
'feature' advantages, though, since it's layer 3 aware.

For example, NAT can 'see' if a real server responds to a packet it's been
sent or not, since it's watching all of the traffic anyway. And if it
doesn't respond within a certain period of time, it can automatically
route that packet to another server. AFAIK, LVS doesn't support this
right now, but, NAT would be the more likely candidate to support it in
the future, as NAT understands all of the IP layer concepts, and DR
doesn't necessarily.

Anyhow, enough talking to hear myself talk ;)

Kyle Sparger

Kyle Sparger wrote:
>

> Okay, so with the second option, you lose the higher return throughput,
> since in the first, your limit is 100 times the number of real servers, to
> a maximum of 100MBit; on the second, it's 100, since that's all the
> director can handle.

I presume what you're saying here is that if you have 100 real-servers, each
with their own connection to the internet, and you are using some service where
the packets from the real-servers are large and the reply packets from
the client are small (eg an ACK) as you would have with ftp, or http where the
hits are large enough), you can get 100x throughput with VS-DR
whereas with VS-NAT, the director would become the bottleneck.

This piece of information was passed around a lot in the early days of LVS
as one of the advantages of VS-DR. However bench testing (see my performance
tests on the Documents page) shows that you
don't get this advantage with VS-DR. The assumption that is wrong is
that small packets take a shorter time to transmit, than do packets
of MTU size. It turns out that (at least with Linux)
that all packets take the same amount of time to transmit. The 100Mbps
throughput only occurs when all packets are MTU size. If each packet
has a payload of 1byte, then the max throughput is 100Mpbs * 1/1500.

The consequence of this is that the zero bandwidth ACKs from the
client are taking up as much ethernet capacity as does the 100Mbps
of packets being sent from the real-server.
1 real-server will take up all the bandwidth of a VS-DR
director. The difference between a VS-DR director and a VS-NAT director
is that the VS-DR director will have a low load average, while the
VS-NAT director will have a high load average.



I like this next bit about VS-NAT being layer-3 aware. I've put it into
the next version of the HOWTO.

> But, it also has the potential for
> 'feature' advantages, though, since it's layer 3 aware.
>
> For example, NAT can 'see' if a real server responds to a packet it's been
> sent or not, since it's watching all of the traffic anyway. And if it
> doesn't respond within a certain period of time, it can automatically
> route that packet to another server. AFAIK, LVS doesn't support this
> right now, but, NAT would be the more likely candidate to support it in
> the future, as NAT understands all of the IP layer concepts, and DR
> doesn't necessarily.

Joe
--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

On Mon, 29 Jan 2001, Joseph Mack wrote:

> The assumption that is wrong is that small packets take a shorter time
> to transmit, than do packets of MTU size.
[snip]
> The consequence of this is that the zero bandwidth ACKs from the
> client are taking up as much ethernet capacity as does the 100Mbps
> of packets being sent from the real-server.

The benefeit is far rarer than I anticipated. Most interesting. Thanks
for educating me :)

The 100MBit ethernet MTU is fixed at 1500, but what if the back-end
servers MTU is higher on layer three? Or if there is little or no ACK
traffic -- say you're using UDP, for example?

But, I will admit that the benefeit is in the rare case, certainly not the
common case. It would take some engineering or a feat of circumstance to
cause it to occur :)

Thanks,

Kyle Sparger - Senior System Administrator
Dialtone Internet - Extremely Fast Web Systems
(954) 581-0097 x 122 - Voice (954) 581-7629 - Fax
ksparger@dialtoneinternet.net
http://www.dialtoneinternet.net
"Forget college, I'm going pro."

> Someone must put back the real server when it is alive. This
> sounds like a user space job.

Absolutely.

> The traffic will not start until we send requests. We have to send L4
> probes to the real server (from the user space) or to probe it with
> requests (LVS from kernel space)?

How about selecting a new server for the connection, and signalling (not
necessarily via signals, mind you...) to a user-space program that the
target should immediately be checked for availability? Then the userspace
program can set up checks to be run, and re-add it later when it becomes
available again -- assuming it really is unavailable.

If the server _is_ available, and acting normally, it should respond in
the first place, though, right? :)

This method might be more efficient than the existing scheduled poll
method, but I'm sure it also has hidden traps.

Thanks,

Kyle Sparger - Senior System Administrator
Dialtone Internet - Extremely Fast Web Systems
(954) 581-0097 x 122 - Voice (954) 581-7629 - Fax
ksparger@dialtoneinternet.net
http://www.dialtoneinternet.net
"Forget college, I'm going pro."

Kyle Sparger wrote:
>
> > of packets being sent from the real-server.
>
> The benefeit is far rarer than I anticipated. Most interesting. Thanks
> for educating me :)

it was a bit of a surprise to me too. You don't need an LVS to test this.
You can see it with netpipe between 2 nodes on a quiet line. You can't
get 100Mbps till the packet size reaches the mtu, sending packets
as fast as the CPU can send them.

> The 100MBit ethernet MTU is fixed at 1500, but what if the back-end
> servers MTU is higher on layer three?

well I guess the ACK for the new MTU would take longer too :-)

Or if there is little or no ACK
> traffic -- say you're using UDP, for example?

now that's an idea, save acks till a timeout or max_packets=10
and then send them in 1 packet. Do delayed ACKs come close to doing this?

Joe

--
Joseph Mack PhD, Senior Systems Engineer, Lockheed Martin
contractor to the National Environmental Supercomputer Center,
mailto:mack.joseph@epa.gov ph# 919-541-0007, RTP, NC, USA

Hello,

On Mon, 29 Jan 2001, Kyle Sparger wrote:

> For example, NAT can 'see' if a real server responds to a packet it's been
> sent or not, since it's watching all of the traffic anyway. And if it
> doesn't respond within a certain period of time, it can automatically
> route that packet to another server. AFAIK, LVS doesn't support this
> right now, but, NAT would be the more likely candidate to support it in
> the future, as NAT understands all of the IP layer concepts, and DR
> doesn't necessarily.

Someone must put back the real server when it is alive. This
sounds like a user space job. The traffic will not start until we send
requests. We have to send L4 probes to the real server (from the user
space) or to probe it with requests (LVS from kernel space)?

> Kyle Sparger


Regards

--
Julian Anastasov <ja@ssi.bg>

Hello,

On Mon, 29 Jan 2001, Kyle Sparger wrote:

> How about selecting a new server for the connection, and signalling (not
> necessarily via signals, mind you...) to a user-space program that the
> target should immediately be checked for availability? Then the userspace
> program can set up checks to be run, and re-add it later when it becomes
> available again -- assuming it really is unavailable.

My first thought is that this can be very complex. I don't
have good ideas.

> If the server _is_ available, and acting normally, it should respond in
> the first place, though, right? :)
>
> This method might be more efficient than the existing scheduled poll
> method, but I'm sure it also has hidden traps.

Agreed. If you have many real servers the L4 checks can eat
many CPU cycles in the director.

There can be another model: register for information
in a real server agent who will notify the director with one
packet (on each 1 to 5 seconds): "All my real services are OK".
If some real service is stopped it is reported. If the host is down
or busy and the agent can't report "I'm alive" for specific time
all real services can be removed from the director user space cluster
software. Of course, this is dangerous if the kernel kills your agent
on OOM ("thanks" to the new OOM killers) but everything can happen.
20 real servers reporting info on each 2 seconds means 10p/sec
which is a low value. With this packet we can even report the weight(s)
for the real services and to use it as iamalive reply.

BTW, I'm working on such system but am too busy with other
things. I'll try to implement the basics and then other people can
extend it.

> Thanks,
>
> Kyle Sparger - Senior System Administrator
> Dialtone Internet - Extremely Fast Web Systems
> (954) 581-0097 x 122 - Voice (954) 581-7629 - Fax
> ksparger@dialtoneinternet.net
> http://www.dialtoneinternet.net
> "Forget college, I'm going pro."


Regards

--
Julian Anastasov <ja@ssi.bg>

> > How about selecting a new server for the connection, and signalling (not
> > necessarily via signals, mind you...) to a user-space program that the
> > target should immediately be checked for availability? Then the userspace
> > program can set up checks to be run, and re-add it later when it becomes
> > available again -- assuming it really is unavailable.
> My first thought is that this can be very complex. I don't
> have good ideas.

Or it can be very simple. You could simply register a program when you
create a virtual service on the director. When a condition occurs that
the LVS code has a reasonable belief that something may go wrong, it
simply spawns off the program with a pre-defined set of arguments; for
example,

program host port time

I think that should probably be sufficient in almost all cases, since if
you have very specific needs for the check, you can simply register a
program which provides them.

And it seems reasonably simple; the hardest part of that would probably
be the code that triggers it.

Thanks,

Kyle Sparger

On Tue, 30 Jan 2001, Julian Anastasov wrote:

>
> Hello,
>
> On Mon, 29 Jan 2001, Kyle Sparger wrote:
>
> > How about selecting a new server for the connection, and signalling (not
> > necessarily via signals, mind you...) to a user-space program that the
> > target should immediately be checked for availability? Then the userspace
> > program can set up checks to be run, and re-add it later when it becomes
> > available again -- assuming it really is unavailable.
>
> My first thought is that this can be very complex. I don't
> have good ideas.
>
> > If the server _is_ available, and acting normally, it should respond in
> > the first place, though, right? :)
> >
> > This method might be more efficient than the existing scheduled poll
> > method, but I'm sure it also has hidden traps.
>
> Agreed. If you have many real servers the L4 checks can eat
> many CPU cycles in the director.
>
> There can be another model: register for information
> in a real server agent who will notify the director with one
> packet (on each 1 to 5 seconds): "All my real services are OK".
> If some real service is stopped it is reported. If the host is down
> or busy and the agent can't report "I'm alive" for specific time
> all real services can be removed from the director user space cluster
> software. Of course, this is dangerous if the kernel kills your agent
> on OOM ("thanks" to the new OOM killers) but everything can happen.
> 20 real servers reporting info on each 2 seconds means 10p/sec
> which is a low value. With this packet we can even report the weight(s)
> for the real services and to use it as iamalive reply.
>
> BTW, I'm working on such system but am too busy with other
> things. I'll try to implement the basics and then other people can
> extend it.
>
> > Thanks,
> >
> > Kyle Sparger - Senior System Administrator
> > Dialtone Internet - Extremely Fast Web Systems
> > (954) 581-0097 x 122 - Voice (954) 581-7629 - Fax
> > ksparger@dialtoneinternet.net
> > http://www.dialtoneinternet.net
> > "Forget college, I'm going pro."
>
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>
>

Hello,

On Tue, 30 Jan 2001, Kyle Sparger wrote:

> > My first thought is that this can be very complex. I don't
> > have good ideas.
>
> Or it can be very simple. You could simply register a program when you
> create a virtual service on the director. When a condition occurs that
> the LVS code has a reasonable belief that something may go wrong, it
> simply spawns off the program with a pre-defined set of arguments; for
> example,

We are talking about the implementation details. The word "simple"
can be applied only to the logic :) Talking from the kernel to the user
space is not so simple. The people use netlink sockets for this. But
may be such interface can be implemented: the cluster software to
receive events. But what other kinds of events we can implement?

> Thanks,
>
> Kyle Sparger


Regards

--
Julian Anastasov <ja@ssi.bg>

Joseph Mack wrote:
>
> When I compared DR (using Julian's martian patch, which allowed the director
> to be the default gw for the real-servers) and NAT, at the same packet
> throughput, the load average was 5 on the NAT director and the keyboard
> and mouse weren't responding anymore, while the DR director had low load
> average (<0.1 I think) and the mouse and keyboard responded just fine. I assume

So this happened at how many Mbit/s ?

--
Florin Andrei

On Tue, 6 Feb 2001, Florin Andrei wrote:

> Joseph Mack wrote:
> >
> > When I compared DR (using Julian's martian patch, which allowed the director
> > to be the default gw for the real-servers) and NAT, at the same packet
> > throughput, the load average was 5 on the NAT director and the keyboard
> > and mouse weren't responding anymore, while the DR director had low load
> > average (<0.1 I think) and the mouse and keyboard responded just fine. I assume
>
> So this happened at how many Mbit/s ?

50Mbps.

My real-servers are 75 MHz pentium classics. They max out at 50Mbps on the
network.

Joe
--
Joseph Mack mack@ncifcrf.gov