Mailing List Archive

Hello,

Greetings, I am new to this list as well and would like to offer some
friendly advice. Get your MySQL databases on to some dedicated servers.
Why run a RDBMS + Webserver on the same machine? You should see some big
improvements...

Regards,
Rajiv Khaneja

-----Original Message-----
From: lvs-users-admin@LinuxVirtualServer.org
[mailto:lvs-users-admin@LinuxVirtualServer.org]On Behalf Of Johan
Isacsson
Sent: February 25, 2001 10:47 PM
To: Lvs-Users
Subject: Introduction and LVS/DR 2.4 realserver questions


Hello,

I'm new to this list so here comes a little introduction.
My name is Johan Isacsson and i'm the CTO of MGON International AB, based in
Sweden.
MGON creates and maintains websites about computer games, traffic is about
25 million impressions/month.
Almost all pages are dynamic, so we need to distribute the load over several
servers. We're running Linux 2.2 with Apache+PHP and MySQL on all
realservers, about 16 in total.
We use custom/scp script for file mirroring but are looking for better ways
of doing that (any suggestions are welcome).
Currently we use LVS/DR to distribute the load and it works very well for
us.
Before that we used IBM Network Dispatcher which i think worked the same way
as LVS/DR do, but it costs lots of money! :P
Well, enough about me, on to the questions!

1. Is the redirect approach working (well) on 2.4, and if so, are there any
drawbacks using that method compared to the hidden arp method in 2.2?
2. Is it a very bad idea to just have 1 NIC on the LVS machine to receive
and forward the requests?

It's a DR setup.

We have another server location with a 4Mbit connection that we don't use
much. We have some servers there serving static files, but i'd like to load
balance them between that location and our main location. Like this:

Location 1 (Max 4Mbit)
Static file servers
FS1
FS2

Location 2
Static file servers
FS3
FS4
Virtual server
VFS1 (FS1,FS2,FS3,FS4 LVS/TUN)

I'd like to use as much as possible of the bandwidth in location 1 (up to
4Mbit) and after that start using the fileservers at location 2. The
bandwith usage can be taken from the router at location 1 using snmp.
If anyone have suggestions about how to set up scheduling for this i'd love
to hear them :)

Thanks for your time,

Johan Isacsson
MGON


_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

Hi Johan,

Johan Isacsson wrote:
>
> Hello,
>
> I'm new to this list so here comes a little introduction.
> My name is Johan Isacsson and i'm the CTO of MGON International AB, based in
> Sweden.
> MGON creates and maintains websites about computer games, traffic is about
> 25 million impressions/month.

Is impressions==connection requests?

> Almost all pages are dynamic, so we need to distribute the load over several
> servers. We're running Linux 2.2 with Apache+PHP and MySQL on all
> realservers, about 16 in total.

And you never got rooted? ;)

> We use custom/scp script for file mirroring but are looking for better ways
> of doing that (any suggestions are welcome).

You might have a look at http://www.linux-ha.org and check out the
links on the first page about Filesystem technologies.

> Currently we use LVS/DR to distribute the load and it works very well for
> us.
> Before that we used IBM Network Dispatcher which i think worked the same way
> as LVS/DR do, but it costs lots of money! :P
> Well, enough about me, on to the questions!

I'm glad you switched to LVS :)

> 1. Is the redirect approach working (well) on 2.4, and if so, are there any
> drawbacks using that method compared to the hidden arp method in 2.2?

Yes, it is working well, no, no drawbacks so far, but the difference is
not redirect <-> hidden arp for 2.4.x <-> 2.2.x. The difference IMHO is
the place where the LVS hooks in and the trespassing of the chains. BTW,
why do you intent to change to 2.4.x if you setup is working fine? And I
reckon that you run your db on a Linux box with fast SCSI controller or
even a RAID. If so I would not advise you to use the 2.4.x kernel since
there are still some issues not solved with RAID and SCSI at least back
here, where I still try to get it running for more then 10 hours. :(

> 2. Is it a very bad idea to just have 1 NIC on the LVS machine to receive
> and forward the requests?

This actually would be the standard approach for LVS-DR.

> It's a DR setup.
>
> We have another server location with a 4Mbit connection that we don't use
> much. We have some servers there serving static files, but i'd like to load
> balance them between that location and our main location. Like this:
>
> Location 1 (Max 4Mbit)
> Static file servers
> FS1
> FS2
>
> Location 2
> Static file servers
> FS3
> FS4
> Virtual server
> VFS1 (FS1,FS2,FS3,FS4 LVS/TUN)
>
> I'd like to use as much as possible of the bandwidth in location 1 (up to
> 4Mbit) and after that start using the fileservers at location 2. The
> bandwith usage can be taken from the router at location 1 using snmp.
> If anyone have suggestions about how to set up scheduling for this i'd love
> to hear them :)

Try it with the lblcr scheduler. Although I actually don't understand
your setup correctly, honestly. Maybe someone else understands it better
than me or you could draw a sketch.

> Thanks for your time,

No problem, I hope I could give you some more information than you already
had.

Best regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`

Hello Rajiv,

We have actually considered both options for quite some time and for now
httpd+mysql on same host seems to be the best configuration. I'll explain
why:
Each mahine can take up to about 200 http connections with 1 Gb RAM (all
realservers have 1 Gb RAM). Since we use PHP the http processes are at least
5 Mb in size, so we have big memory problems if we go any further than that.

At that load the local MySQLd runs fine having about 200 connections.

In a dedicated database setup we have big problems running over 500
connections to MySQL. So instead of having, for example, 2 webservers
sharing 1 database machine and still just get to handle 400 connections we
choose to have 2 web+db machines that can also take 400 connections.
That saves us the cost of 1 db mahine per 2 web servers. In total we save 8
machines, and that's http 1600 connections :)
If anyone have comments on this i'd love to hear them.

One thing we've been thinking about is to serve static images from separate
webservers running boa or some other small and efficent webserver. The
problem with it is that it would make life complicated for the designers and
webmasters unless there was a way of doing it behind the scenes.
One scenario could be that the load balancer would look at the requests and
direct all /images/* to the static virtual server. I know that some
loadbalancers can do that, how about LVS?

Well, thanks for your comments :)

Regards,
Johan Isacsson
MGON



> -----Ursprungligt meddelande-----
> Från: lvs-users-admin@LinuxVirtualServer.org
> [mailto:lvs-users-admin@LinuxVirtualServer.org]För Rajiv Khaneja
> Skickat: den 26 februari 2001 09:05
> Till: lvs-users@LinuxVirtualServer.org
> Ämne: RE: Introduction and LVS/DR 2.4 realserver questions
>
>
> Hello,
>
> Greetings, I am new to this list as well and would like to offer some
> friendly advice. Get your MySQL databases on to some dedicated servers.
> Why run a RDBMS + Webserver on the same machine? You should see some big
> improvements...
>
> Regards,
> Rajiv Khaneja
>
> -----Original Message-----
> From: lvs-users-admin@LinuxVirtualServer.org
> [mailto:lvs-users-admin@LinuxVirtualServer.org]On Behalf Of Johan
> Isacsson
> Sent: February 25, 2001 10:47 PM
> To: Lvs-Users
> Subject: Introduction and LVS/DR 2.4 realserver questions
>
>
> Hello,
>
> I'm new to this list so here comes a little introduction.
> My name is Johan Isacsson and i'm the CTO of MGON International
> AB, based in
> Sweden.
> MGON creates and maintains websites about computer games, traffic is about
> 25 million impressions/month.
> Almost all pages are dynamic, so we need to distribute the load
> over several
> servers. We're running Linux 2.2 with Apache+PHP and MySQL on all
> realservers, about 16 in total.
> We use custom/scp script for file mirroring but are looking for
> better ways
> of doing that (any suggestions are welcome).
> Currently we use LVS/DR to distribute the load and it works very well for
> us.
> Before that we used IBM Network Dispatcher which i think worked
> the same way
> as LVS/DR do, but it costs lots of money! :P
> Well, enough about me, on to the questions!
>
> 1. Is the redirect approach working (well) on 2.4, and if so, are
> there any
> drawbacks using that method compared to the hidden arp method in 2.2?
> 2. Is it a very bad idea to just have 1 NIC on the LVS machine to receive
> and forward the requests?
>
> It's a DR setup.
>
> We have another server location with a 4Mbit connection that we don't use
> much. We have some servers there serving static files, but i'd
> like to load
> balance them between that location and our main location. Like this:
>
> Location 1 (Max 4Mbit)
> Static file servers
> FS1
> FS2
>
> Location 2
> Static file servers
> FS3
> FS4
> Virtual server
> VFS1 (FS1,FS2,FS3,FS4 LVS/TUN)
>
> I'd like to use as much as possible of the bandwidth in location 1 (up to
> 4Mbit) and after that start using the fileservers at location 2. The
> bandwith usage can be taken from the router at location 1 using snmp.
> If anyone have suggestions about how to set up scheduling for
> this i'd love
> to hear them :)
>
> Thanks for your time,
>
> Johan Isacsson
> MGON
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>

Hello Roberto,

> Is impressions==connection requests?

An impression is a loaded banner, and we have one banner per page (that we
count). So in our case one impression is one pageview.
We have about 12 hits (requests) per pageview, so that would be 25M+(12*25M)
= 325 million hits/requests per month.

> You might have a look at http://www.linux-ha.org and check out the
> links on the first page about Filesystem technologies.

I will, thanks :)

> > 1. Is the redirect approach working (well) on 2.4, and if so,
> are there any
> > drawbacks using that method compared to the hidden arp method in 2.2?
>
> Yes, it is working well, no, no drawbacks so far, but the difference is
> not redirect <-> hidden arp for 2.4.x <-> 2.2.x. The difference IMHO is
> the place where the LVS hooks in and the trespassing of the chains. BTW,
> why do you intent to change to 2.4.x if you setup is working fine? And I
> reckon that you run your db on a Linux box with fast SCSI controller or
> even a RAID. If so I would not advise you to use the 2.4.x kernel since
> there are still some issues not solved with RAID and SCSI at least back
> here, where I still try to get it running for more then 10 hours. :(

Thanks for the info, yes we're running SCSI on the realservers and i haven't
checked if the driver is working on 2.4 yet, so it might be a good idea to
wait for a while.

> Try it with the lblcr scheduler. Although I actually don't understand
> your setup correctly, honestly. Maybe someone else understands it better
> than me or you could draw a sketch.

Ok, here's another attempt:

Location 1 (4 Mbit max)

--------- Lan
| FS 1 |---
--------- | -----------
---| Router1 |
--------- | -----------
| FS 2 |--- |
--------- |
Internet------------------
|
|
------------
---| Router 2 |
| ------------
-------- |
| VFS1 |----|
-------- |
| Lan
-------- |
| FS 3 |----|
-------- |
|
-------- |
| FS 4 |----
--------

Router 1 has a limit of 4 Mbit
VFS1 is the virtual file server
FS1-4 are the real file servers

Basicly what i want is to load balance between the file servers but put all
load on FS1 & FS2 until the bandwidth usage on Router 1 is up at 4Mbit, then
start using FS3 & FS4 aswell. The load on this cluster is rapidly changing
so it won't work to put static weigths in the scheduler (at low traffic
Location 1 would be able to take all traffic but at high traffic 4Mbit isn't
enough).
I figure i could have a cron job get the bandwidth usage out of router 1 and
adjust the wieghts for FS3&4 in some way...
Comments are very welcome :)

> No problem, I hope I could give you some more information than you already
> had.

Yepp, thanks for your answer :)

Regards,
Johan Isacsson
MGON

Johan,
"In a dedicated database setup we have big problems running over 500
connections to MySQL."

You shouldn't have a problem with this... MySQL can handle well over that
given a good amount of ram and processing power.

Let me tell you about our setup and hopefully it will help you make a better
decision.

Two LVS directors mon+heartbeat+lvs
6 Web Servers (512mb ram + 733 Mhz)
2 DB Servers (1 GB ram + 866) in Hot Replication (active/active)

We serve aprox 280 million page views per month off this setup and MySQL
does avg 1,200 quieries per second.



About serving static images, there is a web server, khttpd, that can be
compiled into the kernel to serve static pages. Very efficient. Just have
all your images hosted at images.yourdomain.com or use a different port.

"One scenario could be that the load balancer would look at the requests and
direct all /images/* to the static virtual server. I know that some
loadbalancers can do that, how about LVS?"

I don't know if LVS can do this but if it can it would require LVS to read
HTTP request hearders and really slow things down. IMHO no realy point.

Regards,
Rajiv Khaneja

-----Original Message-----
From: lvs-users-admin@LinuxVirtualServer.org
[mailto:lvs-users-admin@LinuxVirtualServer.org]On Behalf Of Johan
Isacsson
Sent: February 26, 2001 12:36 AM
To: lvs-users@LinuxVirtualServer.org
Subject: SV: Introduction and LVS/DR 2.4 realserver questions


Hello Rajiv,

We have actually considered both options for quite some time and for now
httpd+mysql on same host seems to be the best configuration. I'll explain
why:
Each mahine can take up to about 200 http connections with 1 Gb RAM (all
realservers have 1 Gb RAM). Since we use PHP the http processes are at least
5 Mb in size, so we have big memory problems if we go any further than that.

At that load the local MySQLd runs fine having about 200 connections.

In a dedicated database setup we have big problems running over 500
connections to MySQL. So instead of having, for example, 2 webservers
sharing 1 database machine and still just get to handle 400 connections we
choose to have 2 web+db machines that can also take 400 connections.
That saves us the cost of 1 db mahine per 2 web servers. In total we save 8
machines, and that's http 1600 connections :)
If anyone have comments on this i'd love to hear them.

One thing we've been thinking about is to serve static images from separate
webservers running boa or some other small and efficent webserver. The
problem with it is that it would make life complicated for the designers and
webmasters unless there was a way of doing it behind the scenes.
One scenario could be that the load balancer would look at the requests and
direct all /images/* to the static virtual server. I know that some
loadbalancers can do that, how about LVS?

Well, thanks for your comments :)

Regards,
Johan Isacsson
MGON



> -----Ursprungligt meddelande-----
> Från: lvs-users-admin@LinuxVirtualServer.org
> [mailto:lvs-users-admin@LinuxVirtualServer.org]För Rajiv Khaneja
> Skickat: den 26 februari 2001 09:05
> Till: lvs-users@LinuxVirtualServer.org
> Ämne: RE: Introduction and LVS/DR 2.4 realserver questions
>
>
> Hello,
>
> Greetings, I am new to this list as well and would like to offer some
> friendly advice. Get your MySQL databases on to some dedicated servers.
> Why run a RDBMS + Webserver on the same machine? You should see some big
> improvements...
>
> Regards,
> Rajiv Khaneja
>
> -----Original Message-----
> From: lvs-users-admin@LinuxVirtualServer.org
> [mailto:lvs-users-admin@LinuxVirtualServer.org]On Behalf Of Johan
> Isacsson
> Sent: February 25, 2001 10:47 PM
> To: Lvs-Users
> Subject: Introduction and LVS/DR 2.4 realserver questions
>
>
> Hello,
>
> I'm new to this list so here comes a little introduction.
> My name is Johan Isacsson and i'm the CTO of MGON International
> AB, based in
> Sweden.
> MGON creates and maintains websites about computer games, traffic is about
> 25 million impressions/month.
> Almost all pages are dynamic, so we need to distribute the load
> over several
> servers. We're running Linux 2.2 with Apache+PHP and MySQL on all
> realservers, about 16 in total.
> We use custom/scp script for file mirroring but are looking for
> better ways
> of doing that (any suggestions are welcome).
> Currently we use LVS/DR to distribute the load and it works very well for
> us.
> Before that we used IBM Network Dispatcher which i think worked
> the same way
> as LVS/DR do, but it costs lots of money! :P
> Well, enough about me, on to the questions!
>
> 1. Is the redirect approach working (well) on 2.4, and if so, are
> there any
> drawbacks using that method compared to the hidden arp method in 2.2?
> 2. Is it a very bad idea to just have 1 NIC on the LVS machine to receive
> and forward the requests?
>
> It's a DR setup.
>
> We have another server location with a 4Mbit connection that we don't use
> much. We have some servers there serving static files, but i'd
> like to load
> balance them between that location and our main location. Like this:
>
> Location 1 (Max 4Mbit)
> Static file servers
> FS1
> FS2
>
> Location 2
> Static file servers
> FS3
> FS4
> Virtual server
> VFS1 (FS1,FS2,FS3,FS4 LVS/TUN)
>
> I'd like to use as much as possible of the bandwidth in location 1 (up to
> 4Mbit) and after that start using the fileservers at location 2. The
> bandwith usage can be taken from the router at location 1 using snmp.
> If anyone have suggestions about how to set up scheduling for
> this i'd love
> to hear them :)
>
> Thanks for your time,
>
> Johan Isacsson
> MGON
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users
>


_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

Hey Rajiv,

> Johan,
> "In a dedicated database setup we have big problems running over 500
> connections to MySQL."
>
> You shouldn't have a problem with this... MySQL can handle well over that
> given a good amount of ram and processing power.
>
> Let me tell you about our setup and hopefully it will help you
> make a better
> decision.
>
> Two LVS directors mon+heartbeat+lvs
> 6 Web Servers (512mb ram + 733 Mhz)
> 2 DB Servers (1 GB ram + 866) in Hot Replication (active/active)
>
> We serve aprox 280 million page views per month off this setup and MySQL
> does avg 1,200 quieries per second.

We've actually tried dedicated db setup on the live enviroment and after
about 500 connectsions we get terrible performance. I'm sure there are ways
to optimize the setup but i'm not convinced that it would gain us more httpd
connections/machine in the end...
All servers have 2x866, 1 Gb RAM and Ultra160 SCSI and we're using MySQL
replication.
But almost all our pageviews are database dependant and they ask pretty
complicated questions too... :P

> About serving static images, there is a web server, khttpd, that can be
> compiled into the kernel to serve static pages. Very efficient.
> Just have
> all your images hosted at images.yourdomain.com or use a different port.

Is "TUX" stable now? I tried it a while ago and it creashed the machine...

> "One scenario could be that the load balancer would look at the
> requests and
> direct all /images/* to the static virtual server. I know that some
> loadbalancers can do that, how about LVS?"
>
> I don't know if LVS can do this but if it can it would require LVS to read
> HTTP request hearders and really slow things down. IMHO no realy point.

I agree that if it would really slow things down it wont be worth it, but it
would
make things much easier for the webmasters/designers if they didn't have to
keep track
of different FTP accounts etc...

Thanks for your comments!

Regards,
Johan Isacsson
MGON

Hi Johan,

> An impression is a loaded banner, and we have one banner per page (that we
> count). So in our case one impression is one pageview.
> We have about 12 hits (requests) per pageview, so that would be 25M+(12*25M)
> = 325 million hits/requests per month.

Ok, that sounds more reasonable.

> Thanks for the info, yes we're running SCSI on the realservers and i haven't
> checked if the driver is working on 2.4 yet, so it might be a good idea to
> wait for a while.

Yet a better idea is to check now on a testmachine if it works correctly
for you. But I guess you want to run some jfs and ext2 on the servers,
don't you? Just try it out before you switch to 2.2. However I really
recommend you to use the 2.4.x kernel series for a webserver and TUX 2.0
and maybe the newest released zerocopy patches, they are included now in
the ac-tree. For the LVS I recommend the 2.2.x kernel with the 1.0.5 patch.

> Ok, here's another attempt:

Last two questions before I will send you the network I have in mind
for you:
o Where are the webservers?
o What is the VFS and what is its purpose?

> Router 1 has a limit of 4 Mbit
> VFS1 is the virtual file server

I hope you don't mean the LVS.

> FS1-4 are the real file servers

Great, so far so good ...

> Basicly what i want is to load balance between the file servers but put all
> load on FS1 & FS2 until the bandwidth usage on Router 1 is up at 4Mbit, then
> start using FS3 & FS4 aswell. The load on this cluster is rapidly changing
> so it won't work to put static weigths in the scheduler (at low traffic
> Location 1 would be able to take all traffic but at high traffic 4Mbit isn't
> enough).

I have something in mind, but I first need the answers.

> I figure i could have a cron job get the bandwidth usage out of router 1 and
> adjust the wieghts for FS3&4 in some way...
> Comments are very welcome :)

I think you won't need this.

Best regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`

On Mon, 26 Feb 2001, Johan Isacsson wrote:

> One thing we've been thinking about is to serve static images from separate
> webservers running boa or some other small and efficent webserver. The
> problem with it is that it would make life complicated for the designers and
> webmasters unless there was a way of doing it behind the scenes.
> One scenario could be that the load balancer would look at the requests and
> direct all /images/* to the static virtual server. I know that some
> loadbalancers can do that, how about LVS?
>

Sounds like a job for TUX. People on the tux mailing list can probably
help you in designing system. You probably want to go the route of
changing all of your /images/* to point to a separate images server
running tux.

Check out
http://people.redhat.com/~mingo/TUX-patches/
ftp://ftp.redhat.com/pub/redhat/tux/tux-2.0/
http://www.redhat.com/products/software/ecommerce/tux/

The TUX Mailing List
Those interested in TUX usage or development are welcome, if they have not
already done so, to join the TUX mailing list at:
tux-list@redhat.com

You can subscribe using the command:
echo subscribe | mail tux-list-request@redhat.com

Or you can use the web interface at:
http://www.redhat.com/mailing-lists/

Hey Roberto,

> Yet a better idea is to check now on a testmachine if it works correctly
> for you. But I guess you want to run some jfs and ext2 on the servers,
> don't you? Just try it out before you switch to 2.2. However I really
> recommend you to use the 2.4.x kernel series for a webserver and TUX 2.0
> and maybe the newest released zerocopy patches, they are included now in
> the ac-tree. For the LVS I recommend the 2.2.x kernel with the
> 1.0.5 patch.

Thanks, i'll have a look at that :)

> Last two questions before I will send you the network I have in mind
> for you:
> o Where are the webservers?

They are at Location 2, i didn't include them in the figure, but they are a
separate virtual server.
The "File servers" are actually webservers but they serve static content
(downloads). All other webtraffic goes to another virtual server that is
just located at location 2 (using the same LVS though).

> o What is the VFS and what is its purpose?

VFS is the LVS.

> > Router 1 has a limit of 4 Mbit
> > VFS1 is the virtual file server
>
> I hope you don't mean the LVS.

Why? :P It is... I'm planning to use Tunneling.

> I have something in mind, but I first need the answers.

Excellent, thanks! :)

Regards,
Johan Isacsson
MGON

On Mon, 26 Feb 2001, Johan Isacsson wrote:

>
> Ok, here's another attempt:
>
> Location 1 (4 Mbit max)
>
> --------- Lan
> | FS 1 |---
> --------- | -----------
> ---| Router1 |
> --------- | -----------
> | FS 2 |--- |
> --------- |
> Internet------------------
> |
> |
> ------------
> ---| Router 2 |
> | ------------
> -------- |
> | VFS1 |----|
> -------- |
> | Lan
> -------- |
> | FS 3 |----|
> -------- |
> |
> -------- |
> | FS 4 |----
> --------
>
> Router 1 has a limit of 4 Mbit
> VFS1 is the virtual file server
> FS1-4 are the real file servers
>
> Basicly what i want is to load balance between the file servers but put all
> load on FS1 & FS2 until the bandwidth usage on Router 1 is up at 4Mbit, then
> start using FS3 & FS4 aswell. The load on this cluster is rapidly changing
> so it won't work to put static weigths in the scheduler (at low traffic
> Location 1 would be able to take all traffic but at high traffic 4Mbit isn't
> enough).


For this setup, you probably don't want to use 'VFS1' to distribute
traffic to FS1&2. A good way to handle two distinct internet presences is
to use a type of DNS weighted round-robin. You could either put another
LVS box in front of FS1&2, or just add them individually to DNS. The end
result would be two (or three) records in DNS for your cluster (VFS1, and
either a new VFS2 or just FS1&2).

With DNS weighted round-robin, you can use very short TTL for the records,
and use a monitoring program, in conjunction with Dynamic DNS Updates to
remove records for failed servers, or remove records when the traffic
maxes out at the secondary location. About a days worth of scripting. This
would very roughly approximate what Cisco Distributed Director does.

--
Michael Brown
Linux Systems Group
Dell Computer Corp

You might want to have a look at the kernel
httpd accelerator (khttpd). It should be blazing fast
for serving static content as it intercepts
requests on port 80 for filetypes you define,
and serves them. It only passes the connection
to the webserver when it can't figure out what
to do with a request. The only drawback is that
it doesn't yet do virtual hosts. So let's
say that you configure host.foo.com to have
a dedicated ip, and setup all your pages to do
requests to that ip, then setup ktthpd on all
your real machines, combine that with LVS and
you might see a significant increase in speed.

Another thing you might want to consider is
dropping mysql. I've read that postgresql is
much faster and handles concurrency much
better than mysql. I'm currently starting
to have problems with mysql and I'm going
to switch to postgres shortly. Postgresql
will shortly have database replication support,
which means that you'll be able to distribute
requests to different database servers
throughout your network.




info about ktthpd (from the kernel readme):

1. Introduction
---------------
kHTTPd is a http-daemon (webserver) for Linux. kHTTPd is different
from
other webservers in that it runs from within the Linux-kernel as a
module
(device-driver).

kHTTPd handles only static (file based) web-pages, and passes all
requests
for non-static information to a regular userspace-webserver such as
Apache or
Zeus. The userspace-daemon doesn't have to be altered in any way.

Static web-pages are not a very complex thing to serve, but these are
very
important nevertheless, since virtually all images are static, and a
large
portion of the html-pages are static also. A "regular" webserver has
little
added value for static pages, it is simply a "copy file to
network"-operation.
This can be done very efficiently from within the Linux-kernel, for
example
the nfs (network file system) daemon performs a similar task and also
runs
in the kernel.

By "accelerating" the simple case within the kernel, userspace
daemons can
do what they are very good at: Generating user-specific, dynamic
content.

Note: This document sometimes uses "Apache" instead of "any webserver
you
ever might want to use", just for reasons of readability.


2. Quick Start
--------------

1) compile and load the module
2) configure the module in /proc/sys/net/khttpd if needed
3) echo 1 > /proc/sys/net/khttpd/start

unloading:

echo 1 > /proc/sys/net/khttpd/stop
echo 1 > /proc/sys/net/khttpd/unload
rmmod khttpd



More can be found in /usr/src/linux/net/khttpd/README


Regards,

--
Mark Steele
Vice president research and development
Inet Technologies Inc.
msteele@inet-interactif.com

010110010110111101110101001000000110000101110010011001010010000001100100011101010110110101100010

Johan Isacsson wrote:
>
> Hello Rajiv,
>
> We have actually considered both options for quite some time and for now
> httpd+mysql on same host seems to be the best configuration. I'll explain
> why:
> Each mahine can take up to about 200 http connections with 1 Gb RAM (all
> realservers have 1 Gb RAM). Since we use PHP the http processes are at least
> 5 Mb in size, so we have big memory problems if we go any further than that.
>
> At that load the local MySQLd runs fine having about 200 connections.
>
> In a dedicated database setup we have big problems running over 500
> connections to MySQL. So instead of having, for example, 2 webservers
> sharing 1 database machine and still just get to handle 400 connections we
> choose to have 2 web+db machines that can also take 400 connections.
> That saves us the cost of 1 db mahine per 2 web servers. In total we save 8
> machines, and that's http 1600 connections :)
> If anyone have comments on this i'd love to hear them.
>
> One thing we've been thinking about is to serve static images from separate
> webservers running boa or some other small and efficent webserver. The
> problem with it is that it would make life complicated for the designers and
> webmasters unless there was a way of doing it behind the scenes.
> One scenario could be that the load balancer would look at the requests and
> direct all /images/* to the static virtual server. I know that some
> loadbalancers can do that, how about LVS?
>
> Well, thanks for your comments :)
>
> Regards,
> Johan Isacsson
> MGON
>
> > -----Ursprungligt meddelande-----
> > Från: lvs-users-admin@LinuxVirtualServer.org
> > [mailto:lvs-users-admin@LinuxVirtualServer.org]För Rajiv Khaneja
> > Skickat: den 26 februari 2001 09:05
> > Till: lvs-users@LinuxVirtualServer.org
> > Ämne: RE: Introduction and LVS/DR 2.4 realserver questions
> >
> >
> > Hello,
> >
> > Greetings, I am new to this list as well and would like to offer some
> > friendly advice. Get your MySQL databases on to some dedicated servers.
> > Why run a RDBMS + Webserver on the same machine? You should see some big
> > improvements...
> >
> > Regards,
> > Rajiv Khaneja
> >
> > -----Original Message-----
> > From: lvs-users-admin@LinuxVirtualServer.org
> > [mailto:lvs-users-admin@LinuxVirtualServer.org]On Behalf Of Johan
> > Isacsson
> > Sent: February 25, 2001 10:47 PM
> > To: Lvs-Users
> > Subject: Introduction and LVS/DR 2.4 realserver questions
> >
> >
> > Hello,
> >
> > I'm new to this list so here comes a little introduction.
> > My name is Johan Isacsson and i'm the CTO of MGON International
> > AB, based in
> > Sweden.
> > MGON creates and maintains websites about computer games, traffic is about
> > 25 million impressions/month.
> > Almost all pages are dynamic, so we need to distribute the load
> > over several
> > servers. We're running Linux 2.2 with Apache+PHP and MySQL on all
> > realservers, about 16 in total.
> > We use custom/scp script for file mirroring but are looking for
> > better ways
> > of doing that (any suggestions are welcome).
> > Currently we use LVS/DR to distribute the load and it works very well for
> > us.
> > Before that we used IBM Network Dispatcher which i think worked
> > the same way
> > as LVS/DR do, but it costs lots of money! :P
> > Well, enough about me, on to the questions!
> >
> > 1. Is the redirect approach working (well) on 2.4, and if so, are
> > there any
> > drawbacks using that method compared to the hidden arp method in 2.2?
> > 2. Is it a very bad idea to just have 1 NIC on the LVS machine to receive
> > and forward the requests?
> >
> > It's a DR setup.
> >
> > We have another server location with a 4Mbit connection that we don't use
> > much. We have some servers there serving static files, but i'd
> > like to load
> > balance them between that location and our main location. Like this:
> >
> > Location 1 (Max 4Mbit)
> > Static file servers
> > FS1
> > FS2
> >
> > Location 2
> > Static file servers
> > FS3
> > FS4
> > Virtual server
> > VFS1 (FS1,FS2,FS3,FS4 LVS/TUN)
> >
> > I'd like to use as much as possible of the bandwidth in location 1 (up to
> > 4Mbit) and after that start using the fileservers at location 2. The
> > bandwith usage can be taken from the router at location 1 using snmp.
> > If anyone have suggestions about how to set up scheduling for
> > this i'd love
> > to hear them :)
> >
> > Thanks for your time,
> >
> > Johan Isacsson
> > MGON
> >
> >
> > _______________________________________________
> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > Send requests to lvs-users-request@LinuxVirtualServer.org
> > or go to http://www.in-addr.de/mailman/listinfo/lvs-users
> >
> >
> > _______________________________________________
> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > Send requests to lvs-users-request@LinuxVirtualServer.org
> > or go to http://www.in-addr.de/mailman/listinfo/lvs-users
> >
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users

Just another thought, you might
want to change from php to modperl. Modperl
handles higher loads better. Barring that,
bite the bullet and recode your application
as an apache module written in C. You can't
get any faster than building your applications into
the web server.

I know this has nothing to do with your question, just
a friendly suggestion :)

Johan Isacsson wrote:
>
> Hello Roberto,
>
> > Is impressions==connection requests?
>
> An impression is a loaded banner, and we have one banner per page (that we
> count). So in our case one impression is one pageview.
> We have about 12 hits (requests) per pageview, so that would be 25M+(12*25M)
> = 325 million hits/requests per month.
>
> > You might have a look at http://www.linux-ha.org and check out the
> > links on the first page about Filesystem technologies.
>
> I will, thanks :)
>
> > > 1. Is the redirect approach working (well) on 2.4, and if so,
> > are there any
> > > drawbacks using that method compared to the hidden arp method in 2.2?
> >
> > Yes, it is working well, no, no drawbacks so far, but the difference is
> > not redirect <-> hidden arp for 2.4.x <-> 2.2.x. The difference IMHO is
> > the place where the LVS hooks in and the trespassing of the chains. BTW,
> > why do you intent to change to 2.4.x if you setup is working fine? And I
> > reckon that you run your db on a Linux box with fast SCSI controller or
> > even a RAID. If so I would not advise you to use the 2.4.x kernel since
> > there are still some issues not solved with RAID and SCSI at least back
> > here, where I still try to get it running for more then 10 hours. :(
>
> Thanks for the info, yes we're running SCSI on the realservers and i haven't
> checked if the driver is working on 2.4 yet, so it might be a good idea to
> wait for a while.
>
> > Try it with the lblcr scheduler. Although I actually don't understand
> > your setup correctly, honestly. Maybe someone else understands it better
> > than me or you could draw a sketch.
>
> Ok, here's another attempt:
>
> Location 1 (4 Mbit max)
>
> --------- Lan
> | FS 1 |---
> --------- | -----------
> ---| Router1 |
> --------- | -----------
> | FS 2 |--- |
> --------- |
> Internet------------------
> |
> |
> ------------
> ---| Router 2 |
> | ------------
> -------- |
> | VFS1 |----|
> -------- |
> | Lan
> -------- |
> | FS 3 |----|
> -------- |
> |
> -------- |
> | FS 4 |----
> --------
>
> Router 1 has a limit of 4 Mbit
> VFS1 is the virtual file server
> FS1-4 are the real file servers
>
> Basicly what i want is to load balance between the file servers but put all
> load on FS1 & FS2 until the bandwidth usage on Router 1 is up at 4Mbit, then
> start using FS3 & FS4 aswell. The load on this cluster is rapidly changing
> so it won't work to put static weigths in the scheduler (at low traffic
> Location 1 would be able to take all traffic but at high traffic 4Mbit isn't
> enough).
> I figure i could have a cron job get the bandwidth usage out of router 1 and
> adjust the wieghts for FS3&4 in some way...
> Comments are very welcome :)
>
> > No problem, I hope I could give you some more information than you already
> > had.
>
> Yepp, thanks for your answer :)
>
> Regards,
> Johan Isacsson
> MGON
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users

--
Mark Steele
Vice president research and development
Inet Technologies Inc.
msteele@inet-interactif.com

010110010110111101110101001000000110000101110010011001010010000001100100011101010110110101100010

On Thu, 1 Mar 2001 msteele@inet-interactif.com wrote:

> to switch to postgres shortly. Postgresql
> will shortly have database replication support,
> which means that you'll be able to distribute
> requests to different database servers
> throughout your network.

The latest MySQL (3.23.33) supports replication,
I am using 2 DB servers at the moment which are used for mail
and I have experienced no problems at all.

Just thought I would let you know.

Later,
Warren

On Thu, Mar 01, 2001 at 11:28:52AM +0200, Warren Baker mentioned:
> The latest MySQL (3.23.33) supports replication,
> I am using 2 DB servers at the moment which are used for mail
> and I have experienced no problems at all.

I've had a few problems with it - for some reason, one or other servers
would start logging "can't connect to localhost" and start using 100% cpu.
Only fix was to wipe the databases off one machine, and copy them over
again.

Kate

--
When I say 'free', I mean 'free': free from bond, of chain or command:
to go where you will, even to Mordor, Saruman, if you desire. "
-- Gandalf, paraphrasing the choice between Free and Non-free software

> For this setup, you probably don't want to use 'VFS1' to distribute
> traffic to FS1&2. A good way to handle two distinct internet presences is
> to use a type of DNS weighted round-robin. You could either put another
> LVS box in front of FS1&2, or just add them individually to DNS. The end
> result would be two (or three) records in DNS for your cluster (VFS1, and
> either a new VFS2 or just FS1&2).
>
> With DNS weighted round-robin, you can use very short TTL for the records,
> and use a monitoring program, in conjunction with Dynamic DNS Updates to
> remove records for failed servers, or remove records when the traffic
> maxes out at the secondary location. About a days worth of scripting. This
> would very roughly approximate what Cisco Distributed Director does.

I just like to know why it's a bad idea to do this with LVS/TUN?
And is it possible to use tunneling for one virtual server and direct
routing for another on the same LVS box?

If i add a new VFS2 in front of FS1&2, could VFS1 tunnel requests to VFS2
which in turn balance the requests between FS1&2?

I'm not so sure about the DNS weighted round-robin business, wouldn't it put
high load on the DNS if i get very high traffic?
I imagine that LVS would be far more efficent than any DNS implementation (i
might be wrong).

Regards,
Johan Isacsson
MGON

* msteele@inet-interactif.com (msteele@inet-interactif.com) wrote:
> You might want to have a look at the kernel
> httpd accelerator (khttpd). It should be blazing fast
> for serving static content as it intercepts
> requests on port 80 for filetypes you define,
> and serves them. It only passes the connection
> to the webserver when it can't figure out what
> to do with a request. The only drawback is that
> it doesn't yet do virtual hosts. So let's
If you need virtual hosting and want a "blazing fast" web server I
suggest you check out:
http://people.redhat.com/~mingo/TUX-patches/

You'll need to grab both the kernel patch and the user space module but
it's being actively developed (and I'm not sure khttpd is). Tux also
supports cgi.

> Another thing you might want to consider is
> dropping mysql. I've read that postgresql is
> much faster and handles concurrency much
> better than mysql. I'm currently starting
> to have problems with mysql and I'm going
> to switch to postgres shortly. Postgresql
> will shortly have database replication support,
> which means that you'll be able to distribute
> requests to different database servers
> throughout your network.

Mysql's biggest issue with concurrency is it's lack of row level
locking. It can do replication and again it's being developed a lot
more I think than postresql. The option there is to replicate the
database across several machines and even put them behind LVS. The only
caveat is that you must alter your programs to insert/update/delete only
to master.db.yourdomain.com (which is the master db server). But
selects can go to cluster.db.yourdomain.com (and hit any of the
replicated servers, and even the master depending on your level of
comfort and how often you change the db).

> > We have actually considered both options for quite some time and for now
> > httpd+mysql on same host seems to be the best configuration. I'll explain
> > why:
> > Each mahine can take up to about 200 http connections with 1 Gb RAM (all
> > realservers have 1 Gb RAM). Since we use PHP the http processes are at least
> > 5 Mb in size, so we have big memory problems if we go any further than that.

I'm serving static files with TUX so this may or may not be useful. On
a 2 p3 933 with 2GB RAM and 1 acenic gigabit ethernet card I was able to
synthetically simulate (through httperf on 8 clients) 7120 successful
simultaneous connections (through a 100M port on the switch). Switching
the gigabit card to the gig port we got all 8000 connections to be
handled. Our tests started 1000 simulated requests for a 200K file per
client. I admit, we could have a truer result with more clients and a
more randomized request list.

> > At that load the local MySQLd runs fine having about 200 connections.
> >
> > In a dedicated database setup we have big problems running over 500
> > connections to MySQL. So instead of having, for example, 2 webservers
> > sharing 1 database machine and still just get to handle 400 connections we
> > choose to have 2 web+db machines that can also take 400 connections.
> > That saves us the cost of 1 db mahine per 2 web servers. In total we save 8
> > machines, and that's http 1600 connections :)
> > If anyone have comments on this i'd love to hear them.

200 http connections seems low. In a production environment (dual p3
800, raid, 1GB RAM), serving large files (3 - 10MB) we easily saw 800
simultaneous connections using thttpd (www.acme.com/software/thttpd/)
and likely 500 with apache.
Again all of my content is static, but possible with mod perl as
suggested or php caching you could decrease the work a web server has to
do.

> > One thing we've been thinking about is to serve static images from separate
> > webservers running boa or some other small and efficent webserver. The
> > problem with it is that it would make life complicated for the designers and
> > webmasters unless there was a way of doing it behind the scenes.
> > One scenario could be that the load balancer would look at the requests and
> > direct all /images/* to the static virtual server. I know that some
> > loadbalancers can do that, how about LVS?

Definitely this would speed things up. It shouldn't make life that
complicated for people, just use absolute paths for images, something
like <img src=http://images.mydomain.com/images/someimage.png>. Then
have images.mydomain.com point to an LVS cluster of webservers running
tux or thttpd serving images, or just a single machine. As long as your
developers/web masters get in a habit of using a name
(images.mydomain.com) you can change it from a single server to lvs
through DNS or /etc/hosts and you'll be fine. Obviously you'll have to
keep your content synced across servers (but you could have a simple
script making that part easier for developers).

> > > servers. We're running Linux 2.2 with Apache+PHP and MySQL on all
> > > realservers, about 16 in total.

You should also see quite an improvement going from linux 2.2.x to 2.4.2
(or 1).

> > > We use custom/scp script for file mirroring but are looking for
> > > better ways

This takes care of images.yourdomain.com problem.
I've played with this problem a lot though, and if you want to use scp
like tool I'd say rsync, I'm a little more comfortable making sure
everything is in sync rather than hoping it all got copied. Another
option is to have a master server somewhere, and have all your client
machines wget -m (to mirror) or -N (use timestamping to only get
changes) the files they need. Then your script could be:

ssh image1.mydomain.com "updateImages.sh"

You could have it record it's success to a file or a database and just
have another script check the database for any problems (if it could
report a problem), or any process that didn't report finish after it
reported start.

From what I've seen wget is a lot faster than rsync, and even when one
of our servers is maxed out and I can't get an rsync to accomplish I can
ssh in and wget the needed files.


Not really sure who's email I ended up addressing, but that's my 2 cents
worth.

Jeffrey Schoolcraft

On Thu, 1 Mar 2001, Johan Isacsson wrote:

> I just like to know why it's a bad idea to do this with LVS/TUN?
> And is it possible to use tunneling for one virtual server and direct
> routing for another on the same LVS box?

You are using 4mbit of throughput not only on the secondary site, you are
using 8mbit on your primary site. Waste of bandwidth. (4mbit coming into
the primary site, being redirected back out to secondary site.) Also, your
latency will be quite high with all of those hops.

>
> If i add a new VFS2 in front of FS1&2, could VFS1 tunnel requests to VFS2
> which in turn balance the requests between FS1&2?

Not a good idea for the same reason mentioned above.

>
> I'm not so sure about the DNS weighted round-robin business, wouldn't it put
> high load on the DNS if i get very high traffic?

DNS will be loaded _exactly_ the same in the DNS round-robin case as it is
_right now_.

DNS is the least of your problems, here. For every connection to your
servers, you will get _one_ DNS query packet and _one_ DNS reply packet. A
total of _two_ packets for every http request (or less because of
caching). Clients are _already_ using DNS to get your address, you won't
be changing anything there.

> I imagine that LVS would be far more efficent than any DNS implementation (i
> might be wrong).

I think the latency will be much higher if you let LVS redirect every
packet of every connection over a slow link. DNS lets the clients connect
directly to each server.

--
Michael Brown
Linux Systems Group
Dell Computer

Hey Michael,

> You are using 4mbit of throughput not only on the secondary site, you are
> using 8mbit on your primary site. Waste of bandwidth. (4mbit coming into
> the primary site, being redirected back out to secondary site.) Also, your
> latency will be quite high with all of those hops.

....

> I think the latency will be much higher if you let LVS redirect every
> packet of every connection over a slow link. DNS lets the clients connect
> directly to each server.

I though that LVS/TUN worked worked the same way as LVS/DR, where the LVS
forwards the request on to a realserver which answers back directly to the
client?
If that's true then only the initial request will go to the primary site,
then out and in to the secondary site.
After that the client will communicate directly with the realserver at the
secondary location.
Or am i completely wrong here?

All files serverd at the secondary location are big ones (downloads of demos
etc) so the request is a very small part of the whole transfer.
We're wasting some bandwidth here but i imagine it can't be that much?

Regards,
Johan Isacsson
MGON

On Thu, 1 Mar 2001, Johan Isacsson wrote:

> Hey Michael,
>
> > You are using 4mbit of throughput not only on the secondary site, you are
> > using 8mbit on your primary site. Waste of bandwidth. (4mbit coming into
> > the primary site, being redirected back out to secondary site.) Also, your
> > latency will be quite high with all of those hops.
>
> ....
>
> > I think the latency will be much higher if you let LVS redirect every
> > packet of every connection over a slow link. DNS lets the clients connect
> > directly to each server.
>
> I though that LVS/TUN worked worked the same way as LVS/DR, where the LVS
> forwards the request on to a realserver which answers back directly to the
> client?
> If that's true then only the initial request will go to the primary site,
> then out and in to the secondary site.
> After that the client will communicate directly with the realserver at the
> secondary location.
> Or am i completely wrong here?

Ok, I am slightly wrong, and you are slightly wrong :-)
In LVS/TUN the packet flow is asymetric. Traffic from the client to the
server will _always_ go through the LVS server. Return traffic will not.

In your case actual data from the server will go directly to the client,
but the periodic 'acks' that the client sends will go through the LVS
server.

So, you will experience latency in the connection. The ratio of
(Server-to-client-bytes) to (client-to-server-bytes) is very high, but if
you look at the (server-to-client-packets) to (client-to-server-packets)
you will see a very close to 1-1 ratio. This will have a major impact on
your latency. This is because the server has to wait for an 'ack' before
transmitting any more data, and the 'ack' has to travel the long way to
get there.

>
> All files serverd at the secondary location are big ones (downloads of demos
> etc) so the request is a very small part of the whole transfer.
> We're wasting some bandwidth here but i imagine it can't be that much?

Why not have them listed as separate servers (demos.yourdomain.com) and
have the href point there
(http:/demos.yourdomain.com/images/somebigfile.mpg). Then you can setup an
LVS server at the secondary site that pretends to be demos.yourdomain.com.
You won't have the difficulty of setting up DNS, and you wont have the
traffic problems of trying to route there from your primary LVS.

--
Michael Brown