Mailing List Archive

Eivind Bengtsson <ebe at dmi.dk> writes:
> How many open TCP-connections can a standard linuxbox 64bit 2.6.17
> kernel - GB ethernet interface... When is it okay for me to start
> worrying? 1000 connections, 2000, 3 ?

It depends on many factors (hardware, OS version, application, access
pattern etc.) so you'll have to find out for yourself. You'll have to
increase thread_pool_max, as the default is 1000.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Do I understand you correctly

I need to set -p thread_pool_max = 1500
How many thread pools do I have ? (default?)

Howcome I can have more than 1500 connections now, without changing the
Thread_pool_max ?

The only change I've made is to ulimit -n 131072
(which is default in the debian etch 1.1.1 package ...)

Thanks again for any help :-)

/Eivind

Dag-Erling Sm?rgrav wrote:
> Eivind Bengtsson <ebe at dmi.dk> writes:
>> How many open TCP-connections can a standard linuxbox 64bit 2.6.17
>> kernel - GB ethernet interface... When is it okay for me to start
>> worrying? 1000 connections, 2000, 3 ?
>
> It depends on many factors (hardware, OS version, application, access
> pattern etc.) so you'll have to find out for yourself. You'll have to
> increase thread_pool_max, as the default is 1000.
>
> DES


--
Eivind Bengtsson
Systemadministrator - Cand.merc.(dat)
Danmarks Meteorologiske Institut
Lyngbyvej 100
2100 K?benhavn ?
Direkte tlf. : 39157544
Email: ebe at dmi.dk
echo 'This is not a pipe.' | cat -> /dev/tty

Eivind Bengtsson <ebe at dmi.dk> writes:
> Do I understand you correctly
>
> I need to set -p thread_pool_max = 1500

yes, or use the management interface to change it (but you will have
to restart the child)

> How many thread pools do I have ? (default?)

1

> Howcome I can have more than 1500 connections now, without changing
> the Thread_pool_max ?

because 500 of them are in the overflow queue waiting for a worker
thread to become available. The size of the overflow queue relative
to thread_pool_max is governed by overflow_max (default is 100%).

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

Hello again

Dag-Erling Sm?rgrav wrote:
> Eivind Bengtsson <ebe at dmi.dk> writes:
>> Do I understand you correctly
>>
>> I need to set -p thread_pool_max = 1500
>
> yes, or use the management interface to change it (but you will have
> to restart the child)
I'll change it in the initscript anyway ...

>
>> How many thread pools do I have ? (default?)
>
> 1
Can I just raise the number of thread pools ? - what is the penalty ?

>
>> Howcome I can have more than 1500 connections now, without changing
>> the Thread_pool_max ?
>
> because 500 of them are in the overflow queue waiting for a worker
> thread to become available. The size of the overflow queue relative
> to thread_pool_max is governed by overflow_max (default is 100%).
Okay - tricky ...
>
> DES

Thanks
/Eivind

--
Eivind Bengtsson
Systemadministrator - Cand.merc.(dat)
Danmarks Meteorologiske Institut
Lyngbyvej 100
2100 K?benhavn ?
Direkte tlf. : 39157544
Email: ebe at dmi.dk
echo 'This is not a pipe.' | cat -> /dev/tty

Eivind Bengtsson <ebe at dmi.dk> writes:
> Dag-Erling Sm?rgrav wrote:
> > Eivind Bengtsson <ebe at dmi.dk> writes:
> > > How many thread pools do I have ? (default?)
> > 1
> Can I just raise the number of thread pools ? - what is the penalty ?

From varnishd(1):

thread_pools
The number of worker thread pools. Higher values reduce lock con-
tention but increase pressure on the scheduler. Note that a
decrease of this parameter will only take effect after a restart.

The default is 1.

Note that thread_pool_max is the total number of threads, not the
number of threads per pool.

One thing the man page doesn't mention is that connections are
assigned to pools by computing (fd % thread_pools), so increasing the
number of pools will also increase the likelihood that a connection
will remain on the overflow queue even if there are threads available
in other pools. I don't know where the cutoff is, but it's likely to
be at or below sqrt(thread_pool_max).

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

In message <ujrbqanfib4.fsf at false.linpro.no>, =?iso-8859-1?Q?Dag-Erling_Sm=F8rg
rav?= writes:

>One thing the man page doesn't mention is that connections are
>assigned to pools by computing (fd % thread_pools), so increasing the
>number of pools will also increase the likelihood that a connection
>will remain on the overflow queue even if there are threads available
>in other pools. I don't know where the cutoff is, but it's likely to
>be at or below sqrt(thread_pool_max).

In retrospect, I am not convinced that idea/implementation of
multiple thread pools is sufficiently sound. Its one area that
we will have to impove somehow.

Under no circumstances should more than a few pools (< 5) be created.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
> In retrospect, I am not convinced that idea/implementation of
> multiple thread pools is sufficiently sound. Its one area that
> we will have to impove somehow.
>
> Under no circumstances should more than a few pools (< 5) be created.

I wonder...

Why do we have a complex setup with pools, workers within pools etc.?
If we just used the socket descriptor as a direct index into an array
of workers, no locking would be necessary since the descriptor itself
would act as a semaphore. Since the kernel always picks the lowest-
numbered descriptor available, we would always reuse the most recently
used thread, or almost always; using a socket descriptor for a backend
requests would lock out the corresponding worker for the duration of
the request, but that should not be an issue as long as the cache hit
ratio is reasonably high.

DES
--
Dag-Erling Sm?rgrav
Senior Software Developer
Linpro AS - www.linpro.no

In message <ujrabq66r5l.fsf at false.linpro.no>, =?iso-8859-1?Q?Dag-Erling_Sm=F8rg
rav?= writes:
>"Poul-Henning Kamp" <phk at phk.freebsd.dk> writes:
>> In retrospect, I am not convinced that idea/implementation of
>> multiple thread pools is sufficiently sound. Its one area that
>> we will have to impove somehow.
>>
>> Under no circumstances should more than a few pools (< 5) be created.
>
>I wonder...
>
>Why do we have a complex setup with pools, workers within pools etc.?

The idea was to reduce lock contention on the pool mutex.

>If we just used the socket descriptor as a direct index into an array
>of workers, no locking would be necessary since the descriptor itself
>would act as a semaphore.

The majority of time the descriptor is idle, and your proposal essentially
parks one thread per descriptor, so if we did that, a lot of code could
be simplified, but at a massive expansion in the number of threads and
a drastic reduction in CPU cache efficiency.

--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.