Mailing List Archive

> It turns out that NT has a default 64 fd limit on arguments to
> select(). The good news is that you can actually bump the limit up
> to whatever number you want by specifying a define when compiling
> python15.dll.
>
> If you have the ability to rebuild your python15.dll, you can add
> the define:
>
> FD_SETSIZE=1024
>
> to the preprocessor options for the python15 project to raise the
> limit to 1024 fds.
>
> The default 64 fd limit is too low for anyone trying to run
> an async server that handles even a modest load, so I've
> submitted a bug report to python.org asking that the define
> above find its way into the next python release...

Brian,

(Also in response to your bug report.) I'm a little worried that
upping the limit to 1024 would cause some performance problems if
you're making a lot of select() calls. The select allocates three
arrays of length FD_SETSIZE+3; each array item is 12 bytes. This is a
total allocation of more than 36K for a meager select() call! And all
that memory also has to be cleared by the FD_ZERO() call.

If you actually have that many sockets, that's worth paying for (the
socket objects themselves use up just as much memory, and your Python
data structures for the sockets, no matter how small, are probably
several times bigger), but for a more typical program, I see this as a
lot of overhead.

Is there a way that this can be done more dynamically, e.g. by making
the set size as big as needed on windows but no bigger?

(Before you suggest allocating that memory statically, remember it's
possible to call select from multiple threads. Allocating 36K of
thread-local space for each thread also doesn't sound too pleasant.)

--Guido van Rossum (home page: http://www.python.org/~guido/)

Guido wrote:
>
> Brian,
>
> (Also in response to your bug report.) I'm a little worried that
> upping the limit to 1024 would cause some performance problems if
> you're making a lot of select() calls. The select allocates three
> arrays of length FD_SETSIZE+3; each array item is 12 bytes. This is a
> total allocation of more than 36K for a meager select() call!
> And all
> that memory also has to be cleared by the FD_ZERO() call.
>
> If you actually have that many sockets, that's worth paying for (the
> socket objects themselves use up just as much memory, and your Python
> data structures for the sockets, no matter how small, are probably
> several times bigger), but for a more typical program, I see
> this as a
> lot of overhead.
>
> Is there a way that this can be done more dynamically, e.g. by making
> the set size as big as needed on windows but no bigger?
>
> (Before you suggest allocating that memory statically, remember it's
> possible to call select from multiple threads. Allocating 36K of
> thread-local space for each thread also doesn't sound too pleasant.)
>
> --Guido van Rossum (home page: http://www.python.org/~guido/)

Hmm - after going through all of the Win32 sdks, it doesn't
appear to be possible to do it any other way than as a -D
option at compile time, so optimizing for the common case
(folks who _don't_ need large numbers of fds) is reasonable.

Since we distribute a python15.dll with Zope on windows, this
isn't that big a deal for us - we can just compile in a higher
limit in our distributed dll. I was mostly thinking of the win32
users who don't have the ability to rebuild their dll, but
maybe this isn't that much of a problem; I suspect that the
people who write significant socket apps that would run into
this problem probably have access to a compiler if they need it.


Brian Lloyd brian@digicool.com
Software Engineer 540.371.6909
Digital Creations http://www.digicool.com

On Fri, 30 Jul 1999, Brian Lloyd wrote:

> Since we distribute a python15.dll with Zope on windows, this
> isn't that big a deal for us - we can just compile in a higher
> limit in our distributed dll. I was mostly thinking of the win32
> users who don't have the ability to rebuild their dll, but
> maybe this isn't that much of a problem; I suspect that the
> people who write significant socket apps that would run into
> this problem probably have access to a compiler if they need it.

It's a worthy piece of knowledge to document somehow -- I'm not sure where
that should be...

David Ascher writes:
> It's a worthy piece of knowledge to document somehow -- I'm not sure where
> that should be...

Perhaps a paragraph in the library reference? If someone can send
along a clear bit of text (unformatted is fine), I'll be glad to add
it.


-Fred

--
Fred L. Drake, Jr. <fdrake@acm.org>
Corporation for National Research Initiatives