Mailing List Archive

Markus Stenberg wrote:

> Is there some low limit on maximum number of sockets you can have in
> the Python-NT's select call? A program that happens to work
> perfectly on Linux seems to die on NT around 64(?) sockets to the
> 'too many file descriptors in call' error.

Correct. Windows has the same limit (64) in the native
WaitForMultipleObjects calls.

> Any portable ways to bypass it?

Doubt it. A Windows developer would probably have N threads each
handling M sockets.

FWIW, HPUX comes with 3 different flavors of sockets and select. One
version has a low limit, the second a moderate limit and the third
has a limit so huge I can't imagine testing it.

- Gordon

> Is there some low limit on maximum number of sockets you can
> have in the
> Python-NT's select call? A program that happens to work
> perfectly on Linux
> seems to die on NT around 64(?) sockets to the 'too many file
> descriptors
> in call' error.
>
> Any portable ways to bypass it?
>
> -Markus

Hi Markus,

It turns out that NT has a default 64 fd limit on arguments to
select(). The good news is that you can actually bump the limit up
to whatever number you want by specifying a define when compiling
python15.dll.

If you have the ability to rebuild your python15.dll, you can add
the define:

FD_SETSIZE=1024

to the preprocessor options for the python15 project to raise the
limit to 1024 fds.

The default 64 fd limit is too low for anyone trying to run
an async server that handles even a modest load, so I've
submitted a bug report to python.org asking that the define
above find its way into the next python release...


Brian Lloyd brian@digicool.com
Software Engineer 540.371.6909
Digital Creations http://www.digicool.com

Brian Lloyd <Brian@digicool.com> writes:
> > Is there some low limit on maximum number of sockets you can
> > have in the
> > Python-NT's select call? A program that happens to work
> > perfectly on Linux
> > seems to die on NT around 64(?) sockets to the 'too many file
> > descriptors
> > in call' error.
> >
> > Any portable ways to bypass it?
> >
> > -Markus
> It turns out that NT has a default 64 fd limit on arguments to
> select(). The good news is that you can actually bump the limit up
> to whatever number you want by specifying a define when compiling
> python15.dll.

I was hoping for version that works with 'Python out of the box' (I kludged
something in pure python with threads that bypasses this limit, but it
isn't pretty).

> The default 64 fd limit is too low for anyone trying to run
> an async server that handles even a modest load, so I've
> submitted a bug report to python.org asking that the define
> above find its way into the next python release...

Agreed. Unless it causes some Huge Load(tm) somewhere, it should be
definitely raised. (say, 1024 or 4096 sounds reasonable to me)

> Brian Lloyd brian@digicool.com
> Software Engineer 540.371.6909
> Digital Creations http://www.digicool.com

-Markus

--
"...very few phenomena can pull someone out of Deep Hack Mode, with two
noted exceptions: being struck by lightning, or worse, your *computer*
being struck by lightning."
- Matt Welsh

It does cause a huge load, because in Winsock an fd_set is represented as an
array of SOCKETs, not as a bitmap. So in a normal program, which uses
select for simple things like polling a single FD, a large FD_SETSIZE will
degrade performance. For example, if FD_SETSIZE were 4096, then
select.select() would allocate about 48KB (~16KB per fd_set) on the stack
whenever it is called, even if there are just a few calls.

The good news is that, since those fake fd_sets are always passed by
reference, it is safe for the core Python to define a small FD_SETSIZE like
64 while some module overrides the builtin select with another that defines
a larger FD_SETSIZE. Then no changes to "core" python would be required,
and only those who want the bigger fd_sets will get them.

Another thing that you can do is just dynamically allocate an array of
32-bit integers: the first holds the count, and the others hold the SOCKETs.
You can fill this without using the FD_SET macros if you are careful to
avoid duplicate FDs, or you can define FD_SETSIZE to be huge, cast the
allocated memory to fd_set *, and use the macros. (It won't matter that
your buffer is smaller than FD_SETSIZE, because you know you're going to run
out of elements before you run out of buffer.) If your malloc does a
reasonable job of handling small and large objects, and of re-using memory
quickly, this could be pretty much ideal.

Bruce