Mailing List Archive: ActiveState & fork & Perl

ActiveState & fork & Perl

Jun 7, 1999, 9:43 AM

Post #1 of 93 (9364 views)

In case you haven't heard about it, ActiveState has recently signed a
contract with Microsoft to do some work on Perl on win32.

One interesting aspect of this for Python is the specific work being
performed. From the FAQ on this joint effort, one gets, under "What is
the scope of the work that is being done?":

fork()

This implementation of fork() will clone the running interpreter
and create a new interpreter with its own thread, but running in the
same process space. The goal is to achieve functional equivalence to
fork() on UNIX systems without suffering the performance hit of the
process creation overhead on Win32 platforms.

Emulating fork() within a single process needs the ability to run
multiple interpreters concurrently in separate threads. Perl version
5.005 has experimental support for this in the form of the PERL_OBJECT
build option, but it has some shortcomings. PERL_OBJECT needs a C++
compiler, and currently only works on Windows. ActiveState will be
working to provide support for revamped support for the PERL_OBJECT
functionality that will run on every platform that Perl will build on,
and will no longer require C++ to work. This means that other operating
systems that lack fork() but have support for threads (such as VMS and
MacOS) will benefit from this aspect of the work.

Any guesses as to whether we could hijack this work if/when it is released
as Open Source?

--david

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 9:49 AM

Post #2 of 93 (9255 views)

> In case you haven't heard about it, ActiveState has recently signed a
> contract with Microsoft to do some work on Perl on win32.

Have I ever heard of it! :-) David Grove pulled me into one of his
bouts of paranoia. I think he's calmed down for the moment.

> One interesting aspect of this for Python is the specific work being
> performed. From the FAQ on this joint effort, one gets, under "What is
> the scope of the work that is being done?":
>
> fork()
>
> This implementation of fork() will clone the running interpreter
> and create a new interpreter with its own thread, but running in the
> same process space. The goal is to achieve functional equivalence to
> fork() on UNIX systems without suffering the performance hit of the
> process creation overhead on Win32 platforms.
>
> Emulating fork() within a single process needs the ability to run
> multiple interpreters concurrently in separate threads. Perl version
> 5.005 has experimental support for this in the form of the PERL_OBJECT
> build option, but it has some shortcomings. PERL_OBJECT needs a C++
> compiler, and currently only works on Windows. ActiveState will be
> working to provide support for revamped support for the PERL_OBJECT
> functionality that will run on every platform that Perl will build on,
> and will no longer require C++ to work. This means that other operating
> systems that lack fork() but have support for threads (such as VMS and
> MacOS) will benefit from this aspect of the work.
>
> Any guesses as to whether we could hijack this work if/when it is released
> as Open Source?

When I saw this, my own response was simply "those poor Perl suckers
are relying too much of fork()." Am I wrong, and is this also a habit
of Python programmers?

Anyway, I doubt that we coould use their code, as it undoubtedly
refers to reimplementing fork() at the Perl level, not at the C level
(which would be much harder).

--Guido van Rossum (home page: http://www.python.org/~guido/)

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 9:51 AM

Post #3 of 93 (9234 views)

On Mon, 7 Jun 1999, Guido van Rossum wrote:

> When I saw this, my own response was simply "those poor Perl suckers
> are relying too much of fork()." Am I wrong, and is this also a habit
> of Python programmers?

Well, I find the fork() model to be a very simple one to use, much easier
to manage than threads or full-fledged IPC. So, while I don't rely on it
in any crucial way, it's quite convenient at times.

--david

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 9:56 AM

Post #4 of 93 (9245 views)

> Well, I find the fork() model to be a very simple one to use, much easier
> to manage than threads or full-fledged IPC. So, while I don't rely on it
> in any crucial way, it's quite convenient at times.

Can you give a typical example where you use it, or is this just a gut
feeling?

It's also dangerous -- e.g. unexpected errors may percolate down the
wrong stack (many mailman bugs had to do with forking), GUI apps
generally won't be cloned, and some extension libraries don't like to
be cloned either (e.g. ILU).

--Guido van Rossum (home page: http://www.python.org/~guido/)

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 10:02 AM

Post #5 of 93 (9246 views)

On Mon, 7 Jun 1999, Guido van Rossum wrote:

> Can you give a typical example where you use it, or is this just a gut
> feeling?

Well, the latest example was that I wanted to spawn a Python process to do
viewing of NumPy arrays with Tk from within the Python interactive shell
(without using a shell wrapper). It's trivial with a fork(), and
non-trivial with threads. The solution I had to finalize on was to branch
based on OS and do threads where threads are available and fork()
otherwise. Likely 2.05 times as many errors as with a single solution =).

> It's also dangerous -- e.g. unexpected errors may percolate down the
> wrong stack (many mailman bugs had to do with forking), GUI apps
> generally won't be cloned, and some extension libraries don't like to
> be cloned either (e.g. ILU).

More dangerous than threads? Bwaaahaahaa! =). fork() might be
"deceivingly simple in appearance", I grant you that. But sometimes
that's good enough.

It's also possible that fork() without all of its process-handling
relatives isn't useful enough to warrant the effort.

--david

Re: ActiveState & fork & Perl [ In reply to ]

bwarsaw at cnri

Jun 7, 1999, 10:05 AM

Post #6 of 93 (9239 views)

>>>>> "Guido" == Guido van Rossum <guido@cnri.reston.va.us> writes:

Guido> It's also dangerous -- e.g. unexpected errors may percolate
Guido> down the wrong stack (many mailman bugs had to do with
Guido> forking), GUI apps generally won't be cloned, and some
Guido> extension libraries don't like to be cloned either
Guido> (e.g. ILU).

Rambling mode on...

Okay, so you can't guarantee that fork will be everywhere you might
want to run an application. For example, that's one of the main
reasons Mailman hasn't been ported off of Un*x. But you also can't
guarantee that threads will be everywhere either. One of the things
I'd (eventually) like to do is to re-architect Mailman so that it uses
a threaded central server instead of the current one-shot process
model. But there's been debate among the developers because 1)
threads aren't supported everywhere, and 2) thread support isn't
built-in by default anyway.

I wonder if it's feasible or useful to promote threading support in
Python? Thoughts would include building threads in by default if
possible on the platform, integrating Greg's free threading mods,
etc. Providing more integrated support for threads might encourage
programmers to reach for that particular tool instead of fork, which
is crude, but pretty damn handy and easy to use.

Rambling mode off...

-Barry

Re: ActiveState & fork & Perl [ In reply to ]

jim at digicool

Jun 7, 1999, 10:07 AM

Post #7 of 93 (9240 views)

David Ascher wrote:
>
> On Mon, 7 Jun 1999, Guido van Rossum wrote:
>
> > When I saw this, my own response was simply "those poor Perl suckers
> > are relying too much of fork()." Am I wrong, and is this also a habit
> > of Python programmers?
>
> Well, I find the fork() model to be a very simple one to use, much easier
> to manage than threads or full-fledged IPC. So, while I don't rely on it
> in any crucial way, it's quite convenient at times.

Interesting. I prefer threads because they eliminate the *need*
for an IPC. I find locks and the various interesting things you can build
from them to be much easier to deal with and more elegant than IPC.
I wonder if the perl folks are also going to emulate doing IPC in the
same process. Hee hee. :)

Jim

--
Jim Fulton mailto:jim@digicool.com Python Powered!
Technical Director (888) 344-4332 http://www.python.org
Digital Creations http://www.digicool.com http://www.zope.org

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 10:10 AM

Post #8 of 93 (9237 views)

On Mon, 7 Jun 1999, Barry A. Warsaw wrote:

> I wonder if it's feasible or useful to promote threading support in
> Python? Thoughts would include building threads in by default if
> possible on the platform,

That seems a good idea to me. It's a relatively safe thing to enable by
default, no?

> Providing more integrated support for threads might encourage
> programmers to reach for that particular tool instead of fork, which
> is crude, but pretty damn handy and easy to use.

While we're at it, it'd be nice if we could provide a better answer when
someone asks (as "they" often do) "how do I program with threads in
Python" than our usual "the way you'd do it in C". Threading tutorials
are very hard to come by, I've found (I got the ORA multi-threaded
programming in win32, but it's such a monster I've barely looked at it). I
suggest that we allocate about 10% of TimBot's time to that task. If
necessary, we can upgrade it to a dual-CPU setup. With Greg's threading
patches, we could even get it to run on both CPUs efficiently. It could
write about itself. <unplug>

--david

Re: ActiveState & fork & Perl [ In reply to ]

akuchlin at mems-exchange

Jun 7, 1999, 10:20 AM

Post #9 of 93 (9244 views)

David Ascher writes:
>While we're at it, it'd be nice if we could provide a better answer when
>someone asks (as "they" often do) "how do I program with threads in
>Python" than our usual "the way you'd do it in C". Threading tutorials
>are very hard to come by, I've found (I got the ORA multi-threaded

Agreed; I'd love to see a HOWTO on thread programming. I really
liked Andrew Birrell's introduction to threads for Modula-3; see
http://gatekeeper.dec.com/pub/DEC/SRC/research-reports/abstracts/src-rr-035.html
(Postscript and PDF versions available.) Translating its approach to
Python would be an excellent starting point.

--
A.M. Kuchling http://starship.python.net/crew/amk/
"If you had stayed with us, we could have given you life until death."
"Don't I get that anyway?"
-- Stheno and Lyta Hall, in SANDMAN #61: "The Kindly Ones:5"

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 10:24 AM

Post #10 of 93 (9246 views)

> David Ascher writes:
> >While we're at it, it'd be nice if we could provide a better answer when
> >someone asks (as "they" often do) "how do I program with threads in
> >Python" than our usual "the way you'd do it in C". Threading tutorials
> >are very hard to come by, I've found (I got the ORA multi-threaded

Andrew Kuchling chimes in:
> Agreed; I'd love to see a HOWTO on thread programming. I really
> liked Andrew Birrell's introduction to threads for Modula-3; see
> http://gatekeeper.dec.com/pub/DEC/SRC/research-reports/abstracts/src-rr-035.html
> (Postscript and PDF versions available.) Translating its approach to
> Python would be an excellent starting point.

Another idea is for someone to finish the thread tutorial that I
started early 1998 (and never finished because I realized that it
needed the threading module and some thread-safety patches to urllib
for the examples I had in mind to work). It's actually on the website
(but unlinked-to): http://www.python.org/doc/essays/threads.html

--Guido van Rossum (home page: http://www.python.org/~guido/)

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 10:28 AM

Post #11 of 93 (9241 views)

Indeed, it might be better to start with the threading module for the
first tutorial. While I'm also a fan of Birrell's paper, it would
encourage people to start with the low-level thread module, instead of
the higher-level threading module.

So the right answer, of course, is to do both!

Jeremy

Re: ActiveState & fork & Perl [ In reply to ]

bwarsaw at cnri

Jun 7, 1999, 10:36 AM

Post #12 of 93 (9237 views)

>>>>> "DA" == David Ascher <da@ski.org> writes:

>> I wonder if it's feasible or useful to promote threading
>> support in Python? Thoughts would include building threads in
>> by default if possible on the platform,

DA> That seems a good idea to me. It's a relatively safe thing to
DA> enable by default, no?

Don't know how hard it would be to write the appropriate configure
tests, but then again, if it was easy I'd'a figured Guido would have
done it already.

A simple thing would be to change the default sense of "Do we build in
thread support?". Make this true by default, and add a
--without-threads configure flag people can use to turn them off.

-Barry

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 3:37 PM

Post #13 of 93 (9244 views)

BAW> A simple thing would be to change the default sense of "Do we build
BAW> in thread support?". Make this true by default, and add a
BAW> --without-threads configure flag people can use to turn them off.

True enough, but as Guido pointed out, enabling threads by default would
immediately make the Mac a second-class citizen. Test cases and demos would
eventually find their way into the distribution that Mac users could not
run, etc., etc. It may not account for a huge fraction of the Python
development seats, but it seems a shame to leave it out in the cold. Has
there been an assessment of how hard it would be to add thread support to
the Mac? On a scale of 1 to 10 (1: we know how, but it's not implemented
because nobody's needed it so far, 10: drilling for oil on the sun would be
easier), how hard would it be? I assume Jack Jansen is on this list. Jack,
any thoughts? Alpha code? Pre-alpha code?

Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/
skip@mojam.com | Musi-Cal: http://www.musi-cal.com/
518-372-5583

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 3:43 PM

Post #14 of 93 (9237 views)

On Mon, 7 Jun 1999, Skip Montanaro wrote:

> True enough, but as Guido pointed out, enabling threads by default would
> immediately make the Mac a second-class citizen. Test cases and demos would
> eventually find their way into the distribution that Mac users could not
> run, etc., etc. It may not account for a huge fraction of the Python
> development seats, but it seems a shame to leave it out in the cold.

I'm not sure I buy that argument. There are already thread demos in the
current directory, and no one complains. The windows builds are already
threaded by default, and it's not caused any problems that I know of.
Think of it like enabling the *new* module. =)

> Has there been an assessment of how hard it would be to add thread
> support to the Mac?

That's an interesting question, especially since ActiveState lists it as a
machine w/ threads and w/o fork().

--david

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 3:49 PM

Post #15 of 93 (9239 views)

David> I'm not sure I buy that argument. Think of it like enabling the
David> *new* module. =)

That's not quite the same thing. The new module simply exposes some
normally closed-from-Python-code data structures to the Python programmer.
Enabling threads requires some support from the underlying runtime system.
If that was already in place, I suspect the Mac binaries would come with the
thread module enabled by default, yes?

Skip

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 3:58 PM

Post #16 of 93 (9239 views)

On Mon, 7 Jun 1999, Skip Montanaro wrote:

> That's not quite the same thing. The new module simply exposes some
> normally closed-from-Python-code data structures to the Python programmer.
> Enabling threads requires some support from the underlying runtime system.
> If that was already in place, I suspect the Mac binaries would come with the
> thread module enabled by default, yes?

I'm not denying that. It's just that there are lots of things which fall
into that category, like (to take a pointed example =), os.fork(). We
don't have a --with-fork configure flag. We expose to the Python
programmer all of the underlying OS that is 'wrapped' as long as it's
reasonably portable. I think that most unices + win32 is a reasonable
approximation of 'reasonably portable'. And in fact, this change might
motivate someone with Mac fervor to explore adding Python support of Mac
threads.

--david

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 4:06 PM

Post #17 of 93 (9228 views)

David> I think that most unices + win32 is a reasonable approximation of
David> 'reasonably portable'. And in fact, this change might motivate
David> someone with Mac fervor to explore adding Python support of Mac
David> threads.

One can hope... ;-)

Skip

RE: ActiveState & fork & Perl [ In reply to ]

MHammond at skippinet

Jun 7, 1999, 4:06 PM

Post #18 of 93 (9241 views)

> > In case you haven't heard about it, ActiveState has
> recently signed a
> > contract with Microsoft to do some work on Perl on win32.
>
> Have I ever heard of it! :-) David Grove pulled me into one of his
> bouts of paranoia. I think he's calmed down for the moment.

It sounds like a :-), but Im afraid I dont understand that reference.

When I first heard this, two things sprung to mind:
a) Why shouldnt Python push for a similar deal?
b) Something more interesting in the MS/Python space is happening anyway,
so nyah nya nya ;-)

Getting some modest funds to (say) put together and maintain single
core+win32 installers to place on the NT resource kit could only help
Python.

Sometimes I wish we had a few less good programmers, and a few more good
marketting type people ;-)

> Anyway, I doubt that we coould use their code, as it undoubtedly
> refers to reimplementing fork() at the Perl level, not at the C level
> (which would be much harder).

Excuse my ignorance, but how hard would it be to simulate/emulate/ovulate
fork using the Win32 extensions? Python has basically all of the native
Win32 process API exposed, and writing a "fork" in Python that only forked
Python scripts (for example) may be feasable and not too difficult.

It would have obvious limitations, including the fact that it is not
available standard with Python on Windows (just like a working popen now
:-) but if we could follow the old 80-20 rule, and catch 80% of the uses
with 20% of the effort it may be worth investigating.

My knowledge of fork is limited to muttering "something about cloning the
current process", so I may be naive in the extreme - but is this feasible?

Mark.

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 4:10 PM

Post #19 of 93 (9249 views)

David Ascher wrote:
>...
> I'm not denying that. It's just that there are lots of things which fall
> into that category, like (to take a pointed example =), os.fork(). We
> don't have a --with-fork configure flag. We expose to the Python
> programmer all of the underlying OS that is 'wrapped' as long as it's
> reasonably portable. I think that most unices + win32 is a reasonable
> approximation of 'reasonably portable'. And in fact, this change might
> motivate someone with Mac fervor to explore adding Python support of Mac
> threads.

Agreed. Python isn't a least-common-demoninator language. It tries to
make things easy for people. Why should we kill all platforms because of
a lack on one? Having threads by default will make a lot of things much
simpler (in terms of knowing the default platform). Can't tell you how
many times I curse to find that the default RedHat distribution (as of
5.x) did not use threads, even though they are well-supported on Linux.

And about stuff creeping into the distribution: gee... does that mean
that SocketServer doesn't work on the Mac? Threads *and* fork are not
available on Python/Mac, so all you would get is a single-threaded
server. icky. I can't see how adding threads to other platforms will
*hurt* the Macintosh platform... it can only help others.

About the only reason that I can see to *not* make them the default is
the slight speed loss. But that seems a bit bogus, as the interpreter
loop doesn't spend *that* much time mucking with the interp_lock to
allow thread switches. There have also been some real good suggestions
for making it take near-zero time until you actually create that second
thread.

Cheers,
-g

--
Greg Stein, http://www.lyra.org/

Re: ActiveState & fork & Perl [ In reply to ]

fredrik at pythonware

Jun 7, 1999, 4:21 PM

Post #20 of 93 (9238 views)

Mark wrote:
> Excuse my ignorance, but how hard would it be to simulate/emulate/ovulate
> fork using the Win32 extensions? Python has basically all of the native
> Win32 process API exposed, and writing a "fork" in Python that only forked
> Python scripts (for example) may be feasable and not too difficult.
>
> It would have obvious limitations, including the fact that it is not
> available standard with Python on Windows (just like a working popen now
> :-) but if we could follow the old 80-20 rule, and catch 80% of the uses
> with 20% of the effort it may be worth investigating.
>
> My knowledge of fork is limited to muttering "something about cloning the
> current process", so I may be naive in the extreme - but is this feasible?

as an aside, GvR added Windows' "spawn" API in 1.5.2,
so you can at least emulate some common variants of
fork+exec. this means that if someone writes a spawn
for Unix, we would at least catch >0% of the uses with
~0% of the effort ;-)

fwiw, I'm more interested in the "unicode all the way
down" parts of the activestate windows project. more
on that later.

</F>

Re: ActiveState & fork & Perl [ In reply to ]

fredrik at pythonware

Jun 7, 1999, 4:26 PM

Post #21 of 93 (9238 views)

> Not a Mac programmer, but I recall that when Steve Jobs came back,
> they published a schedule that said threads would be available a
> couple releases down the road. Schedules only move one way, so I'd
> guess ActiveState is premature.

http://www.computerworld.com/home/print.nsf/all/990531AAFA

> Perhaps Christian's stackless Python would enable green threads...
>
> (And there are a number of things in the standard distribution which
> don't work on Windows, either; fork and select()ing on file fds).

time to implement channels? (Tcl's unified abstraction
for all kinds of streams that you could theoretically use
something like select on. sockets, pipes, asynchronous
disk I/O, etc).

does select really work on ordinary files under Unix,
btw?

</F>

Re: ActiveState & fork & Perl [ In reply to ]

fredrik at pythonware

Jun 7, 1999, 4:30 PM

Post #22 of 93 (9237 views)

I wrote:

> > Not a Mac programmer, but I recall that when Steve Jobs came back,
> > they published a schedule that said threads would be available a
> > couple releases down the road. Schedules only move one way, so I'd
> > guess ActiveState is premature.
>
> http://www.computerworld.com/home/print.nsf/all/990531AAFA

which was just my way of saying that "did he perhaps
refer to OS X ?".

or are they adding real threads to good old MacOS too?

</F>

Re: ActiveState & fork & Perl [ In reply to ]

fredrik at pythonware

Jun 7, 1999, 4:38 PM

Post #23 of 93 (9239 views)

> Having threads by default will make a lot of things much simpler
> (in terms of knowing the default platform). Can't tell you how
> many times I curse to find that the default RedHat distribution
> (as of 5.x) did not use threads, even though they are well-
> supported on Linux.

I have a vague memory that once upon a time, the standard
X libraries shipped with RedHat weren't thread safe, and
Tkinter didn't work if you compiled Python with threads.

but I might be wrong and/or that may have changed...

</F>

RE: ActiveState & fork & Perl [ In reply to ]

MHammond at skippinet

Jun 7, 1999, 4:42 PM

Post #24 of 93 (9238 views)

> > http://www.computerworld.com/home/print.nsf/all/990531AAFA
>
> which was just my way of saying that "did he perhaps
> refer to OS X ?".
>
> or are they adding real threads to good old MacOS too?

Oh, /F, please dont start adding annotations to your collection of
incredibly obscure URLs - takes away half the fun ;-)

Mark.

Re: ActiveState & fork & Perl [ In reply to ]

Jun 7, 1999, 5:01 PM

Post #25 of 93 (9242 views)

Fredrik Lundh wrote:
>
> > Having threads by default will make a lot of things much simpler
> > (in terms of knowing the default platform). Can't tell you how
> > many times I curse to find that the default RedHat distribution
> > (as of 5.x) did not use threads, even though they are well-
> > supported on Linux.
>
> I have a vague memory that once upon a time, the standard
> X libraries shipped with RedHat weren't thread safe, and
> Tkinter didn't work if you compiled Python with threads.
>
> but I might be wrong and/or that may have changed...

Yes, it has changed. RedHat now ships with a thread-safe X so that they
can use GTK and Gnome (which use threads quite a bit).

There may be other limitations, however, as I haven't tried to do any
threaded GUI programming, especially on a recent RedHat (I'm using a
patched/hacked RH 4.1 system). RedHat 6.0 may even ship with a threaded
Python, but I dunno...

-g

--
Greg Stein, http://www.lyra.org/