Mailing List Archive

Thoughts o' the day:

+ Generators ("semi-coroutines") are wonderful tools and easy to implement
without major changes to the PVM. Icon calls 'em generators, Sather calls
'em iterators, and they're exactly what you need to implement "for thing in
object:" when object represents a collection that's tricky to materialize.
Python needs something like that. OTOH, generators are pretty much limited
to that.

+ Coroutines are more general but much harder to implement, because each
coroutine needs its own stack (a generator only has one stack *frame*-- its
own --to worry about), and C-calling-Python can get into the act. As Sam
said, they're probably no easier to implement than call/cc (but trivial to
implement given call/cc).

+ What may be most *natural* is to forget all that and think about a
variation of Python threads implemented directly via the interpreter,
without using OS threads. The PVM already knows how to handle thread-state
swapping. Given Christian's stackless interpreter, and barring C->Python
cases, I suspect Python can fake threads all by itself, in the sense of
interleaving their executions within a single "real" (OS) thread. Given the
global interpreter lock, Python effectively does only-one-at-a-time anyway.

Threads are harder than generators or coroutines to learn, but

A) Many more people know how to use them already.

B) Generators and coroutines can be implemented using (real or fake)
threads.

C) Python has offered threads since the beginning.

D) Threads offer a powerful mode of control transfer coroutines don't,
namely "*anyone* else who can make progress now, feel encouraged to do so at
my expense".

E) For whatever reasons, in my experience people find threads much easier to
learn than call/cc -- perhaps because threads are *obviously* useful upon
first sight, while it takes a real Zen Experience before call/cc begins to
make sense.

F) Simulated threads could presumably produce much more informative error
msgs (about deadlocks and such) than OS threads, so even people using real
threads could find excellent debugging use for them.

Sam doesn't want to use "real threads" because they're pigs; fake threads
don't have to be. Perhaps

x = y.SOME_ASYNC_CALL(r, s, t)

could map to e.g.

import config
if config.USE_REAL_THREADS:
import threading
else:
from simulated_threading import threading

from config.shared import msg_queue

class Y:
def __init__(self, ...):
self.ready = threading.Event()
...

def SOME_ASYNC_CALL(self, r, s, t):
result = [None] # mutable container to hold the result
msg_queue.put((server_of_the_day, r, s, t, self.ready, result))
self.ready.wait()
self.ready.clear()
return result[0]

where some other simulated thread polls the msg_queue and does ready.set()
when it's done processing the msg enqueued by SOME_ASYNC_CALL. For this to
scale nicely, it's probably necessary for the PVM to cooperate with the
simulated_threading implementation (e.g., a simulated thread that blocks
(like on self.ready.wait()) should be taken out of the collection of
simulated threads the PVM may attempt to resume -- else in Sam's case the
PVM would repeatedly attempt to wake up thousands of blocked threads, and
things would slow to a crawl).

Of course, simulated_threading could be built on top of call/cc or
coroutines too. The point to making threads the core concept is keeping
Guido's brain from exploding. Plus, as above, you can switch to "real
threads" by changing an import statement.

making-sure-the-global-lock-support-hair-stays-around-even-if-greg-
renders-it-moot-for-real-threads<wink>-ly y'rs - tim

Tim Peters wrote:
>
> [Tim]
> > OK. So how do you feel about coroutines? Would sure be nice
> > to have *some* way to get pseudo-parallel semantics regardless of OS.
>
> [David Ascher]
> > I read about coroutines years ago on c.l.py, but I admit I forgot it all.
> > Can you explain them briefly in pseudo-python?
>
> How about real Python? http://www.python.org/tim_one/000169.html contains a
> complete coroutine implementation using threads under the covers (& exactly
> 5 years old tomorrow <wink>). If I were to do it over again, I'd use a
> different object interface (making coroutines objects in their own right
> instead of funneling everything through a "coroutine controller" object),
> but the ideas are the same in every coroutine language. The post contains
> several executable examples, from simple to "literature standard".

What an interesting thread! Unfortunately, all the examples are messed
up since some HTML formatter didn't take care of the python code,
rendering it unreadable. Is there a different version available?

Also, I'd like to read the rest of the threads in
http://www.python.org/tim_one/ but it seems that only your messages
are archived?
Anyway, the citations in http://www.python.org/tim_one/000146.html
show me that you have been through all of this five years
ago, with a five years younger Guido which sounds a bit
different than today.
I had understood him better if I had known that this
is a re-iteration of a somehow dropped or entombed idea.

(If someone has the original archives from that epoche,
I'd be happy to get a copy. Actually, I'm missing all upto
end of 1996.)

A sort snapshot:
Stackless Python is meanwhile nearly alive, with recursion
avoided in ceval. Of course, some modules are left which
still need work, but enough for a prototype. Frames contain
now all necessry state and are now prepared for execution
and thrown back to the evaluator (elevator?).

The key idea was to change the deeply nested functions in a
way, that their last eval_code call happens to be tail recursive.
In ceval.c (and in other not yet changed places), functions
to a lot of preparation, build some parameter, call eval_code
and release the parameter. This was the crux, which I solved
by a new filed in the frame object, where such references
can be stored. The routine can now return with the ready packaged
frame, instead of calling it.

As a minimum facility for future co-anythings,
I provided a hook function for resuming frames, which causes no
overhead in the usual case but allows to override what a frame
does when someone returns control to it. To implement
this is due to some extension module, wether this may
be coroutines or your nice nano-threads, it's possible.

threadedly yours - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home

>> http://www.python.org/tim_one/000169.html

[Christian]
> What an interesting thread! Unfortunately, all the examples are messed
> up since some HTML formatter didn't take care of the python code,
> rendering it unreadable. Is there a different version available?
>
> Also, I'd like to read the rest of the threads in
> http://www.python.org/tim_one/ but it seems that only your messages
> are archived?

Yes, that link is the old Pythonic Award Shrine erected in my memory -- it's
all me, all the time, no mercy, no escape <wink>.

It predates the DejaNews archive, but the context can still be found in

http://www.python.org/search/hypermail/python-1994q2/index.html

There's a lot in that quarter about continuations & coroutines, most from
Steven Majewski, who took a serious shot at implementing all this.

Don't have the code in a more usable form; when my then-employer died, most
of my files went with it.

You can save the file as text, though! The structure of the code is intact,
it's simply that your browswer squashes out the spaces when displaying it.
Nuke the <P> at the start of each code line and what remains is very close
to what was originally posted.

> Anyway, the citations in http://www.python.org/tim_one/000146.html
> show me that you have been through all of this five years
> ago, with a five years younger Guido which sounds a bit
> different than today.
> I had understood him better if I had known that this
> is a re-iteration of a somehow dropped or entombed idea.

You *used* to know that <wink>! Thought you even got StevenM's old code
from him a year or so ago. He went most of the way, up until hitting the
C<->Python stack intertwingling barrier, and then dropped it. Plus Guido
wrote generator.py to shut me up, which works, but is about 3x clumsier to
use and runs about 50x slower than a generator should <wink>.

> ...
> Stackless Python is meanwhile nearly alive, with recursion
> avoided in ceval. Of course, some modules are left which
> still need work, but enough for a prototype. Frames contain
> now all necessry state and are now prepared for execution
> and thrown back to the evaluator (elevator?).
> ...

Excellent! Running off to a movie & dinner now, but will give a more
careful reading tonight.

co-dependent-ly y'rs - tim

Christian Tismer <tismer@appliedbiometrics.com> wrote:
> (If someone has the original archives from that epoche,
> I'd be happy to get a copy. Actually, I'm missing all upto
> end of 1996.)

http://www.egroups.com/group/python-list/info.html
has it all (almost), starting in 1991.

</F>