Mailing List Archive

ob_refcnt access
How about introducing internal macros for explicit ob_refcnt accesses
in the core? Actually, there are a number of places where one can see
"op->ob_refcnt" logic, which could be replaced with _Py_GETREF(op),
_Py_SETREF(op, n) thus decoupling completely the low level refcount
management defined in object.h:

#define _Py_GETREF(op) (((PyObject *)op)->ob_refcnt)
#define _Py_SETREF(op, n) (((PyObject *)op)->ob_refcnt = (n))

Comments?
I've contributed myself to the mess in intobject.c & floatobject.c, so
I thought that such macros would make the code cleaner.


Here's the current state of affairs:

python/dist/src>find . -name "*.[c]" -exec grep ob_refcnt {} \; -print

(void *) v, ((PyObject *) v)->ob_refcnt))
./Modules/_tkinter.c
if (self->arg->ob_refcnt > 1) { \
if (ob->ob_refcnt < 2 || self->fast)
if (args->ob_refcnt > 1) {
./Modules/cPickle.c
if (--inst->ob_refcnt > 0) {
./Objects/classobject.c
if (result->ob_refcnt == 1)
./Objects/fileobject.c
if (PyFloat_Check(p) && p->ob_refcnt != 0)
if (!PyFloat_Check(p) || p->ob_refcnt == 0) {
if (PyFloat_Check(p) && p->ob_refcnt != 0) {
p, p->ob_refcnt, buf);
./Objects/floatobject.c
if (PyInt_Check(p) && p->ob_refcnt != 0)
if (!PyInt_Check(p) || p->ob_refcnt == 0) {
if (PyInt_Check(p) && p->ob_refcnt != 0)
p, p->ob_refcnt, p->ob_ival);
./Objects/intobject.c
assert(v->ob_refcnt == 1); /* Since v will be used as accumulator! */
./Objects/longobject.c
if (op->ob_refcnt <= 0)
op->ob_refcnt, (long)op);
op->ob_refcnt = 1;
if (op->ob_refcnt < 0)
fprintf(fp, "[%d] ", op->ob_refcnt);
./Objects/object.c
if (!PyString_Check(v) || v->ob_refcnt != 1) {
if (key->ob_refcnt == 2 && key == value) {
./Objects/stringobject.c
if (!PyTuple_Check(op) || op->ob_refcnt != 1) {
if (v == NULL || !PyTuple_Check(v) || v->ob_refcnt != 1) {
./Objects/tupleobject.c
if (PyList_Check(seq) && seq->ob_refcnt == 1) {
if (args->ob_refcnt > 1) {
./Python/bltinmodule.c
if (value->ob_refcnt != 1)
./Python/import.c
return PyInt_FromLong((long) arg->ob_refcnt);
./Python/sysmodule.c


--
Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
Re: ob_refcnt access [ In reply to ]
> How about introducing internal macros for explicit ob_refcnt accesses
> in the core?

What problem does this solve?

> Actually, there are a number of places where one can see
> "op->ob_refcnt" logic, which could be replaced with _Py_GETREF(op),
> _Py_SETREF(op, n) thus decoupling completely the low level refcount
> management defined in object.h:
>
> #define _Py_GETREF(op) (((PyObject *)op)->ob_refcnt)
> #define _Py_SETREF(op, n) (((PyObject *)op)->ob_refcnt = (n))

Why the cast? It loses some type-safety, e.g. _Py_GETREF(0) will now
cause a core dump instead of a compile-time error.

> Comments?

I don't see how it's cleaner or saves typing:

op->ob_refcnt
_Py_GETREF(op)

op->ob_refcnt = 1
_Py_SETREF(op, 1)

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: ob_refcnt access [ In reply to ]
marangoz wrote:
>
>
> How about introducing internal macros for explicit ob_refcnt accesses
> in the core? Actually, there are a number of places where one can see
> "op->ob_refcnt" logic, which could be replaced with _Py_GETREF(op),
> _Py_SETREF(op, n) thus decoupling completely the low level refcount
> management defined in object.h:
>
> #define _Py_GETREF(op) (((PyObject *)op)->ob_refcnt)
> #define _Py_SETREF(op, n) (((PyObject *)op)->ob_refcnt = (n))
>
> Comments?

Of course, the above should be (PyObject *)(op)->ob_refcnt. Also, I forgot
to mention that if this detail doesn't hurt code aesthetics, one (I) could
experiment more easily all sort of weird things with refcounting...

I formulated the same wish for malloc & friends some time ago, that is,
use everywhere in the core PyMem_MALLOC, PyMem_FREE etc, which would be
defined for now as malloc, free, but nobody seems to be very excited
about a smooth transition to other kinds of malloc. Hence, I reiterate
this wish, 'cause switching to macros means preparing the code for the
future, even if in the future it remains intact ;-).

Defining these basic interfaces is clearly Guido's job :-) as he points
out in his summary of the last Open Source summit, but nevertheless,
I'm raising the issue to let him see what other people think about this
and allow him to make decisions easier :-)

--
Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
Re: ob_refcnt access [ In reply to ]
On Thu, 24 Jun 1999, Guido van Rossum wrote:
> > How about introducing internal macros for explicit ob_refcnt accesses
> > in the core?
>
> What problem does this solve?

I assume Vladimir was trying to leave the door open for further
ob_refcnt manipulation hooks later, like having objects manage
their own refcounts. Until there's an actual problem to solve
that requires this, though, i'm not sure it's necessary. Are
there obvious reasons to want to allow this?

* * *

While we're talking about refcounts and all, i've had the
argument quite successfully made to me that a reasonably
written garbage collector can be both (a) simple and (b) more
efficient than refcounting. Having spent a good number of
work days doing nothing but debugging crashes by tracing
refcounting bugs, i was easily converted into a believer
once a friend dispelled the notion that garbage collectors
were either slow or horribly complicated. I had always been
scared of them before, but less so now.

Is an incremental GC being considered for a future Python?
I've idly been pondering various tricks by which it could be
made to work with existing extension modules -- here are some
possibilities:

1. Keep the refcounts and let existing code do the usual
thing; introduce a new variant of PyObject_NEW that
puts an object into the "gc-able" pool rather than the
"refcounted" pool.

2. Have Py_DECREF and Py_INCREF just do nothing, and let
the garbage collector guess from the contents of the
structure where the pointers are. (I'm told it's
possible to do this safely, since you can only have
false positives, never false negatives.)

3. Have Py_DECREF and Py_INCREF just do nothing, and ask
the extension module to just provide (in its type
object) a table of where the pointers are in its struct.

And so on; mix and match. What are everyone's thoughts on this one?



-- ?!ng

"All models are wrong; some models are useful."
-- George Box
RE: ob_refcnt access [ In reply to ]
[Ka-Ping Yee, opines about GC]

Ping, I think you're not getting any responses because this has been beaten
to death on c.l.py over the last month (for the 53rd time, no less <wink>).

A hefty percentage of CPython users *like* the reliably timely destruction
refcounting yields, and some clearly rely on it.

Guido recently (10 June) posted the start of a "add GC on top of RC" scheme,
in a thread with the unlikely name "fork()". The combination of cycles,
destructors and resurrection is quite difficult to handle in a way both
principled and useful (Java's way is principled but by most accounts
unhelpful to the point of uselessness).

Python experience with the Boehm collector can be found in the FAQ; note
that the Boehm collector deals with finalizers in cycles by letting cycles
with finalizers leak!

> ...
> While we're talking about refcounts and all, i've had the
> argument quite successfully made to me that a reasonably
> written garbage collector can be both (a) simple and (b) more
> efficient than refcounting.

That's a dubious claim. Sophisticated mark-and-sweep (with or without
compaction) is almost universally acknowledged to beat RC, but simple M&S
has terrible cache behavior (you fill up the address space before reclaiming
anything, then leap all over the address space repeatedly cleaning it up).
Don't discount that, in Python unlike as in most other languages, the simple
loop

for i in xrange(1000000):
pass

creates a huge amount of trash at a furious pace. Under RC it can happily
reuse the same little bit of storage each time around.

> Having spent a good number of work days doing nothing but debugging
> crashes by tracing refcounting bugs,

Yes, we can trade that for tracking down M&S bugs <0.5 wink> -- instead of
INCREF/DECREF macros, you end up with M&S macros marking regions where the
collector must not be run (because you're in a temporarily "inconsistent"
state). That's under sophisticated M&S, though, but is an absolute
nightmare when you miss a pair (the bugs only show up "sometimes", and not
always the same ways -- depends on when M&S happens to run, and "how
inconsistent" you happen to be at the time).

> ...
> And so on; mix and match. What are everyone's thoughts on this one?

I think Python probably needs to clean up cycles, but by some variant of
Guido's scheme on top of RC; I very much dislike the property of his scheme
that objects with destructors may be get destroyed without their destructors
getting invoked, but it seems hard to fix.

Alternatives include Java's scheme (which really has nothing going for it
other than that Java does it <0.3 wink>); Scheme's "guardian" scheme (which
would let the user "get at" cyclic trash with destructors, but refuses to do
anything with them on its own); following Boehm by saying that cycles with
destructors are immortal; following goofier historical precedent by e.g.
destroying such objects in reverse order of creation; or maybe just raising
an exception if a trash cycle containing a destructor is found.

All of those seem a comparative pain to implement, with Java's being the
most painful -- and quite possibly the least satisfying!

it's-a-whale-of-a-lot-easier-in-a-self-contained-universe-or-even-an-
all-c-one-ly y'rs - tim
Re: ob_refcnt access [ In reply to ]
Tim Peters wrote:
>
> [Ka-Ping Yee, opines about GC]
>
> Ping, I think you're not getting any responses because this has been beaten
> to death on c.l.py over the last month (for the 53rd time, no less <wink>).
>
> A hefty percentage of CPython users *like* the reliably timely destruction
> refcounting yields, and some clearly rely on it.

[CG issue dropped, I know the thread]

I know how much of a pain in the .. proper refcounting can be.
Sometimes, after long debugging, I wished it would go. But
finally, I think it is a *really good thing* to have to do
proper refcounting.
The reason is that this causes a lot of discipline, which
improves the whole program. I guess with GC always there,
quite a number of errors stay undetected.

I can say this, since I have been through a week of debugging
now, and I can now publish

full blown first class continuations for Python

yes I'm happy - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home
RE: ob_refcnt access [ In reply to ]
[Christian Tismer]
> ...
> I can say this, since I have been through a week of debugging
> now, and I can now publish
>
> full blown first class continuations for Python
>
> yes I'm happy - chris

You should be! So how come nobody else is <wink/frown>?

Let's fire some imagination here: without the stinkin' C stack snaking its
way thru everything, then with the exception of external system objects
(like open files), the full state of a running Python program is comprised
of objects Python understands and controls.

So with some amount of additional pain we could pickle them. And unpickle
them. Painlessly checkpoint a long computation for possible restarting?
Freeze a program while it's running on your mainframe, download it to your
laptop and resume it while you're on the road? Ship a bug report with the
computation frozen right before the error occurs? Take an app with gobs of
expensive initialization, freeze it after it's "finally ready to go", and
ship the latter instead? Capture the state of an interactive session for
later resumption? Etc.

Not saying those are easy, but getting the C stack out of the way means they
move from impossible to plausible. Maybe it would help get past the
Schemeophobia <wink> if, instead of calling them "continuations", you called
'em "platform-independent potentially picklable threads".

pippt-sounds-as-good-as-it-reads<wink>-ly y'rs - tim
RE: ob_refcnt access [ In reply to ]
> > yes I'm happy - chris
>
> You should be! So how come nobody else is <wink/frown>?

Im a little unhappy as this will break the Active Debugging stuff - ie, the
ability for Python, Java, Perl, VBScript etc to all exist in the same
process, each calling each other, and each being debuggable (makes a
_great_ demo :-)

Im not _really_ unhappy, Im just throwing this in as an FYI.

The Active Debugging interfaces need some way of sorting a call stack. As
many languages may be participating in a debugging session, there is no
implicit ordering available. Inter-language calls are not made via the
debugger, so it has no chance to intercept.

So the solution MS came up with was, surprise surprise, the machine stack!
:-) The assumption is that all languages will make _some_ use of the
stack, so they ask a language to report its "stack base address" and "stack
size". Using this information, the debugger sorts into the correct call
sequence.

Indeed, getting this information (even the half of it I did manage :-) was
painful, and hard to get right.

Ahh, the joys of bleeding-edge technologies :-)

> Let's fire some imagination here: without the stinkin' C
> stack snaking its

I tried, and look what happened :-) Seriously, some if this stuff would be
way cool.

Bit I also understand completely the silence on this issue. When the
thread started, there was much discussion about exactly what the hell these
continuation/coroutine thingies even were. However, there were precious
few real-world examples where they could be used. A few acedemic,
theoretical places, but the only real contender I have seen brought up was
Medusa. There were certainly no clear examples of "as soon as we have
this, I could change abc to take advantage, and this would give us the very
cool xyz"

So, if anyone else if feeling at all like me about this issue, they are
feeling all warm and fuzzy knowing that a few smart people are giving us
the facility to do something we hope we never, ever have to do. :-)

Mark.
RE: ob_refcnt access [ In reply to ]
Mark Hammond writes:
> I tried, and look what happened :-) Seriously, some if this stuff
> would be way cool.
>
> Bit I also understand completely the silence on this issue. When
> the thread started, there was much discussion about exactly what
> the hell these continuation/coroutine thingies even were. However,
> there were precious few real-world examples where they could be
> used. A few acedemic, theoretical places, but the only real
> contender I have seen brought up was Medusa. There were certainly
> no clear examples of "as soon as we have this, I could change abc
> to take advantage, and this would give us the very cool xyz"

Part of the problem is that we didn't have the feature to play with.
Many of the possibilities are showing up now that it's here...

The basic advantage to coroutines is they allow you to turn any
event-driven/state-machine problem into one that is managed with
'normal' control state; i.e., for loops, while loops, nested procedure
calls, etc...

Here are a few possible real-world uses:

==================================================

Parsing. I remember a discussion from a few years back about the
distinction between 'push' and 'pull' model parsers. Coroutines let
you have it both ways; you can write a parser in the most natural way
(pull), but use it as a 'push'; i.e. for a web browser.

==================================================

"http sessions". A single 'thread' of control that is re-entered
whenever a hit from a particular user ('session') comes in to the web
server:

[Apologies to those that have already seen this cheezy example]

def ecommerce (session):
session.login() # sends a login form, waits for it to return
basket = []
while 1:
item = session.shop_for_item()
if item:
basket.append (item)
else:
break
if basket:
session.get_shipping_info()
session.get_payment_info()
session.transact()

'session.shop_for_item()' will resume the main coroutine, which will
resume this coroutine only when a new hit comes in from that
session/user, and 'return' this hit to the while loop.

I have a little web server that uses this idea to play blackjack:

http://www.nightmare.com:7777/
http://www.nightmare.com/stuff/blackjack_httpd.py

[though I'm a little fuzzy on the rules].

Rather than building a state machine that keeps track of where the
user has been, and what they're doing, you can keep all the state in
local variables (like 'basket' above) - in other words, it's a much
more natural style of programming.

==================================================

One of the areas I'm most excited about is GUI coding. All GUI's are
event driven. All GUI code is therefore written in a really twisted,
state-machine fashion; interactions are very complex. OO helps a bit,
but doesn't change the basic difficulty - past a certain point
interesting things become too complex to try...

Mr. Fuchs' paper ("Escaping the event loop: an alternative control
structure for multi-threaded GUIs") does a much better job of
describing this than I can:

http://cs.nyu.edu/phd_students/fuchs/
http://cs.nyu.edu/phd_students/fuchs/gui.ps

==================================================

Tim's example of 'dumping' a computation in the middle and storing it
on disk (or sending it over a network), is not a fantasy... I have a
'stackless' Scheme system that does this right now.

==================================================

Ok, final example. Isn't there an interface in Python to call a
certain function after every so many vm insns? Using coroutines you
could hook into this and provide non-preemptive 'threads' for those
platforms that don't have them. [.And the whole thing would be written
in Python, not in C!]

==================================================

> So, if anyone else if feeling at all like me about this issue, they
> are feeling all warm and fuzzy knowing that a few smart people are
> giving us the facility to do something we hope we never, ever have
> to do. :-)

"When the only tool you have is a hammer, everything looks like a
nail". I saw the guys over in the Scheme shop cutting wood with a
power saw; now I feel like a schmuck with my hand saw.

You are right to be frightened by the strangeness of the underlying
machinery; hopefully a simple and easy-to-understand interface can be
built for the C level as well as Python. I think Christian's 'frame
dispatcher' is fairly clear, and not *that* much of a departure from
the current VM; it's amazing to me how little work really had to be
done!

-Sam
Re: ob_refcnt access [ In reply to ]
Mark Hammond wrote:
>
> > > yes I'm happy - chris
> >
> > You should be! So how come nobody else is <wink/frown>?

(to Tim) I believe this comes simply since following me
would force people to change their way of thinking.
I am through this already, but it was hard for me.
And after accepting to be stackless, there is no way
to go back. Today I'm wondering about my past:
"how could I think of stacks when thinking of programs?"
This is so wrong.

The truth is: Programs are just some data, part of it called code,
part of it is local state, and! its future of computation.
Out, over, roger. All the rest is artificial showstoppers.

> Im a little unhappy as this will break the Active Debugging stuff - ie, the
> ability for Python, Java, Perl, VBScript etc to all exist in the same
> process, each calling each other, and each being debuggable (makes a
> _great_ demo :-)
>
> Im not _really_ unhappy, Im just throwing this in as an FYI.

Well, yet I see no problem.

> The Active Debugging interfaces need some way of sorting a call stack. As
> many languages may be participating in a debugging session, there is no
> implicit ordering available. Inter-language calls are not made via the
> debugger, so it has no chance to intercept.
>
> So the solution MS came up with was, surprise surprise, the machine stack!
> :-) The assumption is that all languages will make _some_ use of the
> stack, so they ask a language to report its "stack base address" and "stack
> size". Using this information, the debugger sorts into the correct call
> sequence.

Now, I can give it a machine stack. There is just a frame dispatcher
sitting on the stack, and it grabs frames from the current thread
state.

> Indeed, getting this information (even the half of it I did manage :-) was
> painful, and hard to get right.

I would have to see the AX interface. But for sure there will
be some method hooks with which I can tell AX how to walk
the frame chain.
And why don't I simply publish frames as COM objects?
This would give you much more than everything else,
I guess.

BTW, as it is now, there is no need to use AX debugging
for Python, since Python can do it alone now. Of course
it makes sense to have it all in the AX environment.
You will be able to modify a running programs local
variables, its evaluation stack, change its code,
change where it returns to, all is doable.

...
> Bit I also understand completely the silence on this issue. When the
> thread started, there was much discussion about exactly what the hell these
> continuation/coroutine thingies even were. However, there were precious
> few real-world examples where they could be used. A few acedemic,
> theoretical places, but the only real contender I have seen brought up was
> Medusa. There were certainly no clear examples of "as soon as we have
> this, I could change abc to take advantage, and this would give us the very
> cool xyz"

The problem was for me, that I had also no understanding what
I was doing, actually. Implemented continuations without an
idea how they work. But Tim and Sam said they were the most
powerful control strucure possible, so I used all my time
to find this out. Now I'm beginning to understand.
And my continuation based coroutine example turns out to
be twenty lines of Python code. Coming soon, after I served
my whining customers.

> So, if anyone else if feeling at all like me about this issue, they are
> feeling all warm and fuzzy knowing that a few smart people are giving us
> the facility to do something we hope we never, ever have to do. :-)

Think of it as just a flare gun in your hands. By reading the fine
print, you will realize that you actually hold an atom bomb,
with a little code taming it for you. :-)

back-to-the-future - ly y'rs - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home
Re: ob_refcnt access [ In reply to ]
> Part of the problem is that we didn't have the feature to play with.
> Many of the possibilities are showing up now that it's here...
>
> The basic advantage to coroutines is they allow you to turn any
> event-driven/state-machine problem into one that is managed with
> 'normal' control state; i.e., for loops, while loops, nested procedure
> calls, etc...
>
> Here are a few possible real-world uses:

Thanks, Sam! Very useful collection of suggestions. (How come I'm
not surprised to see these coming from you ;-)

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: ob_refcnt access [ In reply to ]
> > yes I'm happy - chris
>
> You should be! So how come nobody else is <wink/frown>?

Chris and I have been through this in private, but it seems that as
long as I don't fess up in public I'm afraid it will come back and
I'll get pressure coming at me to endorse Chris' code.

I have no problem with the general concept (see my response to Sam's
post of exciting examples).

But I have a problem with a megapatch like this that affects many
places including very sensitive areas like the main loop in ceval.c.
The problem is simply that I know this is very intricate code, and I
can't accept a patch of this scale to this code before I understand
every little detail of the patch. I'm just too worried otherwise that
there's a reference count bug in it that will very subtly break stuff
and that will take forever to track down; I feel that when I finally
have the time to actually understand the whole patch I'll be able to
prevent that (famous last words).

Please don't expect action or endorsement of Chris' patch from me any
time soon, I'm too busy. However I'd love it if others used the patch
in a real system and related their experiences regarding performance,
stability etc.

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: ob_refcnt access [ In reply to ]
Guido van Rossum wrote:
>
> > > yes I'm happy - chris
> >
> > You should be! So how come nobody else is <wink/frown>?
>
> Chris and I have been through this in private, but it seems that as
> long as I don't fess up in public I'm afraid it will come back and
> I'll get pressure coming at me to endorse Chris' code.

Please let me add a few comments.

> I have no problem with the general concept (see my response to Sam's
> post of exciting examples).

This is the most worthful statement I can get.
And see below.

> But I have a problem with a megapatch like this that affects many
> places including very sensitive areas like the main loop in ceval.c.

Actually it is a rather small patch, but the implicit semantic
change is rather hefty.

> The problem is simply that I know this is very intricate code, and I
> can't accept a patch of this scale to this code before I understand
> every little detail of the patch. I'm just too worried otherwise that
> there's a reference count bug in it that will very subtly break stuff
> and that will take forever to track down; I feel that when I finally
> have the time to actually understand the whole patch I'll be able to
> prevent that (famous last words).

I never expected to see this patch go into Python right now.
The current public version is an alpha 0.2. Meanwhile I have
0.3, with again new patches, and a completely reworked policy
of frame refcounting.
Even worse, there is a night mare of more work which I simply
had no time for. All the instance and onbect code must be
carefully changed, since they still need to call back in
a recursive way. This is hard to change until I have a better
mechanism to generate all the callbacks. For instance, I cannot
switch tasks in an __init__ at this time. Although I can
do so in regular methods. But this is all half-baked.

In other words, the danger is by far not over, but still
in the growing phase. I believe I should work on and maintain
this until I'm convinced that there are not more refcount
bugs than before, and until I have evicted every recursion
which is a serious impact. This is still months of work.

When I release the final version, I will pay $100 to the first
person who finds a refcount bug which I introduced. But not
before. I don't want to waste Guido's time, and for sure not
now with this bloody fresh code.

What I needed to know is wether I am on the right track or
if I'm wasting my time. But since I have users already,
it is no waste at all. What I really could use were some
hints about API design.

Guido, thank you for Python - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home
RE: ob_refcnt access [ In reply to ]
[Mark Hammond]
> Im a little unhappy as this [stackless Python] will break the Active
> Debugging stuff ...
> ...
> So the solution MS came up with was, surprise surprise, the machine stack!
> :-) The assumption is that all languages will make _some_ use of the
> stack, so they ask a language to report its "stack base address"
> and "stack size". Using this information, the debugger sorts into the
> correct call sequence.

Mark, you can't *really* believe Chris is incapable of hacking around this,
right?

It's not even clear there's something to be hacked around, since Python is
only Python and there's nothing Christian can do to stop other languages
that call into Python from using the machine stack, or to call other
languages from Python without using the machine stack. So Python "shows up
on the stack" no matter what, cross-language.

> ...
> Bit I also understand completely the silence on this issue. When the
> thread started, there was much discussion about exactly what the
> hell these continuation/coroutine thingies even were.

The Fuchs paper Sam referenced explained it in simple C terms: a
continuation is exactly what C setjmp/longjmp would do if setjmp saved (&
longjmp restored) the C stack *in addition* to the program counter and
machine registers (which they already save/restore). That's all there is to
it, at heart: objects capture data state, continuations capture control
flow state. Whenever the OS services an interrupt and drops into kernel
mode, it captures a continuation for user mode -- they don't *call* it that,
but that's what they're doing, and it's as practical as a pencil (well,
*more* practical, actually <wink>).

> However, there were precious few real-world examples where they could
> be used.

Nobody asked for any before now <0.5 wink> -- and I see Sam provided some
marvelous ones in response to this.

> A few acedemic, theoretical places,

I think you undervalue those: people working on the underpinnings of
languages strive very hard to come up with the simplest possible examples
that don't throw away the core of the problem to be solved. That doesn't
mean the theoreticians are too air-headed to understand "real world
problems"; it's much more that, e.g., "if you can't compare the fringes of
two trees cleanly, you can't possibly do anything harder than that cleanly
either -- but if you can do this little bit cleanly, we have strong reason
to believe there's a large class of difficult real problems you can also do
cleanly".

If you need a "practical" example of that, picture e.g. a structure-based
diff engine for HTML source. Which are really trees defined by tags, and
where text-based comparison can be useless (you don't care if "<LI>" moved
from column 12 of line 16 to column 1 of line 17, but you care a lot if the
*number* of <LI> tags changed -- so have you have to compare two trees *as*
trees).

But that's a problem easy enough for generators to solve cleanly. Knuth
writes a large (for his books) elevator-simulation program to illustrate
coroutines (which are more powerful than generators), and complains that he
can't come up with a simpler example that illustrates any point worth
making. And he's right! The "literature standard" text-manipulation
example at the end of my coroutine module illustrates what Sam was talking
about wrt writing straightforward "pull" algorithms for a "push" process,
but even that one can be solved with simpler pipeline control flow.

At least for *that*, nobody who ever used Unix would doubt the real-world
utility of the pipeline model for a nanosecond <1e-9 wink>.

If you want a coroutine example, go to a restaurant and order a meal. When
you leave, glance back *real quick*. If everyone in the restaurant is dead,
they were a meal-generating subroutine; but if they're still serving other
customers, your meal-eating coroutine and their meal-generating coroutine
worked to mutual benefit <wink>.

> but the only real contender I have seen brought up was Medusa. There were
> certainly no clear examples of "as soon as we have this, I could change
> abc to take advantage, and this would give us the very cool xyz"
>
> So, if anyone else if feeling at all like me about this issue, they are
> feeling all warm and fuzzy knowing that a few smart people are giving us
> the facility to do something we hope we never, ever have to do. :-)

Actually, you'll want to do it a lot <wink>. Christian & I have bantered
about this a few times a year in pvt, usually motivated by some horrendous
kludge posted to c.l.py to solve a problem that any Assistant Professor of
Medieval English could solve without effort in Icon. The *uses* aren't
esoteric at all.

or-at-least-not-more-than-you-make-'em-ly y'rs - tim
RE: ob_refcnt access [ In reply to ]
[.Tim tells me it will all be obvious if I just think a little harder
<wink>]

Your points about "acedemic examples" is well taken. The reality is that,
even given these simple examples (which I dared deride as acedemic), the
simple fact is Im not seeing "the point".

I seriously dont doubt all all you say. However, as Sam and Chris have
said many times, it is just a matter of changing the way to you think.

Interestingly:
Chris said it recently, and continues to say it.
Sam said it to me _years_ ago, and said it repeatedly, but hasnt said it
recently.
Tim hasnt really said it yet :-)

This is almost certainly because when your brain does switch, it is a
revelation, and really not too hard at all. But after a while, you forget
the switch ever took place.

Closest analogy I can think of is OO programming. In my experience trying
to _learn_ OO programming from a few misc examples and texts was pointless
and very hard. You need a language to play with it in. And when you have
one, your brain makes the switch, you see the light, and you can't see what
was ever mysterious about it. And you tell everyone its easy; "just change
the way you think about data" :-)
But to all us here, OO programming is just so obvious it goes without
saying. Occasionaly a newbie will have trouble with OO concepts in Python,
and I personally have trouble seeing what could _possibly_ be difficult
about understanding these very simple concepts. So Im just as guilty, just
not in this particular case :-)

So, short of all us here going and discovering the light using a different
language (perish the thought :), my original point stands that until Chris'
efforts give us something we can easily play with, some of use _still_ wont
see what all the fuss is about. (Although I admit it has nothing to do with
either the examples or the applicability of the technology to all sorts of
things)

Which leaves you poor guys in a catch 22 - without noise of some sort from
the rest of us, its hard to keep the momentum going, but without basically
a fully working Python with continuations, we wont be making much noise.

But-I-will-thank-you-all-personally-and-profusely-when-I-do-see-the-light,
ly

Mark.