Mailing List Archive

Tim Peters wrote:
>
> [M.-A. Lemburg]
> > Wouldn't a generic builtin for these kinds of things be
> > better, e.g. a function returning a default value in case
> > an exception occurs... something like:
> >
> > tryexcept(list.pop(), IndexError, default)
> >
> > which returns default in case an IndexError occurs. Don't
> > think this would be much faster that the explicit try:...except:
> > though...
>
> As a function (builtin or not), tryexcept will never get called if
> list.pop() raises an exception.

Dang. You're right...

> tryexcept would need to be a new statement
> type, and the compiler would have to generate code akin to
>
> try:
> whatever = list.pop()
> except IndexError:
> whatever = default
>
> If you want to do it in a C function instead to avoid the Python-level
> exception overhead, the compiler would have to wrap list.pop() in a lambda
> in order to delay evaluation until the C code got control; and then you've
> got worse overhead <wink>.

Oh well, forget the whole idea then. list.pop() is really not
needed that often anyways to warrant the default arg thing, IMHO.
dict.get() and getattr() have the default arg as performance
enhancement and I believe that you wouldn't get all that much
better performance on average by adding a second optional
argument to list.pop().

BTW, there is a generic get() function in mxTools (you know where...)
in case someone should be looking for such a beast. It works
with all sequences and mappings.

Also, has anybody considered writing list.pop(..,default) this way:

if list:
obj = list.pop()
else:
obj = default

No exceptions, no changes, fast as hell :-)

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 162 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

"M.-A. Lemburg" wrote:
...
> Also, has anybody considered writing list.pop(..,default) this way:
>
> if list:
> obj = list.pop()
> else:
> obj = default
>
> No exceptions, no changes, fast as hell :-)

Yes, that's the best way to go, I think.
But wasn't the primary question directed on
an atomic function which is thread-safe?
I'm not sure, this thread has grown too fast :-)

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home

Christian Tismer wrote:
>
> "M.-A. Lemburg" wrote:
> ...
> > Also, has anybody considered writing list.pop(..,default) this way:
> >
> > if list:
> > obj = list.pop()
> > else:
> > obj = default
> >
> > No exceptions, no changes, fast as hell :-)
>
> Yes, that's the best way to go, I think.
> But wasn't the primary question directed on
> an atomic function which is thread-safe?
> I'm not sure, this thread has grown too fast :-)

I think that was what Jim had in mind in the first place.
Hmm, so maybe we're not after lists after all: maybe what
we need is access to the global interpreter lock in Python,
so that we can write:

sys.lock.acquire()
if list:
obj = list.pop()
else:
obj = default
sys.lock.release()

Or maybe we need some general lock in the thread module for these
purposes... don't know. It's been some time since I used
threads.

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 162 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

Christian Tismer wrote:
>
> "M.-A. Lemburg" wrote:
> ...
> > Also, has anybody considered writing list.pop(..,default) this way:
> >
> > if list:
> > obj = list.pop()
> > else:
> > obj = default
> >
> > No exceptions, no changes, fast as hell :-)
>
> Yes, that's the best way to go, I think.
> But wasn't the primary question directed on
> an atomic function which is thread-safe?

Right. And the above code doesn't solve this
problem. Tim's code *does* solve the problem.
It's the code we were using. It is a bit
verbose though.


> I'm not sure, this thread has grown too fast :-)

Don't they all?

Jim

--
Jim Fulton mailto:jim@digicool.com Python Powered!
Technical Director (888) 344-4332 http://www.python.org
Digital Creations http://www.digicool.com http://www.zope.org

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission. Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.

Tim Peters wrote:
> As a function (builtin or not), tryexcept will never get called if
> list.pop() raises an exception.

M.-A. Lemburg writes:
> Oh well, forget the whole idea then. list.pop() is really not

Giving up already? Wouldn't you just love this as an expression
operator (which could work)?
How about:

top = list.pop() excepting IndexError, default

Hehehe... ;-)


-Fred

--
Fred L. Drake, Jr. <fdrake@acm.org>
Corporation for National Research Initiatives

Fred> Giving up already? Wouldn't you just love this as an expression
Fred> operator (which could work)?
Fred> How about:

Fred> top = list.pop() excepting IndexError, default

Why not go all the way to Perl with

top = list.pop() unless IndexError

???

;-)

Skip

Skip Montanaro writes:
> Why not go all the way to Perl with
>
> top = list.pop() unless IndexError


Trying to kill me, Skip? ;-)
Actually, the semantics are different. If we interpret that using
the Perl semantics for "unless", don't we have the same thing as:

if not IndexError:
top = list.pop()

Since IndexError will normally be a non-empty string or a class,
this is pretty much:

if 0:
top = list.pop()

which certainly isn't quite as interesting. ;-)


-Fred

--
Fred L. Drake, Jr. <fdrake@acm.org>
Corporation for National Research Initiatives

Fred> Skip Montanaro writes:
>> Why not go all the way to Perl with
>>
>> top = list.pop() unless IndexError


Fred> Trying to kill me, Skip? ;-)

Nope, just a flesh wound. I'll wait for the resulting infection to really
do you in. ;-)

Fred> Actually, the semantics are different. If we interpret that using
Fred> the Perl semantics for "unless", don't we have the same thing as:

Yes, but the flavor is the same. Reading Perl code that uses the unless
keyword always seemed counterintuitive to me. Something like

x = y unless foo;

always reads to me like, "Assign y to x. No, wait a minute. I forgot
something. Only do that if foo isn't true." What was so bad about

if (!foo) { x = y; }

That was my initial reaction to the use of the trailing except.

We argue a lot in the Python community about whether or not a proposed
language feature increases the expressive power of the language or not
(which is a good idea in my opinion). The Perl community has apparently
never been afflicted with that disease.

smiles all 'round...

Skip

> ...
> Hmm, so maybe we're not after lists after all: maybe what
> we need is access to the global interpreter lock in Python,
> so that we can write:
>
> sys.lock.acquire()
> if list:
> obj = list.pop()
> else:
> obj = default
> sys.lock.release()

The thread attempting the sys.lock.acquire() necessarily already owns the
global lock, so the attempt to acquire it is a guaranteed deadlock --
arguably not helpful <wink>.

> Or maybe we need some general lock in the thread module for these
> purposes... don't know. It's been some time since I used
> threads.

Jim could easily allocate a list lock for this purpose if that's what he
wanted; and wrap it in a class with a nice interface too. He'd eventually
end up with the std Queue.py module, though.

But if he doesn't want the overhead of an exception when the queue is empty,
he sure doesn't want the comparatively huge overhead of a (any flavor of)
lock either (which may drag the OS into the picture).

There's nothing wrong with wanting a fast thread-safe queue! I just don't
like the idea of adding an otherwise-ugly new gimmick to core lists for it;
also have to wonder about Jim's larger picture if he's writing stuff in
Python that's *so* time-critical that the overhead of an ordinary exception
from time to time is a genuine problem. The verbosity of the alternative
can be hidden in a lock-free class or function, if it's the clumsiness
instead of the time that's grating.

Tim Peters wrote:
>
> > ...
> > Hmm, so maybe we're not after lists after all: maybe what
> > we need is access to the global interpreter lock in Python,
> > so that we can write:
> >
> > sys.lock.acquire()
> > if list:
> > obj = list.pop()
> > else:
> > obj = default
> > sys.lock.release()
>
> The thread attempting the sys.lock.acquire() necessarily already owns the
> global lock, so the attempt to acquire it is a guaranteed deadlock --
> arguably not helpful <wink>.

True, sys.lock.acquire() would have to set a flag *not* to release
the lock until the next call to sys.lock.release(), which then
clears this flag again. Sort of a lock for the unlocking the lock ;-)

Could this work, or am I having a mind twister somewhere in
there again ?

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 160 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg writes:

> True, sys.lock.acquire() would have to set a flag *not* to release
> the lock until the next call to sys.lock.release(), which then
> clears this flag again. Sort of a lock for the unlocking the lock
> ;-)
>
> Could this work, or am I having a mind twister somewhere in
> there again ?

Sounds like a critical section to me. On Windows, those are
lightweight and very handy. You can build one with Python thread
primitives, but unfortunately, they come out on the heavy side.

Locks come in 4 types, categorized by whether they can be released
only by the owning thread, and whether they can be acquired
recursively. The interpreter lock is in the opposite quadrant from a
critical section, so "sys.lock.freeze()" and "sys.lock.thaw()" have
little chance of having an efficient implementation on any platform.

A shame. That would be pretty cool.


- Gordon

[M.-A. Lemburg]
> ...
> Oh well, forget the whole idea then. list.pop() is really not
> needed that often anyways to warrant the default arg thing, IMHO.
> dict.get() and getattr() have the default arg as performance
> enhancement

I like their succinctness too; count = dict.get(key, 0) is helpfully
"slimmer" than either of

try:
count = dict[key]
except KeyError:
count = 0

or

count = 0
if dict.has_key(key):
count = dict[key]

> and I believe that you wouldn't get all that much better performance
> on average by adding a second optional argument to list.pop().

I think you wouldn't at *all*, except in Jim's novel case. That is, when a
list is empty, it's usually the signal to get out of a loop, and you can
either test

if list:
item = list.pop()
else:
break

today or

item = list.pop(-1, marker)
if item is marker:
break

tomorrow. The second way doesn't buy anything to my eye, and the first way
is very often the pretty

while list:
item = list.pop()

if-it-weren't-for-jim's-use-i'd-see-no-use-at-all-ly y'rs - tim

Gordon McMillan wrote:
>
> M.-A. Lemburg writes:
>
> > True, sys.lock.acquire() would have to set a flag *not* to release
> > the lock until the next call to sys.lock.release(), which then
> > clears this flag again. Sort of a lock for the unlocking the lock
> > ;-)
> >
> > Could this work, or am I having a mind twister somewhere in
> > there again ?
>
> Sounds like a critical section to me. On Windows, those are
> lightweight and very handy. You can build one with Python thread
> primitives, but unfortunately, they come out on the heavy side.
>
> Locks come in 4 types, categorized by whether they can be released
> only by the owning thread, and whether they can be acquired
> recursively. The interpreter lock is in the opposite quadrant from a
> critical section, so "sys.lock.freeze()" and "sys.lock.thaw()" have
> little chance of having an efficient implementation on any platform.

Actually, I think all that's needed is another global like
the interpreter_lock in ceval.c. Since this lock is only
accessed via abstract functions, I presume the unlock flag could
easily be added.

The locking section would only focus on Python, though: other
threads could still be running provided they don't execute
Python code, e.g. write data to a spooler. So it's not really
the equivalent of a critical section as the one you can define in C.

PS: I changed the subject line... hope this doesn't kill the
thread ;)

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 158 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

> Actually, I think all that's needed is another global like
> the interpreter_lock in ceval.c. Since this lock is only
> accessed via abstract functions, I presume the unlock flag could
> easily be added.

Well, my personal opinion is that this is really quite wrong. The most
obvious thing to me is that we are exposing an implementation detail we all
would dearly like to see removed one day - the global interpreter lock.

But even if we ignore that, it seems to me that you are describing an
application abstraction, not a language abstraction. This thread started
with Jim wanting a thread-safe, atomic list operation. This is not an
unusual requirement (ie, a thread-safe, atomic operation), so languages
give you access to primitives that let you build this.

To my mind, you are asking for the equivilent of a C function that says
"suspend all threads except me, cos Im doing something _really_ important".
C does not provide that, and I have never thought it should. As Gordon
said, Win32 has critical sections, but these are really just lightweight
locks. I really dont see how Python is different - it gives you all the
tools you need to build these abstractions.

I really dont see what you are after that can not be done with a lock. If
the performance is a problem, then to paraphrase the Timbot, it may be
questionable if you are using Python appropriately in this case.

Mark.

Mark Hammond wrote:
>
> > Actually, I think all that's needed is another global like
> > the interpreter_lock in ceval.c. Since this lock is only
> > access

[.The previous mail got truncated due to insufficient disk space;
here is a summary...]

Mark Hammond wrote:
>
> > Actually, I think all that's needed is another global like
> > the interpreter_lock in ceval.c. Since this lock is only
> > accessed via abstract functions, I presume the unlock flag could
> > easily be added.
>
> Well, my personal opinion is that this is really quite wrong. The most
> obvious thing to me is that we are exposing an implementation detail we all
> would dearly like to see removed one day - the global interpreter lock.
>
> But even if we ignore that, it seems to me that you are describing an
> application abstraction, not a language abstraction. This thread started
> with Jim wanting a thread-safe, atomic list operation. This is not an
> unusual requirement (ie, a thread-safe, atomic operation), so languages
> give you access to primitives that let you build this.
>
> To my mind, you are asking for the equivilent of a C function that says
> "suspend all threads except me, cos Im doing something _really_ important".
> C does not provide that, and I have never thought it should. As Gordon
> said, Win32 has critical sections, but these are really just lightweight
> locks. I really dont see how Python is different - it gives you all the
> tools you need to build these abstractions.
>
> I really dont see what you are after that can not be done with a lock. If
> the performance is a problem, then to paraphrase the Timbot, it may be
> questionable if you are using Python appropriately in this case.

The locked section may not be leading in the right direction,
but it surely helps in situations where you cannot otherwise
enforce useage of an object specific lock, e.g. for builtin
file objects (some APIs insist on getting the real thing, not
a thread safe wrapper).

Here is a hack that let's you do much the same with an
unpatched Python interpreter:

sys.setcheckinterval(sys.maxint) # *)
# >=10 Python OPs to flush the ticker counter and have the new
# check interavl setting take effect:
0==0; 0==0; 0==0; 0==0
try:

...lock section...

finally:
sys.setcheckinterval(10)

*) sys.setcheckinterval should really return the previous value
so that we can reset the value to the original one afterwards.

Note that the lock section may not call code which uses the
Py_*_ALLOW_THREADS macros.

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: 157 days left
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/

[Marc writes]

> The locked section may not be leading in the right direction,
> but it surely helps in situations where you cannot otherwise
> enforce useage of an object specific lock, e.g. for builtin
> file objects (some APIs insist on getting the real thing, not
> a thread safe wrapper).

Really, all this boils down to is that you want a Python-ish critical
section - ie, a light-weight lock. This presumably would be desirable if
it could be shown Python locks are indeed "heavy" - I know that from the C
POV they may be considered as such, but I havent seen many complaints about
lock speed from Python.

So in an attempt to get _some_ evidence, I wrote a test program that used
the Queue module to append 10000 integers then remove them all. I then
hacked the queue module to remove all locking, and ran the same test. The
results were 2.4 seconds for the non-locking version, vs 3.8 for the
standard version. Without time (or really inclination <wink>) to take this
further, it _does_ appear a native Python "critical section" could indeed
save a few milli-seconds for a few real-world apps.

So if we ignore the implementation details Marc started spelling, does the
idea of a Python "critical section" appeal? Could simply be a built-in way
of saying "no other _Python_ threads should run" (and of-course the "allow
them again"). The semantics could be simply to ensure the Python program
integrity - it need say nothing about the Python internal "state" as such.

Mark.