Mailing List Archive

map() methods (was: Re: [Patches] Review (was: Please review before applying))
[Moving this to python-dev because it's a musing

> > The main point is to avoid string.*.
>
> Agreed. Also replacing map by a loop might not even be slower.
> What remains as open question: Several modules need access
> to string constants, and they therefore still have to import
> string.
> Is there an elegant solution to this?

import string

> That's why i asked for some way to access "".__class__ or
> whatever, to get into some common namespace with the constants.

I dunno. However, I've noticed that in many situations where map() could
be used with a string.* function (*if* you care about the speed-up and
you don't care about the readability issue), there's no equivalent
that uses the new string methods. This stems from the fact that map()
wants a function, not a method.

Python 3000 solves this partly, assuming types and classes are unified
there.

Where in 1.5 we wrote

map(string.strip, L)

in Python 3K we will be able to write

map("".__class__.strip, L)

However, this is *still* not as powerful as

map(lambda s: s.strip(), L)

because the former requires that all items in L are in fact strings,
while the latter works for anything with a strip() method (in
particular Unicode objects and UserString instances).

Maybe Python 3000 should recognize map(lambda) and generate more
efficient code for it...

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
[Guido]
> ...
> However, this is *still* not as powerful as
>
> map(lambda s: s.strip(), L)
>
> because the former requires that all items in L are in fact strings,
> while the latter works for anything with a strip() method (in
> particular Unicode objects and UserString instances).
>
> Maybe Python 3000 should recognize map(lambda) and generate more
> efficient code for it...

[s.strip() for s in L]

That is, list comprehensions solved the speed, generality and clarity
problems here before they were discovered <wink>.
Re: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
> > Maybe Python 3000 should recognize map(lambda) and generate more
> > efficient code for it...
>
> [s.strip() for s in L]
>
> That is, list comprehensions solved the speed, generality and clarity
> problems here before they were discovered <wink>.

Ah! I knew there had to be a solution without lambda! :-)

--Guido van Rossum (home page: http://www.python.org/~guido/)
RE: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
Tim> [s.strip() for s in L]

Tim> That is, list comprehensions solved the speed, generality and
Tim> clarity problems here before they were discovered <wink>.

What is the status of list comprehensions in Python? I remember some work
being done several months ago. They definitely don't appear to be in the
1.6a2. Was there some reason to defer them until later?

Skip
RE: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
[Skip Montanaro]
> What is the status of list comprehensions in Python? I remember some work
> being done several months ago. They definitely don't appear to be in the
> 1.6a2. Was there some reason to defer them until later?

Greg Ewing posted a patch to c.l.py that implemented a good start on the
proposal. But nobody has pushed it. I had hoped to, but ran out of time;
not sure Guido even knows about Greg's patch.

Perhaps the 1.6 source distribution could contain a new "intriguing
experimental patches" directory? Greg's list-comp and Christian's Stackless
have enough fans that this would probably be appreciated. Perhaps some
other things too, if we all run out of time (thinking mostly of Vladimir's
malloc cleanup and NeilS's gc).
Re: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
> Greg Ewing posted a patch to c.l.py that implemented a good start on the
> proposal. But nobody has pushed it. I had hoped to, but ran out of time;
> not sure Guido even knows about Greg's patch.

I vaguely remember, but not really. We did use his f(*args, **kwargs)
patches as a starting point for a 1.6 feature though -- if the list
comprehensions are in a similar state, they'd be great to start but
definitely need work.

> Perhaps the 1.6 source distribution could contain a new "intriguing
> experimental patches" directory? Greg's list-comp and Christian's Stackless
> have enough fans that this would probably be appreciated. Perhaps some
> other things too, if we all run out of time (thinking mostly of Vladimir's
> malloc cleanup and NeilS's gc).

Perhaps a webpage woule make more sense? There's no point in loading
every download with this. And e.g. stackless evolves at a much faster
page than core Python.

I definitely want Vladimir's patches in -- I feel very guilty for not
having reviewed his latest proposal yet. I expect that it's right on
the mark, but I understand if Vladimir wants to wait with preparing
yet another set of patches until I'm happy with the design...

--Guido van Rossum (home page: http://www.python.org/~guido/)
Re: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
Hi, I'm back on-line.

> [Tim]
> > Perhaps the 1.6 source distribution could contain a new "intriguing
> > experimental patches" directory? Greg's list-comp and Christian's
> > Stackless have enough fans that this would probably be appreciated.
> > Perhaps some other things too, if we all run out of time (thinking
> > mostly of Vladimir's malloc cleanup and NeilS's gc).

I'd be in favor of including gc as an optional (experimental) feature.
I'm quite confident that it will evolve into a standard feature, in its
current or in an improved state. The overall strategy looks good, but
there are some black spots w.r.t its cost, both in speed and space.
Neil reported in private mail something like 5-10% mem increase, but
I doubt that the picture is so optimistic. My understanding is that
these numbers reflect the behavior of the Linux VMM in terms of effectively
used pages. In terms of absolute, peak requested virtual memory, things
are probably worse than that. We're still unclear on this...

For 1.6, the gc option would be a handy tool for detecting cyclic trash.
It will answer some expectations, and I believe we're ready to give some
good feedback on its functioning, its purpose, its limitations, etc.
By the time 1.6 is finalized, I expect that we'll know roughly its cost
in terms of mem overhead. Overall, it would be nice to have it in the
distrib as an experimental feature -- it would both bootstrap some useful
feedback, and would encourage enthousiasts to look more closely at DSA/GC
(DSA - dynamic storage allocation). By 1.7 (with Py3K on the horizon),
we would have a good understanding on what to do with gc and how to do it.

If I go one step further, what I expect is that the garbage collector
would be enabled together with a Python-specific memory allocator which
will compensate the cost introduced by the collector. There will some
some stable state again (in terms of speed and size) similar to what we
have now, but with a bonus pack of additional memory services.

> I definitely want Vladimir's patches in -- I feel very guilty for not
> having reviewed his latest proposal yet. I expect that it's right on
> the mark, but I understand if Vladimir wants to wait with preparing
> yet another set of patches until I'm happy with the design...

Yes, I'd prefer to wait and get it right. There's some basis, but it
needs careful rethinking again. I'm willing to fit in the 1.6 timeline
but I understand very well that it's a matter of time :-).

--
Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
RE: map() methods (was: Re: [Patches] Review (was: Please review before applying)) [ In reply to ]
[Vladimir Marangozov, on NeilS's gc patch]
> ...
> The overall strategy looks good, but there are some black spots
> w.r.t its cost, both in speed and space. Neil reported in private
> mail something like 5-10% mem increase, but I doubt that the picture
> is so optimistic. My understanding is that these numbers reflect
> the behavior of the Linux VMM in terms of effectively used pages. In
> terms of absolute, peak requested virtual memory, things are probably
> worse than that. We're still unclear on this...

Luckily, that's what Open Source is all about: if we have to wait for you
(or Neil, or Guido, or anyone else) to do a formal study of the issue, the
patch will never go in. Put the code out there and let people try it, and
50 motivated users will run the only 50 tests that really matter: i.e.,
does their real code suffer or not? If so, a few of them may even figure
out why.

less-thought-more-eyeballs-ly y'rs - tim