Mailing List Archive

David Steuber wrote:

> Exactly how slow are we talking about here?

Depends completely on who you're talking to.

> I want to use Python for an interactive program. Speed is nice, but
> it only has to be fast enough to not frustrate the user. I also
> expect to use it for prototyping. The plan is to use the Python
> interpreter in the application for an extension and control
> language. Again, speed is nice, but it may not be critical.

Most of the specific complaints about Python's slowness come from
people doing regular expression searches of large text files and
comparing the speed with Perl. No question, Perl wins.

Another set of complaints is accompanied by code, and the observation
that "this seems slow, how can I speed it up". Usually, an
experienced Pythoner can significantly speed it up (albeit
sometimes at the expense of clarity).

This thread is a bit unusual in being a general complaint, based on
measuring Python's function call overhead. Again, there's some truth
there, but it doesn't appear to be based on a user's perception of
slowness of an app.

Put another way - I use Python extensively. Every once in a while I
find I need to profile and optimize some subsection (usually just the
inner layers of nested loops).

- Gordon

> Exactly how slow are we talking about here?

I haven't noticed a problem with Python's speed. I used to do a lot of
interactive work in MATLAB (which is VERY slow). So from my perspective I
was getting a huge boost in speed. :-)


> I want to use Python for an interactive program.

Anecdotal evidence from me says no problem.

Gordon McMillan wrote:

> Most of the specific complaints about Python's slowness come from
> people doing regular expression searches of large text files and
> comparing the speed with Perl. No question, Perl wins.
>

...Although, not by enough to it worth programming in perl:

I have a program that pre-processes web server log files for me before
I run them through Analog. I wrote it three times - in perl, python and
java - as a speed and style comparison.

The program basically gets rid of junk in the file that would through off
statistics - it checks for likely web spiders, and filters out those
entries. It filters out any entries I cause while checking the site.
So, by nature it does lots of regex work on large (multi-megabyte) files.

At first my perl program was 3x faster than the other two, but it was a
typical quick perl hack:

while(<>) {
next if /----/;
...
print;
}

...as my requirements got more complex I rewrote it to be more modular,
using data structures and function calls. That really bumped up the perl
execution time. Now the speed of the three programs is very close -
close enough that the coding style of python or java outweighs any
benefit of perl's speed. I haven't done strict tests - just observed the
programs in real-world use.

- Robb

>>>>> "Gordon" == Gordon McMillan <gmcm@hypernet.com> writes:

[deletia]
Gordon> This thread is a bit unusual in being a general complaint,
Gordon> based on measuring Python's function call overhead. Again,
Gordon> there's some truth there, but it doesn't appear to be
Gordon> based on a user's perception of slowness of an app.

Yes the problem is that the more you do in Python the more function call overhead
becomes the dominating factor for the performance of Python. Of course sometimes you
can escape by using builtin functions or even write your own C
functions just to speed up things.

I really don't want to be forced to do that because I would like to
write as much as possible in Python, just because it's so much more
productive and so much more fun.

I was comparing it with a free Smalltalk implementation (made by the
inventors of Smalltalk) because I think this is what the performance
should be.

The squeak virtual machine is written in a (small)
subset of Smalltalk and is automatically translated to C.
It's amazing to see that the resulting code is still faster than most
comparable scripting languages.


Markus
--
Markus Kohler mailto:markus_kohler@hp.com

Markus Kohler wrote:
>
> >>>>> "Gordon" == Gordon McMillan <gmcm@hypernet.com> writes:
>
> [deletia]
> Gordon> This thread is a bit unusual in being a general complaint,
> Gordon> based on measuring Python's function call overhead. Again,
> Gordon> there's some truth there, but it doesn't appear to be
> Gordon> based on a user's perception of slowness of an app.
>
> Yes the problem is that the more you do in Python the more function call overhead
> becomes the dominating factor for the performance of Python. Of course sometimes you
> can escape by using builtin functions or even write your own C
> functions just to speed up things.
>
> I really don't want to be forced to do that because I would like to
> write as much as possible in Python, just because it's so much more
> productive and so much more fun.

Have you tried the P2C "compiler" ?

http://lima.mudlib.org/~rassilon/p2c/

> I was comparing it with a free Smalltalk implementation (made by the
> inventors of Smalltalk) because I think this is what the performance
> should be.
>
> The squeak virtual machine is written in a (small)
> subset of Smalltalk and is automatically translated to C.
> It's amazing to see that the resulting code is still faster than most
> comparable scripting languages.

The last conference had some papers about "compiling" byte-code
to C... as I remember the results were not too thrilling: the
Python function call is one single byte code and thus eliminating
the loop doesn't give you the expected performance increase.

To enhance the function call performance, you'd probably have
to dig very deep into the interpreter's core and change some
concepts as well (e.g. the execution frame concept vs. C stack,
dynamic lookups vs. variable slots). All this is doable, but not
without breaking code.

--
Marc-Andre Lemburg
______________________________________________________________________
Y2000: Y2000: 241 days left
Business: http://www.lemburg.com/
Python Pages: http://starship.python.net/crew/lemburg/

M.-A. Lemburg wrote:
>
> Markus Kohler wrote:
> >
> > >>>>> "Gordon" == Gordon McMillan <gmcm@hypernet.com> writes:
> >
> > [deletia]
> > Gordon> This thread is a bit unusual in being a general complaint,
> > Gordon> based on measuring Python's function call overhead. Again,
> > Gordon> there's some truth there, but it doesn't appear to be
> > Gordon> based on a user's perception of slowness of an app.
> >
> > Yes the problem is that the more you do in Python the more function call overhead
> > becomes the dominating factor for the performance of Python. Of course sometimes you
> > can escape by using builtin functions or even write your own C
> > functions just to speed up things.
> >
> > I really don't want to be forced to do that because I would like to
> > write as much as possible in Python, just because it's so much more
> > productive and so much more fun.
>
> Have you tried the P2C "compiler" ?
>
> http://lima.mudlib.org/~rassilon/p2c/

Yes, the speedup is not that great. As far as I can remember not more
tham 30%.
From what quantify tells me it's still spending a lot of time in
function calling code.

>
> > I was comparing it with a free Smalltalk implementation (made by the
> > inventors of Smalltalk) because I think this is what the performance
> > should be.
> >
> > The squeak virtual machine is written in a (small)
> > subset of Smalltalk and is automatically translated to C.
> > It's amazing to see that the resulting code is still faster than most
> > comparable scripting languages.
>
> The last conference had some papers about "compiling" byte-code
> to C... as I remember the results were not too thrilling: the
> Python function call is one single byte code and thus eliminating
> the loop doesn't give you the expected performance increase.
>
> To enhance the function call performance, you'd probably have
> to dig very deep into the interpreter's core and change some
> concepts as well (e.g. the execution frame concept vs. C stack,
> dynamic lookups vs. variable slots). All this is doable, but not
> without breaking code.
>
> --

I will be posting some suggestions to comp.lang.python soon.
Also I don't know thery are feasible, I think some of them could
speed up things noticable.

Markus
--
Markus Kohler mailto:markus_kohler@hp.com

Markus Kohler wrote:
>
> M.-A. Lemburg wrote:
> >
> > Markus Kohler wrote:
> > >
> > > >>>>> "Gordon" == Gordon McMillan <gmcm@hypernet.com> writes:

[much deleted]

> > Have you tried the P2C "compiler" ?
> >
> > http://lima.mudlib.org/~rassilon/p2c/
>
> Yes, the speedup is not that great. As far as I can remember not more
> tham 30%.
> >From what quantify tells me it's still spending a lot of time in
> function calling code.

P2C does all it can do to get the interpreter overhead away.
For me this turns out as a nice try but the wrong idea.
These 30% generate a C code bloat of hundreds of kilobytes
of C code, while the internal function call overhead is still
there, eating probably at least as much.

If such code could be generated in memory on the fly, well
then I'd consider it. Perhaps we should provide a C compiler
as an optional extension module for a couple of platforms
and use it just for time critical cases.

Still there is the 30% limitation which makes me doubt
if it makes sense to us this approach at all. If we were
able to break the C API after certain objects are known,
and if we had access to some processable description of
their functionality at run time, then this limitation
could be overcome.
This is one advantage of JPython which has no internal
language barrier. A good JIT can optimize away, since
everything is Java machine code internally. And as I read
in the JPython list, with the latest compilers JPython
turns out to be slightly faster than CPython for some tests.

ciao - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home

Christian Tismer wrote:
>
> Markus Kohler wrote:
> >
> > M.-A. Lemburg wrote:
> > >
> > > Markus Kohler wrote:
> > > >
> > > > >>>>> "Gordon" == Gordon McMillan <gmcm@hypernet.com> writes:
>
> [much deleted]
>
> > > Have you tried the P2C "compiler" ?
> > >
> > > http://lima.mudlib.org/~rassilon/p2c/
> >
> > Yes, the speedup is not that great. As far as I can remember not more
> > tham 30%.
> > >From what quantify tells me it's still spending a lot of time in
> > function calling code.
>
> P2C does all it can do to get the interpreter overhead away.
> For me this turns out as a nice try but the wrong idea.
> These 30% generate a C code bloat of hundreds of kilobytes
> of C code, while the internal function call overhead is still
> there, eating probably at least as much.

Yes. Could it be that the problem is that for the CALL_FUNCTION byte
code
call code_eval2 is called recursively ?


>
> If such code could be generated in memory on the fly, well
> then I'd consider it. Perhaps we should provide a C compiler
> as an optional extension module for a couple of platforms
> and use it just for time critical cases.
>
> Still there is the 30% limitation which makes me doubt
> if it makes sense to us this approach at all. If we were
> able to break the C API after certain objects are known,
> and if we had access to some processable description of
> their functionality at run time, then this limitation
> could be overcome.

IMHO the 30% are not worth the code bloat most times.

> This is one advantage of JPython which has no internal
> language barrier. A good JIT can optimize away, since
> everything is Java machine code internally. And as I read
> in the JPython list, with the latest compilers JPython
> turns out to be slightly faster than CPython for some tests.
>
Interesting ...

Markus
--
Markus Kohler mailto:markus_kohler@hp.com

Christian Tismer wrote:
>
> Markus Kohler wrote:
> >
> > M.-A. Lemburg wrote:
> > >
> > > Markus Kohler wrote:
> > > >
> > > > >>>>> "Gordon" == Gordon McMillan <gmcm@hypernet.com> writes:
>
> [much deleted]
>
> > > Have you tried the P2C "compiler" ?
> > >
> > > http://lima.mudlib.org/~rassilon/p2c/
> >
> > Yes, the speedup is not that great. As far as I can remember not more
> > tham 30%.
> > >From what quantify tells me it's still spending a lot of time in
> > function calling code.
>
> P2C does all it can do to get the interpreter overhead away.
> For me this turns out as a nice try but the wrong idea.
> These 30% generate a C code bloat of hundreds of kilobytes
> of C code, while the internal function call overhead is still
> there, eating probably at least as much.
>

P2C uses PyEval_CallObjectWithKeywords which several levels later calls
vgetargs1
which seems to parse(!) the arguments.
The P2C compiled version of tak spends more then 50% in this routine.

Markus
--
Markus Kohler mailto:markus_kohler@hp.com