Mailing List Archive

On Tue, May 10, 2022 at 3:15 AM Chris Angelico <rosuav@gmail.com> wrote:

> > It is often the case that developer write Code in Python and then
> convert to a C extension module for performance regions.
> >
> > A C extension module has a lot of boiler plate code - for instance the
> Structures required for each class, the functions for Module initialization
> etc.
> >
> > My Idea is a simple tool that uses introspection tools to take a Python
> module and to generate the relevant boiler plate for the module - including
> blank functions for the module classes and for methods. This tool would use
> type annotations (if given) to make sensible choices for parameter and
> attribute types, including using int and float directly rather than
> Internal objects (depending on tool options).
>

Two things to say about this:
1) Sometimes abandoning a pure python module for a C extension for
performance is a mistake - because Pypy is probably going to be much faster
with the pure python module
2) I've had some luck using m4 to maintain a single source file that is
used to automatically generate both pure python and cython. This is a
little like using cpp in a C project.

For examples of #2, perhaps see:
https://stromberg.dnsalias.org/~strombrg/treap/
https://stromberg.dnsalias.org/svn/rolling_checksum_mod/trunk/
https://stromberg.dnsalias.org/~strombrg/sort-comparison/

It's often nice to keep the lines of the pure-python and cython having a
1-1 relationship, so that tracebacks report useful line numbers either
way. However, in the treap example I've dispensed with that because some
methods were almost identical but had some boilerplate - and m4 was able to
handle that nicely at the cost of lines being 1-1.

HTH
--
https://mail.python.org/mailman/listinfo/python-list

On 5/10/22 5:14 AM, Chris Angelico wrote:
> On Tue, 10 May 2022 at 19:57, anthony.flury
> <anthony.flury@btinternet.com> wrote:
>>
>> On 10/05/2022 09:20, Chris Angelico wrote:
>>
>> On Tue, 10 May 2022 at 18:06, anthony.flury via Python-ideas
>> <python-ideas@python.org> wrote:
>>
>> A proposal for a new tool to be implemented -
>>
>> It is often the case that developer write Code in Python and then convert to a C extension module for performance regions.
>>
>> A C extension module has a lot of boiler plate code - for instance the Structures required for each class, the functions for Module initialization etc.
>>
>> My Idea is a simple tool that uses introspection tools to take a Python module and to generate the relevant boiler plate for the module - including blank functions for the module classes and for methods. This tool would use type annotations (if given) to make sensible choices for parameter and attribute types, including using int and float directly rather than Internal objects (depending on tool options).
>>
>> Yep, that's an awesome idea! Are you aware of Cython? You might be
>> able to make use of that.
>>
>> Chris, Thank you.
>>
>> I am aware of Cython but that isn't quite what I had in mind. I want a tool for a developer who doesn't want to continue to support the Python 'prototype' for whatever reason, ie where they want a complete conversion to C.
>>
>> It might even be possible with inspection of the AST to write some of the code inside the C functions - but that is not for release 0.1 :-)
>>
> You may still be able to take advantage of Cython as part of the
> process. One thing that's really cool about source code is that,
> fundamentally, it's all text... and Python is *great* at manipulating
> text files :) It might be that you can write a script that transforms
> a Python module into a Cython module, which can then be compiled
> as-is, or further processed as needed.
>
> BTW, not sure which list you're intending to discuss this on, so I'm
> just replying on the same list you sent this message to.
>
> ChrisA

This might be what you are looking for?
<https://mypyc.readthedocs.io/en/latest/introduction.html>
--
https://mail.python.org/mailman/listinfo/python-list

On 10/05/2022 15:04, Dan Stromberg wrote:
>
> On Tue, May 10, 2022 at 3:15 AM Chris Angelico <rosuav@gmail.com> wrote:
>
> > It is often the case that developer write Code in Python and
> then convert to a C extension module for performance regions.
> >
> > A C extension module has a lot of boiler plate code - for
> instance the Structures required for each class, the functions for
> Module initialization etc.
> >
> > My Idea is a simple tool that uses introspection tools to take a
> Python module and to generate the relevant boiler plate for the
> module - including blank functions for the module classes and for
> methods. This tool would use type annotations (if given) to make
> sensible choices for parameter and attribute types, including
> using int and float directly rather than Internal objects
> (depending on tool options).
>
>
> Two things to say about this:
> 1) Sometimes abandoning a pure python module for a C extension for
> performance is a mistake - because Pypy is probably going to be much
> faster with the pure python module

Dan,

Thanks for your response, but I think PyPy will have a long way to go to
make JIT generated code more efficient than hand crafted C extension.

for instance PyPy will almost certainly be unable to determine if
integers, floats etc will need to use the Python runtime or can be
optimized to use direct C types, another example would be lists - there
are some cases where the C extension will always be a lot more efficient
using C arrays than using the C list runtime machinery.

While there might be some cases where PyPy will be more efficient, I
don't think that will be the case for all programs, and that ignores the
fact that PyPi has major issues with the CAPI.


> 2) I've had some luck using m4 to maintain a single source file that
> is used to automatically generate both pure python and cython.  This
> is a little like using cpp in a C project.
>
> For examples of #2, perhaps see:
> https://stromberg.dnsalias.org/~strombrg/treap/
> https://stromberg.dnsalias.org/svn/rolling_checksum_mod/trunk/
> https://stromberg.dnsalias.org/~strombrg/sort-comparison/
>
> It's often nice to keep the lines of the pure-python and cython having
> a 1-1 relationship, so that tracebacks report useful line numbers
> either way.  However, in the treap example I've dispensed with that
> because some methods were almost identical but had some boilerplate -
> and m4 was able to handle that nicely at the cost of lines being 1-1.
>
> HTH
--
Anthony Flury
*Moble*: +44 07743 282707
*Home*: +44 (0)1206 391294
*email*: anthony.flury@btinternet.com
--
https://mail.python.org/mailman/listinfo/python-list