Mailing List Archive: PEP 674: Disallow using macros as l-value

On Tue, Dec 7, 2021 at 7:55 PM Sebastian Berg
<sebastian@sipsolutions.net> wrote:
> One thing we once did in NumPy (for a runtime problem), was to
> intentionally break everyone at pre-release/dev time to point out what
> code needed fixing. Then flip the switch back at release time as to
> not break production.
> After a long enough time we enabled it for release mode.
>
> Not saying that it was nice, but it was the only alternative would have
> been to never fix it.
>
> A similar switch could be worthwhile if it helps Victor with
> experimenting on the dev-branch or reach a useful amount of projects.
> Of course, I am not sure it would do either...

Py_TYPE() and Py_SIZE() macros have been converted to static inline
functions in May 2020 (Python 3.10 dev cycle):
https://github.com/python/cpython/commit/ad3252bad905d41635bcbb4b76db30d570cf0087

Honestly, when the change was merged, I didn't expect that it would
break anything. Sadly, it broke multiple projects, including important
ones like Cython and numpy. So it was reverted in Python 3.10.

Once enough projects got prepared for these changes, I merged again
the change in Python 3.11. Then more affected projects have been
discovered.

I also worked on tooling to discover affected projects before users
start complaining ;-) And I wrote the PEP 674 which documents and
justify these changes ;-)

Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EMOJENDQVQOJQA5OHQ6OC5KCYC4PSR4A/
Code of Conduct: http://python.org/psf/codeofconduct/

Hi,

The HPy project is great and I'm supporting this cool project! As I
wrote, IMO it's the best solution for the long term.

My first concern is that right now, HPy is not mature nor widely used,
and CPython still has to continue supporting a large amount of C
extensions which are not likely to be ported to HPy soon.

Well, we can do nothing, continue supporting these C extensions
unmodified and we're good.

My second concern is the inability to evolve CPython internals to
implement new optimizations, and the inefficient implementation of the
current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve
these problems when the last C extensions will be ported to HPy: when
it will be reasonable to drop CPython public C API. Since Python 2.7
is still used in 2021, I expect that porting most C extensions to HPy
will take a few years.

Porting a C extension to HPy can be done incrementally on CPython. But
the port is not cheap, a lot of code has to be adapted.

I'm proposing to make the C API "less bad" by fixing the parts which
are clearly identified as "bad". Only a minority of C extensions
require changes, and on these C extensions, usually only a few lines
need to be changed.

For example, numpy is made of 307,300 lines of C code, and adapting
numpy to the PEP 674 only modified 11 lines (now using Py_SET_TYPE or
Py_SET_SIZE) and to add 4 lines (to define Py_SET_TYPE/Py_SET_SIZE for
Python 3.8 and older). I expect that porting numpy to HPy will require
to modify more lines.

Right now, it's hard to bet which approach is the best: fix the
current C API, or focus on HPy. IMO it would be too risky to only bet
on HPy.

At least, the main advantage of HPy is that it's currently fully
developed outside CPython and it doesn't require any change in the C
API!

Victor

On Wed, Dec 8, 2021 at 6:21 PM Tim Felgentreff
<tim.felgentreff@oracle.com> wrote:
>
> Hi,
>
> On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
> > I wonder if there's a role for HPy in this context? What if instead of
> > evolving the stable ABI and the limited API, instead we were to focus on
> > first-class support for HPy?
>
> The hope with the HPy project is indeed to provide a C API that is close to the
> original API to make porting easy and have it perform as close to the existing
> API as possible. At the same time, HPy is sufficently removed to be a good "C
> extension API" (as opposed to a stable subset of the CPython implementation API)
> that does not leak implementation details. To ensure this latter property is why
> we try to develop everything in parallel for CPython, PyPy, and GraalVM Python.
>
> On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
> > Providing HPy as a first-class citizen in CPython, as already done in
> > PyPy, would be great to promote HPy! However, HPy evolves quickly and
> > so needs to be released more frequently than CPython. At least, we
> > could promote it more in the C API documentation, as we already
> > promote Cython.
>
> Indeed, at this point HPy is still evolving very fast. We are still solving
> issues while migrating NumPy, have begun adding support for HPy to Cython,
> and will start working on pybind11 soon. I believe by the time we have these
> users of the existing C API working, HPy should be in a state where it is
> generally useful and can be deemed stable enough that further development can
> follow a more stable process.
>
> > For me, HPy is the only valid stable API and stable ABI in the long
> > term which is efficient on any Python implementation. Its design is
> > very different than the C API: HPy avoids all C API design mistakes,
> > it doesn't leak any implementation detail.
>
> In the long run the HPy project would like to become a promoted API to
> write Python C extensions.
>
> Tim Felgentreff
> (on behalf of the HPy team)
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2NFJC7UAVIQZ6BNV4NBTY3THI7OIJIMF/
> Code of Conduct: http://python.org/psf/codeofconduct/

--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OKQC45FPL6PII2V5UICVL76VOSQILTYW/
Code of Conduct: http://python.org/psf/codeofconduct/