Mailing List Archive

On 07. 02. 22 14:26, Victor Stinner wrote:
> On Mon, Feb 7, 2022 at 2:08 PM Petr Viktorin <encukou@gmail.com> wrote:
>> Basically, instead of "We'll remove this API now because it prevents
>> moving to a hypothetical moving garbage collector", it should be "Here
>> is a moving garbage collector that speeds Python up by 30%, but to add
>> it we need to remove these 30 deprecated APIs". The deprecation can be
>> proactive, but not the removal.
>
> PEP 674 gives 3 concrete examples of issues already affecting the
> CPython nogil fork, HPy and GraalPython. They are not hypothetical.

HPy and GraalPython are different projects, and their issues don't
affect current users of CPython.
It would be great to make an easy-to-use API that would be compatible
with HPy and GraalPython. But not at the cost of breaking existing code.
Are the advantages of moving to HPy or Graalpython worth porting the
code? IMO that's a question each extension author should be able to ask.
They shouldn't be forced to do the port by CPython.

As for nogil, that's a promising experiment. If it turns out to be
successful, let's remove the parts that are blocking it. Do we already
know which parts those are?

>
> CPython is also affected by these issues, but the benefits of PEP 674
> (alone) are too indirect, so I chose to avoid mentioning CPython
> issues directly, to avoid confusion.

IMO, CPython issues are the ones most relevant to CPython. If yo're
bringing them up, could you be more specific about them?
If we don't discuss them, how do we know if there better ways to solve
them than what you're proposing?


> It's possible to workaround them: more or less copy/paste CPython
> inefficient code, as PyPy did years ago. The problem is that the
> workaround is inefficient and so PyPy cpyext remains slow. Well, HPy
> address the cpyext performance problem for PyPy and GraalPython ;-)

Hooray! So, the issue is addressed, and we don't need to break code that
doesn't care about performance on PyPy! Sounds perfect!
More seriously: I expect that there are a lot of extension authors that
value API stability over performance on PyPy. They're doing nothing
wrong and they don't deserve to be punished for using API they had
available (and docs they had available) some years ago.

> I don't think that the question is if there is a real problem or not.
> The question is what's the best migration plan to move existing C
> extensions towards a better API which don't suffer from these
> problems.
>
> Once 95% of C extensions will use the limited C API, we would still
> not be able to change Python internals, because of the 5% remaining C
> extensions which are stuck at the legacy C API.

And don't forget extensions that use the "bad" parts of limited API,
like using Py_TYPE as l-value in Python 3.10-.
IMO, the required ABI changes will be more drastic than the API changes.

When we want to change CPython internals, IMO we should know the exact
set of APIs that'll need to be removed. Without an actual proposed
change, I don't think we can know that.
HPy is a good indicator of what API should be deprecated (i.e. where we
should find a better way of doing things), but I don't think it gives
good reasons for breaking changes.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4A7X7WT2MGXXIZUFEMUE6QBVRWPUMZC2/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Feb 7, 2022 at 2:26 PM Victor Stinner <vstinner@python.org> wrote:
> CPython is also affected by these issues, but the benefits of PEP 674
> (alone) are too indirect, so I chose to avoid mentioning CPython
> issues directly, to avoid confusion.

A concrete example of problem caused by exposing structures in the C
API (unrelated to PEP 674). It's a tricky problem...

typedef struct {
PyObject_VAR_HEAD
Py_hash_t ob_shash;
char ob_sval[1];
} PyBytesObject;

The "char ob_sval[1];" syntax used to declare an array is an undefined
behavior if the array is longer in memory. On a bytes object of 4
bytes, accessing ob_sval[3] works, but is an undefined behavior.

=> see https://bugs.python.org/issue40120 for details

The problem can be solved by using "char ob_sval[];" syntax, but we
cannot use this syntax in the public C header, since it causes
compiler errors if the header is built with a C++ compiler (not to
build Python itself, but build a C++ extension using the Python C
API). Removing the structure from the public C API would solve the C++
issue.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LXZCN6IZYZH6XTHXKWSRIFJOHTCB4MAN/
Code of Conduct: http://python.org/psf/codeofconduct/

On 07. 02. 22 15:29, Victor Stinner wrote:
> On Mon, Feb 7, 2022 at 2:26 PM Victor Stinner <vstinner@python.org> wrote:
>> CPython is also affected by these issues, but the benefits of PEP 674
>> (alone) are too indirect, so I chose to avoid mentioning CPython
>> issues directly, to avoid confusion.
>
> A concrete example of problem caused by exposing structures in the C
> API (unrelated to PEP 674). It's a tricky problem...
>
> typedef struct {
> PyObject_VAR_HEAD
> Py_hash_t ob_shash;
> char ob_sval[1];
> } PyBytesObject;
>
> The "char ob_sval[1];" syntax used to declare an array is an undefined
> behavior if the array is longer in memory. On a bytes object of 4
> bytes, accessing ob_sval[3] works, but is an undefined behavior.
>
> => see https://bugs.python.org/issue40120 for details
>
> The problem can be solved by using "char ob_sval[];" syntax, but we
> cannot use this syntax in the public C header, since it causes
> compiler errors if the header is built with a C++ compiler (not to
> build Python itself, but build a C++ extension using the Python C
> API). Removing the structure from the public C API would solve the C++
> issue.

That sounds like a good candidate for deprecation, and adding proper
getters if we didn't have them. But I don't think we actually need to
remove the deprecated struct from the public C API.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3NYGRFIDNBL2MBR66O4HKYVZYS2WEVP5/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, 7 Feb 2022 15:29:39 +0100
Victor Stinner <vstinner@python.org> wrote:
> On Mon, Feb 7, 2022 at 2:26 PM Victor Stinner <vstinner@python.org> wrote:
> > CPython is also affected by these issues, but the benefits of PEP 674
> > (alone) are too indirect, so I chose to avoid mentioning CPython
> > issues directly, to avoid confusion.
>
> A concrete example of problem caused by exposing structures in the C
> API (unrelated to PEP 674). It's a tricky problem...
>
> typedef struct {
> PyObject_VAR_HEAD
> Py_hash_t ob_shash;
> char ob_sval[1];
> } PyBytesObject;
>
> The "char ob_sval[1];" syntax used to declare an array is an undefined
> behavior if the array is longer in memory. On a bytes object of 4
> bytes, accessing ob_sval[3] works, but is an undefined behavior.
>
> => see https://bugs.python.org/issue40120 for details
>
> The problem can be solved by using "char ob_sval[];" syntax, but we
> cannot use this syntax in the public C header, since it causes
> compiler errors if the header is built with a C++ compiler (not to
> build Python itself, but build a C++ extension using the Python C
> API). Removing the structure from the public C API would solve the C++
> issue.

You could also have something like:

typedef struct {
PyObject_VAR_HEAD
Py_hash_t ob_shash;
#ifdef __cpluscplus
char ob_sval[1];
#else
char ob_sval[];
#endif
} PyBytesObject;


Regards

Antoine.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HWXZIISALVHAHT4HSBG33S7IFKLIYBNG/
Code of Conduct: http://python.org/psf/codeofconduct/

On 03. 02. 22 1:40, Guido van Rossum wrote:
[...]
>
> I understand the CPython is stuck supporting the de-facto standard C API
> for a long time. But unless we pick a "north star" (as people call it
> nowadays) of what we want to support in say 5-10 years, the situation
> will never improve.
>
> My point about "getting one chance to get it right in the next decade"
> is that we have to pick that north star, so we can tell users which
> horse to bet on. If the north star we pick is HPy, things will be clear.
> If it is evolving the C API things will also be clear. But I think we
> have to pick one, and stick to it so users (i.e., package
> maintainers/developers) have clarity.

A few months later, here's a draft of a “north star” document.
Is this close to what you had in mind?

https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu9Mqc/edit#


Please comment (here or there) as appropriate :)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DUWBMGLEYP6VFFT7OMMA6KJNJKTEY47R/
Code of Conduct: http://python.org/psf/codeofconduct/

On 4/4/2022 11:19 AM, Petr Viktorin wrote:
> On 03. 02. 22 1:40, Guido van Rossum wrote:
> [...]
>>
>> I understand the CPython is stuck supporting the de-facto standard C
>> API for a long time. But unless we pick a "north star" (as people call
>> it nowadays) of what we want to support in say 5-10 years, the
>> situation will never improve.
>>
>> My point about "getting one chance to get it right in the next decade"
>> is that we have to pick that north star, so we can tell users which
>> horse to bet on. If the north star we pick is HPy, things will be
>> clear. If it is evolving the C API things will also be clear. But I
>> think we have to pick one, and stick to it so users (i.e., package
>> maintainers/developers) have clarity.
>
> A few months later, here's a draft of a “north star” document.
> Is this close to what you had in mind?
>
> https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu9Mqc/edit#

"[.There is a proposal for an additional “unstable” ring for even deeper
integration -- compilers/debuggers. I'm listing it here even though it's
not status quo :)]

Private API -- internal use only (with specific exceptions)"

The Private API is the unstable ring, subject to change each bug-fix
release. The proposal is for a semi-stable ring, stable for each
version. I agree that 'Semi-public' would be a good name.

"(with specific exceptions)" seems like a rough edge. Would most (all?)
exceptions go in the Semi-public ring?

It seems to me that a big driver for this discussion is the current push
to get Python-level results much faster, which most people agree is a
good thing. That requires eliminating duplication, doing some thing
faster, and somethings not at all. That in turn requires revising
internal structures along with code paths. And that in turn requires
revision of external code deeply integrated with the core. I think it
worth making it clear that the resulting pain is needed for something
that benefits most Python users.

--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RTO6CPSNIZ5L6N7NGDLGOOKNHDKQIOCJ/
Code of Conduct: http://python.org/psf/codeofconduct/

IMO it would be better to keep the HPy design as the long term goal:

* Refer to Python objects with opaque handles
* All structures are opaque (with a few exceptions, like PyType_Spec)

It will likely take multiple iterations (Python releases) to reach
this goal, and incompatible C API changes may need a PEP (like PEP
674), but IMO it's good to keep this goal in mind.

Otherwise, it's not easy to understand the rationale for changes like
https://peps.python.org/pep-0674/ "PEP 674 – Disallow using macros as
l-values".

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3FYHB74CF6XBADFRLUVFV6NUZKXRSBSY/
Code of Conduct: http://python.org/psf/codeofconduct/

6 months ago, I wrote a different document based on HPy Manifesto:

"PEP: Taking the Python C API to the Next Level"
https://mail.python.org/archives/list/python-dev@python.org/message/RA7Q4JAUEITJBOUAXFEJKRRM2RR3QSZI/

Victor

On Mon, Apr 4, 2022 at 5:20 PM Petr Viktorin <encukou@gmail.com> wrote:
>
> On 03. 02. 22 1:40, Guido van Rossum wrote:
> [...]
> >
> > I understand the CPython is stuck supporting the de-facto standard C API
> > for a long time. But unless we pick a "north star" (as people call it
> > nowadays) of what we want to support in say 5-10 years, the situation
> > will never improve.
> >
> > My point about "getting one chance to get it right in the next decade"
> > is that we have to pick that north star, so we can tell users which
> > horse to bet on. If the north star we pick is HPy, things will be clear.
> > If it is evolving the C API things will also be clear. But I think we
> > have to pick one, and stick to it so users (i.e., package
> > maintainers/developers) have clarity.
>
> A few months later, here's a draft of a “north star” document.
> Is this close to what you had in mind?
>
> https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu9Mqc/edit#
>
>
> Please comment (here or there) as appropriate :)
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DUWBMGLEYP6VFFT7OMMA6KJNJKTEY47R/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NMMFZSCOEKQDCGUAVIOWV3NYZ5M6NQMJ/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 5 Apr 2022 22:54:00 +0200
Victor Stinner <vstinner@python.org> wrote:
> IMO it would be better to keep the HPy design as the long term goal:
>
> * Refer to Python objects with opaque handles
> * All structures are opaque (with a few exceptions, like PyType_Spec)

If the HPy design is the long term goal, why not just recommend that
people use HPy? And keep the C API for expert users with specific
needs that are not accomodated by HPy.

To me, it seems that trying to change the C API to be "like HPy" is
creating a lot of work, churn and pain for little gain.

(and, yes, perhaps HPy needs to be funded or supported by the PSF if it
doesn't advance fast enough)

Regards

Antoine.


>
> It will likely take multiple iterations (Python releases) to reach
> this goal, and incompatible C API changes may need a PEP (like PEP
> 674), but IMO it's good to keep this goal in mind.
>
> Otherwise, it's not easy to understand the rationale for changes like
> https://peps.python.org/pep-0674/ "PEP 674 – Disallow using macros as
> l-values".
>
> Victor



_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VMSSS46YWQTJYYSPZVCJXNUPY4L5YSIT/
Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Apr 20, 2022 at 10:03 AM Antoine Pitrou <antoine@python.org> wrote:
> If the HPy design is the long term goal, why not just recommend that
> people use HPy? And keep the C API for expert users with specific
> needs that are not accomodated by HPy.
>
> To me, it seems that trying to change the C API to be "like HPy" is
> creating a lot of work, churn and pain for little gain.

If you put HPy aside, "Fixing" the C API has multiple advantages for
CPython and its (C API) users.

For consumers of the C API (C extensions, Cython, pybind11, etc.),
once most implementation details will be hidden, the C API will become
way more stable. The "C API > Porting to Python x.y" section of What's
New in Python x.y should become way shorter. Or at least, the number
of impacted C extensions should be way smaller. Sadly, fixing the C
API to hide implementation details requires to adapt (modify) C
extensions. Even if usually only a few lines should be changed, and
the pythoncapi-compat project now automates most of these changes.

For CPython, no longer leaking implementation details allow to change
"any implementation detail" without getting a heavy and annoying
backfire from grumpy users should be very comfortable. In Python 3.9,
3.10 and 3.11 development cycles, we got backfire multiple times, and
each time, it was really unpleasant both for CPython core devs and for
C extensions maintainers (both have legit concerns and use cases).

While these changes should ease the migration to HPy, it's not my
goal. HPy requires to add a "ctx" parameter, it's a different API
(there are multiple subtle differences).


> (and, yes, perhaps HPy needs to be funded or supported by the PSF if it
> doesn't advance fast enough)

What can be done in practice for that? If I understood correctly,
Oracle is sponsoring the project since they want to use HPy for
GraalPython (of their GraalVM).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PB7BG7W76BQXP3KS3C6XT5GOFJS24LZB/
Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, 20 Apr 2022 12:52:53 +0200
Victor Stinner <vstinner@python.org> wrote:
> On Wed, Apr 20, 2022 at 10:03 AM Antoine Pitrou <antoine@python.org> wrote:
> > If the HPy design is the long term goal, why not just recommend that
> > people use HPy? And keep the C API for expert users with specific
> > needs that are not accomodated by HPy.
> >
> > To me, it seems that trying to change the C API to be "like HPy" is
> > creating a lot of work, churn and pain for little gain.
>
> If you put HPy aside, "Fixing" the C API has multiple advantages for
> CPython and its (C API) users.

With the caveat that the "fixing" probably requires users to fix their
packages as well.

> For consumers of the C API (C extensions, Cython, pybind11, etc.),
> once most implementation details will be hidden, the C API will become
> way more stable.

The *API* is quite stable already if you don't use the private/internal
functions. Perhaps you're thinking about the ABI?

> > (and, yes, perhaps HPy needs to be funded or supported by the PSF if it
> > doesn't advance fast enough)
>
> What can be done in practice for that? If I understood correctly,
> Oracle is sponsoring the project since they want to use HPy for
> GraalPython (of their GraalVM).

Also, Anaconda recently hired Antonio Cuni, hopefully giving him
sufficient time to work on HPy. So perhaps nothing needs to be done in
practice.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QKWLX5223UD2ZCDYJLRDEXISFVFYWU7P/
Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Apr 20, 2022 at 1:44 PM Antoine Pitrou <antoine@python.org> wrote:
> > For consumers of the C API (C extensions, Cython, pybind11, etc.),
> > once most implementation details will be hidden, the C API will become
> > way more stable.
>
> The *API* is quite stable already if you don't use the private/internal
> functions. Perhaps you're thinking about the ABI?

In Fedora, we update Python early during Python alpha versions, and
sadly it's common that many C extensions are incompatible (need to be
modified) at each 3.x release. A single minor incompatible change is
enough to require changing a C extension.

I believe that once the C API will leak less implementation details,
changing Python will impact less C extensions. HPy API looks more
stable by design: it's way smaller and only expose the bare minimum.

I took notes on (Python and C API) incompatible changes, impacting
most Python projects and C extensions, from Python 3.7 to Python 3.11:
https://github.com/vstinner/vstinner.github.io/blob/pelican/draft/python-incompatible-changes.rst

"C API > Porting to Python 3.11" section is quite long, PyFrameObject
and PyThreadState structures changed a lot (PyFrameObject moved to the
internal C API):
https://docs.python.org/dev/whatsnew/3.11.html#id6

"C API > Porting to Python 3.10":
https://docs.python.org/dev/whatsnew/3.10.html#id2

"C API > Porting to Python 3.9":
https://docs.python.org/dev/whatsnew/3.9.html#id2

"Porting to Python 3.8 > Changes in C API":
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/M6MINEKX7XVQ72IVI3D4XTP2O2J5LD3F/
Code of Conduct: http://python.org/psf/codeofconduct/