Hello,
Victor and others are organizing the C-API to make it clearer what's
public and what's private (see Victor's file-based stats:
https://pythoncapi.readthedocs.io/stats.html)
It became clear that we need a tier between public and private C-API.
PEP 689 proposes such API, with an opt-in macro and transition period.
Please discuss.
And if you can think of a better name, that would be great :)
The PEP is at: https://peps.python.org/pep-0689/
Thread where this started:
https://mail.python.org/archives/list/python-dev@python.org/thread/MA4FQ7G6F35NG3TUN6RQPXRGXTYMFMDY/#MA4FQ7G6F35NG3TUN6RQPXRGXTYMFMDY
Implementation notes: https://github.com/python/cpython/issues/91744
Here's the current version for easy quoting:
----------
PEP: 689
Title: Semi-stable C API tier
Author: Petr Viktorin <encukou@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 523
Created: 22-Apr-2022
Python-Version: 3.11
Abstract
========
Some functions and types of the C-API are designated *semi-stable*,
meaning that they will not change in patch (bugfix/security) releases,
but may change between minor releases (e.g. between 3.11 and 3.12) without
deprecation warnings.
Motivation & Rationale
======================
The Python C-API is currently divided into `three tiers
<https://devguide.python.org/c-api/>`__:
- Limited API, with high compatibility expectations
- Public API, which follows the :pep:`backwards compatibility policy
<387>`, and requires deprecation warnings before changes
- Private (internal) API, which can change at any time.
We need a tier between Public and Private to accommodate tools like
advanced debuggers and JIT compilers, which need access to CPython
internals but assume the C-API they use does not change in patch releases.
Setting Stability Expectations
------------------------------
Currently, there are no guarantees for the internal API -- that is, anything
that requires ``Py_BUILD_CORE`` or is named with a leading underscore.
This API can change without warning at any time, and code that uses it
is pinned to a specific build of Python.
However, in practice, even the internal API usually happens to be stable
in patch releases:
- Some CPython core developers take this as an an unwritten rule.
- Patch releases only contain bugfixes, which are unlikely to
change the API.
Semi-stable API will make the stability expectations more explicit.
It will also hopefully encourage existing users of the private API to
reach out to python-dev, so we can expose, standardize and test an API
for some of their use cases.
Reserving underscores for Private API
-------------------------------------
:pep:`523` introduced functions for use by debuggers and JIT compilers,
which are stable only across minor releases.
The functions names have leading underscores to suggest their limited
stability.
However, leading underscores usually mark *fully private* API.
CPython developers familiar with the “underscore means internal”
convention are unlikely to check if underscored functions they are
changing are documented and used outside CPython itself.
This proposal brings us a bit closer to reserving underscores
only for truly private, unstable, hands-off API.
Warning about API that is changed often
---------------------------------------
The ``PyCode_New()`` family is an example of functions that are
documented as unstable, and also often change in practice.
Moving it to the semi-stable tier will make its status obvious even
to people who don't read the docs carefully enough, and will make it
hard to use accidentally.
Changes during the Beta period
------------------------------
Since the API itself can change continuously up until Beta 1 (feature
freeze)
of a minor version, major users of this API are unlikely to test
Alpha releases and provide feedback.
It is very difficult to determine what needs to be exposed as semi-stable.
Additions to the semi-stable tier will count as *stabilization*,
and will be allowed up to Release Candidate 1.
Specification
=============
Several functions and types (“APIs”) will be moved to a new
*semi-stable* tier.
They will be expected to stay stable across patch releases,
but may change or be removed without warning in minor releases (3.x.0),
including Alpha and Beta releases of 3.x.0.
When they change significantly, code that uses them should no longer compile
(e.g. arguments should be added/removed, or a function should be renamed,
but the semantic meaning of an argument should not change).
Their definitions will be moved to a new directory, ``Include/semistable/``,
and will be included from ``Python.h``.
From Python 3.12 on, these APIs will only be usable when the
``Py_USING_SEMI_STABLE_API`` macro is defined.
CPython will only define the macro for building CPython itself
(``Py_BUILD_CORE``).
To make transition to semi-stable API easier,
in Python 3.11 the APIs will be available without
``Py_USING_SEMI_STABLE_API``
defined. In this case, using them will generate a deprecation warning on
compilers that support ``Py_DEPRECATED``.
A similar deprecation period will be used when making more APIs semi-stable
in the future:
- When moving from public API, the deprecation period should follow Python's
backwards compatibility policy (currently, it should last at least
two releases).
- When moving from public API that is documented as unstable,
the deprecation period can only last one release.
- When moving from private API or adding new API, no deprecation period
is necessary.
Leading underscores will be removed from the names of the moved APIs.
The old underscored name of a renamed API will be available (as an alias
using ``#define``) at least until that API changes.
The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be
documented,
and documentation of each semi-stable API will be updated.
Adjustments during Beta periods
-------------------------------
New APIs can be added to the semi-stable tier, and private APIs can be moved
to it, up to the first release candidate of a new minor version.
Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta
period.
In the Beta period, no API may be moved to more private tier, e.g.
what is public in Beta 1 must stay public until the final release.
Initial semi-stable API
-----------------------
The following API will initially be semi-stable.
The set may be adjusted for 3.11.
Code object constructors:
- ``PyCode_New()``
- ``PyCode_NewWithPosOnlyArgs()``
Frame evaluation API (PEP 523):
- ``_PyFrameEvalFunction``
- ``_PyInterpreterState_GetEvalFrameFunc()``
- ``_PyInterpreterState_SetEvalFrameFunc()``
- ``_PyEval_RequestCodeExtraIndex()``
- ``_PyCode_GetExtra()``
- ``_PyCode_SetExtra()``
- ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct)
- ``_PyFrame_GetFrameObject``
- ``PyEval_EvalFrameDefault``
(new function that calls ``_PyEval_EvalFrameDefault``, but takes
``PyFrameObject`` rather than ``_PyInterpreterFrame``)
(Leading underscores will be removed as mentioned above.)
Backwards Compatibility
=======================
The C API backwards compatibility story will be made clearer.
How to Teach This
=================
The changes affect advanced C programmers, who should consult the
updated reference documentation, devguide and/or What's New document·.
Reference Implementation
========================
https://github.com/python/cpython/issues/91744
Rejected Ideas
==============
It might be good to add a similar tier in the Python (not C) API,
e.g. for ``types.CodeType``.
However, the opt-in mechanism would need to be different (if any).
This is outside the scope of the PEP.
Open Issues
===========
- “Semi-stable” is not a perfect name.
- The exact set of exposed API may change.
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PQXSP7E2B6KNXTJ2AERWMKKX42YP5D6O/
Code of Conduct: http://python.org/psf/codeofconduct/
Victor and others are organizing the C-API to make it clearer what's
public and what's private (see Victor's file-based stats:
https://pythoncapi.readthedocs.io/stats.html)
It became clear that we need a tier between public and private C-API.
PEP 689 proposes such API, with an opt-in macro and transition period.
Please discuss.
And if you can think of a better name, that would be great :)
The PEP is at: https://peps.python.org/pep-0689/
Thread where this started:
https://mail.python.org/archives/list/python-dev@python.org/thread/MA4FQ7G6F35NG3TUN6RQPXRGXTYMFMDY/#MA4FQ7G6F35NG3TUN6RQPXRGXTYMFMDY
Implementation notes: https://github.com/python/cpython/issues/91744
Here's the current version for easy quoting:
----------
PEP: 689
Title: Semi-stable C API tier
Author: Petr Viktorin <encukou@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Requires: 523
Created: 22-Apr-2022
Python-Version: 3.11
Abstract
========
Some functions and types of the C-API are designated *semi-stable*,
meaning that they will not change in patch (bugfix/security) releases,
but may change between minor releases (e.g. between 3.11 and 3.12) without
deprecation warnings.
Motivation & Rationale
======================
The Python C-API is currently divided into `three tiers
<https://devguide.python.org/c-api/>`__:
- Limited API, with high compatibility expectations
- Public API, which follows the :pep:`backwards compatibility policy
<387>`, and requires deprecation warnings before changes
- Private (internal) API, which can change at any time.
We need a tier between Public and Private to accommodate tools like
advanced debuggers and JIT compilers, which need access to CPython
internals but assume the C-API they use does not change in patch releases.
Setting Stability Expectations
------------------------------
Currently, there are no guarantees for the internal API -- that is, anything
that requires ``Py_BUILD_CORE`` or is named with a leading underscore.
This API can change without warning at any time, and code that uses it
is pinned to a specific build of Python.
However, in practice, even the internal API usually happens to be stable
in patch releases:
- Some CPython core developers take this as an an unwritten rule.
- Patch releases only contain bugfixes, which are unlikely to
change the API.
Semi-stable API will make the stability expectations more explicit.
It will also hopefully encourage existing users of the private API to
reach out to python-dev, so we can expose, standardize and test an API
for some of their use cases.
Reserving underscores for Private API
-------------------------------------
:pep:`523` introduced functions for use by debuggers and JIT compilers,
which are stable only across minor releases.
The functions names have leading underscores to suggest their limited
stability.
However, leading underscores usually mark *fully private* API.
CPython developers familiar with the “underscore means internal”
convention are unlikely to check if underscored functions they are
changing are documented and used outside CPython itself.
This proposal brings us a bit closer to reserving underscores
only for truly private, unstable, hands-off API.
Warning about API that is changed often
---------------------------------------
The ``PyCode_New()`` family is an example of functions that are
documented as unstable, and also often change in practice.
Moving it to the semi-stable tier will make its status obvious even
to people who don't read the docs carefully enough, and will make it
hard to use accidentally.
Changes during the Beta period
------------------------------
Since the API itself can change continuously up until Beta 1 (feature
freeze)
of a minor version, major users of this API are unlikely to test
Alpha releases and provide feedback.
It is very difficult to determine what needs to be exposed as semi-stable.
Additions to the semi-stable tier will count as *stabilization*,
and will be allowed up to Release Candidate 1.
Specification
=============
Several functions and types (“APIs”) will be moved to a new
*semi-stable* tier.
They will be expected to stay stable across patch releases,
but may change or be removed without warning in minor releases (3.x.0),
including Alpha and Beta releases of 3.x.0.
When they change significantly, code that uses them should no longer compile
(e.g. arguments should be added/removed, or a function should be renamed,
but the semantic meaning of an argument should not change).
Their definitions will be moved to a new directory, ``Include/semistable/``,
and will be included from ``Python.h``.
From Python 3.12 on, these APIs will only be usable when the
``Py_USING_SEMI_STABLE_API`` macro is defined.
CPython will only define the macro for building CPython itself
(``Py_BUILD_CORE``).
To make transition to semi-stable API easier,
in Python 3.11 the APIs will be available without
``Py_USING_SEMI_STABLE_API``
defined. In this case, using them will generate a deprecation warning on
compilers that support ``Py_DEPRECATED``.
A similar deprecation period will be used when making more APIs semi-stable
in the future:
- When moving from public API, the deprecation period should follow Python's
backwards compatibility policy (currently, it should last at least
two releases).
- When moving from public API that is documented as unstable,
the deprecation period can only last one release.
- When moving from private API or adding new API, no deprecation period
is necessary.
Leading underscores will be removed from the names of the moved APIs.
The old underscored name of a renamed API will be available (as an alias
using ``#define``) at least until that API changes.
The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be
documented,
and documentation of each semi-stable API will be updated.
Adjustments during Beta periods
-------------------------------
New APIs can be added to the semi-stable tier, and private APIs can be moved
to it, up to the first release candidate of a new minor version.
Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta
period.
In the Beta period, no API may be moved to more private tier, e.g.
what is public in Beta 1 must stay public until the final release.
Initial semi-stable API
-----------------------
The following API will initially be semi-stable.
The set may be adjusted for 3.11.
Code object constructors:
- ``PyCode_New()``
- ``PyCode_NewWithPosOnlyArgs()``
Frame evaluation API (PEP 523):
- ``_PyFrameEvalFunction``
- ``_PyInterpreterState_GetEvalFrameFunc()``
- ``_PyInterpreterState_SetEvalFrameFunc()``
- ``_PyEval_RequestCodeExtraIndex()``
- ``_PyCode_GetExtra()``
- ``_PyCode_SetExtra()``
- ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct)
- ``_PyFrame_GetFrameObject``
- ``PyEval_EvalFrameDefault``
(new function that calls ``_PyEval_EvalFrameDefault``, but takes
``PyFrameObject`` rather than ``_PyInterpreterFrame``)
(Leading underscores will be removed as mentioned above.)
Backwards Compatibility
=======================
The C API backwards compatibility story will be made clearer.
How to Teach This
=================
The changes affect advanced C programmers, who should consult the
updated reference documentation, devguide and/or What's New document·.
Reference Implementation
========================
https://github.com/python/cpython/issues/91744
Rejected Ideas
==============
It might be good to add a similar tier in the Python (not C) API,
e.g. for ``types.CodeType``.
However, the opt-in mechanism would need to be different (if any).
This is outside the scope of the PEP.
Open Issues
===========
- “Semi-stable” is not a perfect name.
- The exact set of exposed API may change.
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PQXSP7E2B6KNXTJ2AERWMKKX42YP5D6O/
Code of Conduct: http://python.org/psf/codeofconduct/