Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Python>
  3. Dev
1 2  View All
PEP 674: Disallow using macros as l-value
vstinner at python
Nov 30, 2021, 10:27 AM
Post #1 of 27 (2513 views)
Permalink
Hi,

I propose to disallow using macros as l-value. Read and comment the
plain text below, or read the PEP 674 online,
https://python.github.io/peps/pep-0674/

While I'm not sure that the proposed changes are really controversial,
I decided to write a formal PEP since the incompatible changes are not
following the PEP 387 deprecation process and so a format PEP 387
exception is better. Well, the list of modified macros is also quite
long. Moreover, a PEP is a way to document and announce the changes
;-)

The Py_TYPE() and Py_SIZE() changes are already approved the Steering
Council, but I prefer to list them in the PEP:
https://github.com/python/steering-council/issues/79

In practice, I'm only aware of 4 projects impacted by these changes,
and I wrote the pythoncapi_compat project which updates automatically
C extensions: add Python 3.11 support, without losing support for
older Python versions. I already prepared major projects like Cython
and numpy for these changes (in total, 14 impacted projects have
already been updated).

Victor

---

PEP: 674
Title: Disallow using macros as l-value
Author: Victor Stinner <vstinner@python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Oct-2021
Python-Version: 3.11

Abstract
========

Incompatible C API change disallowing using macros as l-value to allow
evolving CPython internals and to ease the C API implementation on other
Python implementation.

In practice, the majority of projects impacted by these incompatible
changes should only have to make two changes:

* Replace ``Py_TYPE(obj) = new_type;``
with ``Py_SET_TYPE(obj, new_type);``.
* Replace ``Py_SIZE(obj) = new_size;``
with ``Py_SET_SIZE(obj, new_size);``.


Rationale
=========

Using a macro as a l-value
--------------------------

In the Python C API, some functions are implemented as macro because
writing a macro is simpler than writing a regular function. If a macro
exposes directly a struture member, it is technically possible to use
this macro to not only get the structure member but also set it.

Example with the Python 3.10 ``Py_TYPE()`` macro::

#define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)

This macro can be used as a **r-value** to **get** an object type::

type = Py_TYPE(object);

It can also be used as **l-value** to **set** an object type::

Py_TYPE(object) = new_type;

It is also possible to set an object reference count and an object size
using ``Py_REFCNT()`` and ``Py_SIZE()`` macros.

Setting directly an object attribute relies on the current exact CPython
implementation. Implementing this feature in other Python
implementations can make their C API implementation less efficient.

CPython nogil fork
------------------

Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch
<https://github.com/colesbury/nogil/>`_. This fork has no
``PyObject.ob_refcnt`` member, but a more elaborated implementation for
reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code
fails with a compiler error.

Merging the nogil fork into the upstream CPython main branch requires
first to fix this C API compatibility issue. It is a concrete example of
a Python optimization blocked indirectly by the C API.

This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
has been already modified to disallow using it as a l-value.

HPy project
-----------

The `HPy project <https://hpyproject.org/>`_ is a brand new C API for
Python using only handles and function calls: handles are opaque,
structure members cannot be accessed directly,and pointers cannot be
dereferenced.

Disallowing the usage of macros as l-value helps the migration of
existing C extensions to HPy by reducing differences between the C API
and the HPy API.

PyPy cpyext module
------------------

In PyPy, when a Python object is accessed by the Python C API, the PyPy
``cpyext`` module has to convert PyPy object to a CPython object. While
PyPy objects are designed to be efficient with the PyPy JIT compiler,
CPython objects are less efficient and increase the memory usage.

This PEP alone is not enough to get rid of the CPython objects in the
PyPy ``cpyext`` module, but it is a step towards this long term goal.
PyPy already supports HPy which is a better solution in the long term.


Specification
=============

Disallow using macros as l-value
--------------------------------

PyObject and PyVarObject macros
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead
* ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead

"GET" macros
^^^^^^^^^^^^

* ``PyByteArray_GET_SIZE()``
* ``PyBytes_GET_SIZE()``
* ``PyCFunction_GET_CLASS()``
* ``PyCFunction_GET_FLAGS()``
* ``PyCFunction_GET_FUNCTION()``
* ``PyCFunction_GET_SELF()``
* ``PyCell_GET()``
* ``PyCode_GetNumFree()``
* ``PyDict_GET_SIZE()``
* ``PyFunction_GET_ANNOTATIONS()``
* ``PyFunction_GET_CLOSURE()``
* ``PyFunction_GET_CODE()``
* ``PyFunction_GET_DEFAULTS()``
* ``PyFunction_GET_GLOBALS()``
* ``PyFunction_GET_KW_DEFAULTS()``
* ``PyFunction_GET_MODULE()``
* ``PyHeapType_GET_MEMBERS()``
* ``PyInstanceMethod_GET_FUNCTION()``
* ``PyList_GET_SIZE()``
* ``PyMemoryView_GET_BASE()``
* ``PyMemoryView_GET_BUFFER()``
* ``PyMethod_GET_FUNCTION()``
* ``PyMethod_GET_SELF()``
* ``PySet_GET_SIZE()``
* ``PyTuple_GET_SIZE()``
* ``PyUnicode_GET_DATA_SIZE()``
* ``PyUnicode_GET_LENGTH()``
* ``PyUnicode_GET_LENGTH()``
* ``PyUnicode_GET_SIZE()``
* ``PyWeakref_GET_OBJECT()``

"AS" macros
^^^^^^^^^^^

* ``PyByteArray_AS_STRING()``
* ``PyBytes_AS_STRING()``
* ``PyFloat_AS_DOUBLE()``
* ``PyUnicode_AS_DATA()``
* ``PyUnicode_AS_UNICODE()``

PyUnicode macros
^^^^^^^^^^^^^^^^

* ``PyUnicode_1BYTE_DATA()``
* ``PyUnicode_2BYTE_DATA()``
* ``PyUnicode_4BYTE_DATA()``
* ``PyUnicode_DATA()``
* ``PyUnicode_IS_ASCII()``
* ``PyUnicode_IS_COMPACT()``
* ``PyUnicode_IS_READY()``
* ``PyUnicode_KIND()``
* ``PyUnicode_READ()``
* ``PyUnicode_READ_CHAR()``

PyDateTime "GET" macros
^^^^^^^^^^^^^^^^^^^^^^^

* ``PyDateTime_DATE_GET_FOLD()``
* ``PyDateTime_DATE_GET_HOUR()``
* ``PyDateTime_DATE_GET_MICROSECOND()``
* ``PyDateTime_DATE_GET_MINUTE()``
* ``PyDateTime_DATE_GET_SECOND()``
* ``PyDateTime_DATE_GET_TZINFO()``
* ``PyDateTime_DELTA_GET_DAYS()``
* ``PyDateTime_DELTA_GET_MICROSECONDS()``
* ``PyDateTime_DELTA_GET_SECONDS()``
* ``PyDateTime_GET_DAY()``
* ``PyDateTime_GET_MONTH()``
* ``PyDateTime_GET_YEAR()``
* ``PyDateTime_TIME_GET_FOLD()``
* ``PyDateTime_TIME_GET_HOUR()``
* ``PyDateTime_TIME_GET_MICROSECOND()``
* ``PyDateTime_TIME_GET_MINUTE()``
* ``PyDateTime_TIME_GET_SECOND()``
* ``PyDateTime_TIME_GET_TZINFO()``

PyDescr macros
^^^^^^^^^^^^^^

* ``PyDescr_NAME()``
* ``PyDescr_TYPE()``

Port C extensions to Python 3.11
--------------------------------

In practice, the majority of projects impacted by these PEP incompatible
changes should only have to make two changes:

* Replace ``Py_TYPE(obj) = new_type;``
with ``Py_SET_TYPE(obj, new_type);``.
* Replace ``Py_SIZE(obj) = new_size;``
with ``Py_SET_SIZE(obj, new_size);``.

The `pythoncapi_compat project
<https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to
update automatically C extensions: add Python 3.11 support without
losing support with older Python versions. The project provides a header
file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and
``Py_SET_SIZE()`` functions to Python 3.8 and older.

PyTuple_GET_ITEM() and PyList_GET_ITEM()
----------------------------------------

The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left
unchanged.

The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and
``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to
the inner ``PyObject**`` array.

Changing these macros would require to add a new API to get access to
the inner array which is out of the scope of this PEP.


Backwards Compatibility
=======================

The proposed C API changes are backward incompatible on purpose. In
practice, only a minority of third party projects are affected (16
projects are known to be broken) and `most of them have already been
updated for these changes
<https://bugs.python.org/issue39573#msg401378>`__ (12 on 16).

Most projects are broken by ``Py_TYPE()`` and ``Py_SIZE()`` changes.
These two macros have been converted to static inline macro in Python
3.10 alpha versions, but the change had to be reverted since it broke
too many projects. In the meanwhile, many projects, like Cython, have
been prepared for this change by using ``Py_SET_TYPE()`` and
``Py_SET_SIZE()``. For example, projects using Cython only have to
regenerate their outdated generated C code to become compatible.

For the "GET" functions like ``PyDict_GET_SIZE()``, no project in the PyPI
top 5000 projects use these functions as l-value.

The ``PyFloat_AS_DOUBLE()`` function is not used as a l-value in the
PyPI top 5000 projects.

The ``PyBytes_AS_STRING()`` and ``PyByteArray_AS_STRING()`` are used as
l-value but only to modify string characters, not to override the
``PyBytesObject.ob_sval`` or ``PyByteArrayObject.ob_start`` member.
For example, Cython uses the following code which remains valid::

PyByteArray_AS_STRING(string)[i] = (char) v;

This change does not follow the PEP 387 deprecation process. There is no
known way to emit a deprecation warning when a macro is used as a
l-value, but not when it's used differently (ex: r-value).


Rejected Idea: Leave the macros as they are
===========================================

The documentation of each function can discourage developers to use
macros to modify Python objects.

If these is a need to make an assignment, a setter function can be added
and the macro documentation can require to use the setter function. For
example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and
the ``Py_TYPE()`` documentation now requires to use the
``Py_SET_TYPE()`` function to set an object type.

If developers use macros as l-value, it's their responsibility when
their code breaks, not the Python responsibility. We are operating under
the consenting adults principle: we expect users of the Python C API to
use it as documented and expect them to take care of the fallout, if
things break when they don't.

This idea was rejected because only few developers read the
documentation, and only a minority is tracking changes of the Python C
API documentation. The majority of developers are only using CPython and
so are not aware of compatibility issues with other Python
implementations.

Moreover, continuing to allow using macros as l-value does not solve
issues of the nogil, PyPy and HPy projects.


Macros already modified
=======================

The following C API macros have already been modified to disallow using
them as l-value:

* ``PyCell_SET()``
* ``PyList_SET_ITEM()``
* ``PyTuple_SET_ITEM()``
* ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used
* ``_PyGCHead_SET_FINALIZED()``
* ``_PyGCHead_SET_NEXT()``
* ``asdl_seq_GET()``
* ``asdl_seq_GET_UNTYPED()``
* ``asdl_seq_LEN()``
* ``asdl_seq_SET()``
* ``asdl_seq_SET_UNTYPED()``

For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a
compiler error as expected.


References
==========

* `Python C API: Add functions to access PyObject
<https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October
2021) article by Victor Stinner
* `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value
<https://bugs.python.org/issue45476>`_
(October 2021)
* `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions
<https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/>`_
(September 2021)
* `[C API] Avoid accessing PyObject and PyVarObject members directly:
add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type
<https://bugs.python.org/issue39573>`__ (February 2020)
* `bpo-30459: PyList_SET_ITEM could be safer
<https://bugs.python.org/issue30459>`_ (May 2017)


Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KPIJPPJ6XVNOLGZQD2PFGMT7LBJMTTCO/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
guido at python
Nov 30, 2021, 10:34 AM
Post #2 of 27 (2513 views)
Permalink
How about *not* asking for an exception and just following the PEP 387
process? Is that really too burdensome?

On Tue, Nov 30, 2021 at 10:30 AM Victor Stinner <vstinner@python.org> wrote:

> Hi,
>
> I propose to disallow using macros as l-value. Read and comment the
> plain text below, or read the PEP 674 online,
> https://python.github.io/peps/pep-0674/
>
> While I'm not sure that the proposed changes are really controversial,
> I decided to write a formal PEP since the incompatible changes are not
> following the PEP 387 deprecation process and so a format PEP 387
> exception is better. Well, the list of modified macros is also quite
> long. Moreover, a PEP is a way to document and announce the changes
> ;-)
>
> The Py_TYPE() and Py_SIZE() changes are already approved the Steering
> Council, but I prefer to list them in the PEP:
> https://github.com/python/steering-council/issues/79
>
> In practice, I'm only aware of 4 projects impacted by these changes,
> and I wrote the pythoncapi_compat project which updates automatically
> C extensions: add Python 3.11 support, without losing support for
> older Python versions. I already prepared major projects like Cython
> and numpy for these changes (in total, 14 impacted projects have
> already been updated).
>
> Victor
>
> ---
>
> PEP: 674
> Title: Disallow using macros as l-value
> Author: Victor Stinner <vstinner@python.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 30-Oct-2021
> Python-Version: 3.11
>
> Abstract
> ========
>
> Incompatible C API change disallowing using macros as l-value to allow
> evolving CPython internals and to ease the C API implementation on other
> Python implementation.
>
> In practice, the majority of projects impacted by these incompatible
> changes should only have to make two changes:
>
> * Replace ``Py_TYPE(obj) = new_type;``
> with ``Py_SET_TYPE(obj, new_type);``.
> * Replace ``Py_SIZE(obj) = new_size;``
> with ``Py_SET_SIZE(obj, new_size);``.
>
>
> Rationale
> =========
>
> Using a macro as a l-value
> --------------------------
>
> In the Python C API, some functions are implemented as macro because
> writing a macro is simpler than writing a regular function. If a macro
> exposes directly a struture member, it is technically possible to use
> this macro to not only get the structure member but also set it.
>
> Example with the Python 3.10 ``Py_TYPE()`` macro::
>
> #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)
>
> This macro can be used as a **r-value** to **get** an object type::
>
> type = Py_TYPE(object);
>
> It can also be used as **l-value** to **set** an object type::
>
> Py_TYPE(object) = new_type;
>
> It is also possible to set an object reference count and an object size
> using ``Py_REFCNT()`` and ``Py_SIZE()`` macros.
>
> Setting directly an object attribute relies on the current exact CPython
> implementation. Implementing this feature in other Python
> implementations can make their C API implementation less efficient.
>
> CPython nogil fork
> ------------------
>
> Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch
> <https://github.com/colesbury/nogil/>`_. This fork has no
> ``PyObject.ob_refcnt`` member, but a more elaborated implementation for
> reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code
> fails with a compiler error.
>
> Merging the nogil fork into the upstream CPython main branch requires
> first to fix this C API compatibility issue. It is a concrete example of
> a Python optimization blocked indirectly by the C API.
>
> This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro
> has been already modified to disallow using it as a l-value.
>
> HPy project
> -----------
>
> The `HPy project <https://hpyproject.org/>`_ is a brand new C API for
> Python using only handles and function calls: handles are opaque,
> structure members cannot be accessed directly,and pointers cannot be
> dereferenced.
>
> Disallowing the usage of macros as l-value helps the migration of
> existing C extensions to HPy by reducing differences between the C API
> and the HPy API.
>
> PyPy cpyext module
> ------------------
>
> In PyPy, when a Python object is accessed by the Python C API, the PyPy
> ``cpyext`` module has to convert PyPy object to a CPython object. While
> PyPy objects are designed to be efficient with the PyPy JIT compiler,
> CPython objects are less efficient and increase the memory usage.
>
> This PEP alone is not enough to get rid of the CPython objects in the
> PyPy ``cpyext`` module, but it is a step towards this long term goal.
> PyPy already supports HPy which is a better solution in the long term.
>
>
> Specification
> =============
>
> Disallow using macros as l-value
> --------------------------------
>
> PyObject and PyVarObject macros
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead
> * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead
>
> "GET" macros
> ^^^^^^^^^^^^
>
> * ``PyByteArray_GET_SIZE()``
> * ``PyBytes_GET_SIZE()``
> * ``PyCFunction_GET_CLASS()``
> * ``PyCFunction_GET_FLAGS()``
> * ``PyCFunction_GET_FUNCTION()``
> * ``PyCFunction_GET_SELF()``
> * ``PyCell_GET()``
> * ``PyCode_GetNumFree()``
> * ``PyDict_GET_SIZE()``
> * ``PyFunction_GET_ANNOTATIONS()``
> * ``PyFunction_GET_CLOSURE()``
> * ``PyFunction_GET_CODE()``
> * ``PyFunction_GET_DEFAULTS()``
> * ``PyFunction_GET_GLOBALS()``
> * ``PyFunction_GET_KW_DEFAULTS()``
> * ``PyFunction_GET_MODULE()``
> * ``PyHeapType_GET_MEMBERS()``
> * ``PyInstanceMethod_GET_FUNCTION()``
> * ``PyList_GET_SIZE()``
> * ``PyMemoryView_GET_BASE()``
> * ``PyMemoryView_GET_BUFFER()``
> * ``PyMethod_GET_FUNCTION()``
> * ``PyMethod_GET_SELF()``
> * ``PySet_GET_SIZE()``
> * ``PyTuple_GET_SIZE()``
> * ``PyUnicode_GET_DATA_SIZE()``
> * ``PyUnicode_GET_LENGTH()``
> * ``PyUnicode_GET_LENGTH()``
> * ``PyUnicode_GET_SIZE()``
> * ``PyWeakref_GET_OBJECT()``
>
> "AS" macros
> ^^^^^^^^^^^
>
> * ``PyByteArray_AS_STRING()``
> * ``PyBytes_AS_STRING()``
> * ``PyFloat_AS_DOUBLE()``
> * ``PyUnicode_AS_DATA()``
> * ``PyUnicode_AS_UNICODE()``
>
> PyUnicode macros
> ^^^^^^^^^^^^^^^^
>
> * ``PyUnicode_1BYTE_DATA()``
> * ``PyUnicode_2BYTE_DATA()``
> * ``PyUnicode_4BYTE_DATA()``
> * ``PyUnicode_DATA()``
> * ``PyUnicode_IS_ASCII()``
> * ``PyUnicode_IS_COMPACT()``
> * ``PyUnicode_IS_READY()``
> * ``PyUnicode_KIND()``
> * ``PyUnicode_READ()``
> * ``PyUnicode_READ_CHAR()``
>
> PyDateTime "GET" macros
> ^^^^^^^^^^^^^^^^^^^^^^^
>
> * ``PyDateTime_DATE_GET_FOLD()``
> * ``PyDateTime_DATE_GET_HOUR()``
> * ``PyDateTime_DATE_GET_MICROSECOND()``
> * ``PyDateTime_DATE_GET_MINUTE()``
> * ``PyDateTime_DATE_GET_SECOND()``
> * ``PyDateTime_DATE_GET_TZINFO()``
> * ``PyDateTime_DELTA_GET_DAYS()``
> * ``PyDateTime_DELTA_GET_MICROSECONDS()``
> * ``PyDateTime_DELTA_GET_SECONDS()``
> * ``PyDateTime_GET_DAY()``
> * ``PyDateTime_GET_MONTH()``
> * ``PyDateTime_GET_YEAR()``
> * ``PyDateTime_TIME_GET_FOLD()``
> * ``PyDateTime_TIME_GET_HOUR()``
> * ``PyDateTime_TIME_GET_MICROSECOND()``
> * ``PyDateTime_TIME_GET_MINUTE()``
> * ``PyDateTime_TIME_GET_SECOND()``
> * ``PyDateTime_TIME_GET_TZINFO()``
>
> PyDescr macros
> ^^^^^^^^^^^^^^
>
> * ``PyDescr_NAME()``
> * ``PyDescr_TYPE()``
>
> Port C extensions to Python 3.11
> --------------------------------
>
> In practice, the majority of projects impacted by these PEP incompatible
> changes should only have to make two changes:
>
> * Replace ``Py_TYPE(obj) = new_type;``
> with ``Py_SET_TYPE(obj, new_type);``.
> * Replace ``Py_SIZE(obj) = new_size;``
> with ``Py_SET_SIZE(obj, new_size);``.
>
> The `pythoncapi_compat project
> <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to
> update automatically C extensions: add Python 3.11 support without
> losing support with older Python versions. The project provides a header
> file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and
> ``Py_SET_SIZE()`` functions to Python 3.8 and older.
>
> PyTuple_GET_ITEM() and PyList_GET_ITEM()
> ----------------------------------------
>
> The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left
> unchanged.
>
> The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and
> ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to
> the inner ``PyObject**`` array.
>
> Changing these macros would require to add a new API to get access to
> the inner array which is out of the scope of this PEP.
>
>
> Backwards Compatibility
> =======================
>
> The proposed C API changes are backward incompatible on purpose. In
> practice, only a minority of third party projects are affected (16
> projects are known to be broken) and `most of them have already been
> updated for these changes
> <https://bugs.python.org/issue39573#msg401378>`__ (12 on 16).
>
> Most projects are broken by ``Py_TYPE()`` and ``Py_SIZE()`` changes.
> These two macros have been converted to static inline macro in Python
> 3.10 alpha versions, but the change had to be reverted since it broke
> too many projects. In the meanwhile, many projects, like Cython, have
> been prepared for this change by using ``Py_SET_TYPE()`` and
> ``Py_SET_SIZE()``. For example, projects using Cython only have to
> regenerate their outdated generated C code to become compatible.
>
> For the "GET" functions like ``PyDict_GET_SIZE()``, no project in the PyPI
> top 5000 projects use these functions as l-value.
>
> The ``PyFloat_AS_DOUBLE()`` function is not used as a l-value in the
> PyPI top 5000 projects.
>
> The ``PyBytes_AS_STRING()`` and ``PyByteArray_AS_STRING()`` are used as
> l-value but only to modify string characters, not to override the
> ``PyBytesObject.ob_sval`` or ``PyByteArrayObject.ob_start`` member.
> For example, Cython uses the following code which remains valid::
>
> PyByteArray_AS_STRING(string)[i] = (char) v;
>
> This change does not follow the PEP 387 deprecation process. There is no
> known way to emit a deprecation warning when a macro is used as a
> l-value, but not when it's used differently (ex: r-value).
>
>
> Rejected Idea: Leave the macros as they are
> ===========================================
>
> The documentation of each function can discourage developers to use
> macros to modify Python objects.
>
> If these is a need to make an assignment, a setter function can be added
> and the macro documentation can require to use the setter function. For
> example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and
> the ``Py_TYPE()`` documentation now requires to use the
> ``Py_SET_TYPE()`` function to set an object type.
>
> If developers use macros as l-value, it's their responsibility when
> their code breaks, not the Python responsibility. We are operating under
> the consenting adults principle: we expect users of the Python C API to
> use it as documented and expect them to take care of the fallout, if
> things break when they don't.
>
> This idea was rejected because only few developers read the
> documentation, and only a minority is tracking changes of the Python C
> API documentation. The majority of developers are only using CPython and
> so are not aware of compatibility issues with other Python
> implementations.
>
> Moreover, continuing to allow using macros as l-value does not solve
> issues of the nogil, PyPy and HPy projects.
>
>
> Macros already modified
> =======================
>
> The following C API macros have already been modified to disallow using
> them as l-value:
>
> * ``PyCell_SET()``
> * ``PyList_SET_ITEM()``
> * ``PyTuple_SET_ITEM()``
> * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used
> * ``_PyGCHead_SET_FINALIZED()``
> * ``_PyGCHead_SET_NEXT()``
> * ``asdl_seq_GET()``
> * ``asdl_seq_GET_UNTYPED()``
> * ``asdl_seq_LEN()``
> * ``asdl_seq_SET()``
> * ``asdl_seq_SET_UNTYPED()``
>
> For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a
> compiler error as expected.
>
>
> References
> ==========
>
> * `Python C API: Add functions to access PyObject
> <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October
> 2021) article by Victor Stinner
> * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value
> <https://bugs.python.org/issue45476>`_
> (October 2021)
> * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions
> <
> https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/
> >`_
> (September 2021)
> * `[C API] Avoid accessing PyObject and PyVarObject members directly:
> add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type
> <https://bugs.python.org/issue39573>`__ (February 2020)
> * `bpo-30459: PyList_SET_ITEM could be safer
> <https://bugs.python.org/issue30459>`_ (May 2017)
>
>
> Copyright
> =========
>
> This document is placed in the public domain or under the
> CC0-1.0-Universal license, whichever is more permissive.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KPIJPPJ6XVNOLGZQD2PFGMT7LBJMTTCO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Nov 30, 2021, 10:52 AM
Post #3 of 27 (2513 views)
Permalink
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
> How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?

The Backward Compatibility section gives an explanation:

"This change does not follow the PEP 387 deprecation process. There is no
known way to emit a deprecation warning when a macro is used as a
l-value, but not when it's used differently (ex: r-value)."

Apart of compiler warnings, one way to implement the PEP 387
"deprecation process" would be to announce the change in two "What's
New in Python 3.X?" documents. But I expect that it will not be
efficient. Extract of the Rejected Idea section:

"(...) only few developers read the documentation, and only a minority
is tracking changes of the Python C API documentation."

In my experience, even if a DeprecationWarning is emitted at runtime,
developers miss or ignore it. See the recent "[Python-Dev] Do we need
to remove everything that's deprecated?" discussion and complains
about recent removal of deprecated features, like:

* collections.MutableMapping was deprecated for 7 Python versions
(deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
removed again in 3.11
* the "U" open() flag was deprecated for 10 Python versions
(deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
removed again in 3.11

For this specific PEP changes, I consider that the number of impacted
projects is low enough to skip a deprecation process: only 4 projects
are known to be impacted. One year ago (Python 3.10), 16 were
impacted, and 12 have already been updated in the meanwhile. I'm
talking especially about Py_TYPE() and Py_SIZE() changes which, again,
has been approved by the Steering Council.

Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AAQCCRSAKBAPCKWWT5BQTBPN7TL7WHVJ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Dec 1, 2021, 5:48 AM
Post #4 of 27 (2513 views)
Permalink
On Tue, Nov 30, 2021 at 7:52 PM Victor Stinner <vstinner@python.org> wrote:
> For this specific PEP changes, I consider that the number of impacted
> projects is low enough to skip a deprecation process: only 4 projects
> are known to be impacted. One year ago (Python 3.10), 16 were
> impacted, and 12 have already been updated in the meanwhile.

I ran a code search in PyPI top 5000 projects: my tool downloaded 4760
tarball and ZIP archives. On 4760 projects, only 16 of them (0.3%) are
affected by the PEP 674: 99.7% (4744) are not affected ;-) IMO it's
manageable to help these 16 projects to be updated for Python 3.11 (if
the PEP is accepted) before the final version (scheduled for
2022-10-03).

I expected that projects would only be affected by Py_TYPE() and
Py_SIZE() changes. But I also found exactly 2 projects affected by the
PyDescr_TYPE() and PyDescr_NAME() changes: code generated by SWIG in
M2Crypto and mecab-python3 projects. I will investigate how to update
SWIG for that.

More details in the issue: https://bugs.python.org/issue45476#msg407410

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YSBCFCJIQWWGDR5WQPDDNUEI4UWKTJ6R/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
encukou at gmail
Dec 7, 2021, 6:39 AM
Post #5 of 27 (2513 views)
Permalink
On 30. 11. 21 19:52, Victor Stinner wrote:
> On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
>> How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
>
> The Backward Compatibility section gives an explanation:
>
> "This change does not follow the PEP 387 deprecation process. There is no
> known way to emit a deprecation warning when a macro is used as a
> l-value, but not when it's used differently (ex: r-value)."
>
> Apart of compiler warnings, one way to implement the PEP 387
> "deprecation process" would be to announce the change in two "What's
> New in Python 3.X?" documents. But I expect that it will not be
> efficient. Extract of the Rejected Idea section:
>
> "(...) only few developers read the documentation, and only a minority
> is tracking changes of the Python C API documentation."
>
> In my experience, even if a DeprecationWarning is emitted at runtime,
> developers miss or ignore it. See the recent "[Python-Dev] Do we need
> to remove everything that's deprecated?" discussion and complains
> about recent removal of deprecated features, like:
>
> * collections.MutableMapping was deprecated for 7 Python versions
> (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> removed again in 3.11
> * the "U" open() flag was deprecated for 10 Python versions
> (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> removed again in 3.11
>
> For this specific PEP changes, I consider that the number of impacted
> projects is low enough to skip a deprecation process: only 4 projects
> are known to be impacted. One year ago (Python 3.10), 16 were
> impacted, and 12 have already been updated in the meanwhile. I'm
> talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> has been approved by the Steering Council.


The current version of the PEP looks nice, but I don't think the
rationale is strong enough.
I believe we should:
- Mark the l-value usage as deprecated in the docs,
- And then do nothing until we find an actual case where this issue
blocks development (or is actively dangerous for users).

Specifically, for each of the Rationale parts:


## Using a macro as a l-value

The practice was often discouraged (e.g. by GET in many of the names),
or even relatively widely and successfully used (e.g. Py_TYPE before
Python 3.9).

If we would deprecate using Py_REFCNT as l-value in the docs, but wait
with the conversion until it was *actually* needed, we would not lose
anything:
- Users would *still* have no period of visible compiler warnings or
DeprecationWarning.
- There would be more time for users to react to the documentation
warning. Or even come up with a linter, or try compiling their favorite
extensions with HPy/nogil and fixing the issues *on their own schedule*.


## CPython nogil fork

In CPython, we cannot change structs that are part of the stable ABI --
such as PyObject.ob_refcnt. IMO, getting rid of the macros that access
ob_refcnt is a relatively small part of this issue.

AFAICS, the technical change is trivial compared to nogil, and can be
easily made in the nogil fork -- if it is actually necessary in the end.


## HPy project

There is no reason for "Searching and replacing Py_SET_SIZE()".
If this change was not made, and Py_SIZE was semi-mechanically replaced
by HPy_Length(), then misuses of Py_SIZE (using is as l-value) can be
detected very easily: just compile against HPy.
And if the autoreplacer is smart enough to see when it should use
HPyTupleBuilder/HPyListBuilder, then it can surely see when Py_SIZE is
(mis)used as l-value!

There will always be some changes necessary when porting extensions to HPy.
CPython should definitely indicate what the best practice is, so that
HPY adopters have an easy time convincing projects to take their pull
requests -- but it should not break code for people who don't care about
HPy yet.


## GraalVM Python

This PEP is not enough to get rid of wrappers in GraalVM, yet it forces
users of CPython to adapt. Is it a part of a *plan* to remove wrappers,
or just a minor step in what looks like the general direction?
I do agree this PEP looks like a good step towards a long-term goal. But
even so, it should be made when it *actually benefits* existing users,
or allows some concrete good thing -- an optimization, a more
maintainable implementation, something that outweighs the need for churn
in code that worked up to now.



Overall, the Rationale seems hollow to me. It's breaking existing code
-- however bad that code is -- in the name of ideals rather than
concrete improvements.
Until disallowing macros as l-values allows concrete improvements in
CPython, it should be the job of linters.


FWIW, I do encourage alternative implementations to just not support
l-value macros. There are only few projects doing this, and the fix is
often (but not always!) easy. This should be a very small part of
porting something to a different Python implementation (but I could
definitely be wrong here, please correct me).

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RRX7LV4NDG4Z76NG2FPWI44AGMRQANIE/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
antoine at python
Dec 7, 2021, 7:36 AM
Post #6 of 27 (2513 views)
Permalink
On Tue, 7 Dec 2021 15:39:25 +0100
Petr Viktorin <encukou@gmail.com> wrote:

> On 30. 11. 21 19:52, Victor Stinner wrote:
> > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
> >> How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
> >
> > The Backward Compatibility section gives an explanation:
> >
> > "This change does not follow the PEP 387 deprecation process. There is no
> > known way to emit a deprecation warning when a macro is used as a
> > l-value, but not when it's used differently (ex: r-value)."
> >
> > Apart of compiler warnings, one way to implement the PEP 387
> > "deprecation process" would be to announce the change in two "What's
> > New in Python 3.X?" documents. But I expect that it will not be
> > efficient. Extract of the Rejected Idea section:
> >
> > "(...) only few developers read the documentation, and only a minority
> > is tracking changes of the Python C API documentation."
> >
> > In my experience, even if a DeprecationWarning is emitted at runtime,
> > developers miss or ignore it. See the recent "[Python-Dev] Do we need
> > to remove everything that's deprecated?" discussion and complains
> > about recent removal of deprecated features, like:
> >
> > * collections.MutableMapping was deprecated for 7 Python versions
> > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > removed again in 3.11
> > * the "U" open() flag was deprecated for 10 Python versions
> > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > removed again in 3.11
> >
> > For this specific PEP changes, I consider that the number of impacted
> > projects is low enough to skip a deprecation process: only 4 projects
> > are known to be impacted. One year ago (Python 3.10), 16 were
> > impacted, and 12 have already been updated in the meanwhile. I'm
> > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> > has been approved by the Steering Council.
>
>
> The current version of the PEP looks nice, but I don't think the
> rationale is strong enough.
> I believe we should:
> - Mark the l-value usage as deprecated in the docs,
> - And then do nothing until we find an actual case where this issue
> blocks development (or is actively dangerous for users).

Is there a way to emit a compilation warning when those macros are used
as l-values? Even if only enabled on some compilers.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
jsbueno at python
Dec 7, 2021, 8:54 AM
Post #7 of 27 (2513 views)
Permalink
Sorry for stepping in - but I am seeing too many arguments in favour
of the rules because "they are the rules", and just Victor arguing with
what is met in the "real world".

But if this update can be done by a simple search/replace on the C source
of projects,
I can only perceive two scenarios this will affect: well maintained
projects,
for which it is fixable in minutes, and stale packages, no longer
released
that "happen to work" when someone downloads and builds for new
Python versions. In these cases, the build will fail. If the person trying
the build can't fix it, but can take the error to a proper, or high
visibility,
forum, someone will be able to come to the fix, leading to renewed
visibility for the otherwise stale package.



On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou <antoine@python.org> wrote:

> On Tue, 7 Dec 2021 15:39:25 +0100
> Petr Viktorin <encukou@gmail.com> wrote:
>
> > On 30. 11. 21 19:52, Victor Stinner wrote:
> > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org>
> wrote:
> > >> How about *not* asking for an exception and just following the PEP
> 387 process? Is that really too burdensome?
> > >
> > > The Backward Compatibility section gives an explanation:
> > >
> > > "This change does not follow the PEP 387 deprecation process. There is
> no
> > > known way to emit a deprecation warning when a macro is used as a
> > > l-value, but not when it's used differently (ex: r-value)."
> > >
> > > Apart of compiler warnings, one way to implement the PEP 387
> > > "deprecation process" would be to announce the change in two "What's
> > > New in Python 3.X?" documents. But I expect that it will not be
> > > efficient. Extract of the Rejected Idea section:
> > >
> > > "(...) only few developers read the documentation, and only a minority
> > > is tracking changes of the Python C API documentation."
> > >
> > > In my experience, even if a DeprecationWarning is emitted at runtime,
> > > developers miss or ignore it. See the recent "[Python-Dev] Do we need
> > > to remove everything that's deprecated?" discussion and complains
> > > about recent removal of deprecated features, like:
> > >
> > > * collections.MutableMapping was deprecated for 7 Python versions
> > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > > * the "U" open() flag was deprecated for 10 Python versions
> > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > >
> > > For this specific PEP changes, I consider that the number of impacted
> > > projects is low enough to skip a deprecation process: only 4 projects
> > > are known to be impacted. One year ago (Python 3.10), 16 were
> > > impacted, and 12 have already been updated in the meanwhile. I'm
> > > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> > > has been approved by the Steering Council.
> >
> >
> > The current version of the PEP looks nice, but I don't think the
> > rationale is strong enough.
> > I believe we should:
> > - Mark the l-value usage as deprecated in the docs,
> > - And then do nothing until we find an actual case where this issue
> > blocks development (or is actively dangerous for users).
>
> Is there a way to emit a compilation warning when those macros are used
> as l-values? Even if only enabled on some compilers.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
sebastian at sipsolutions
Dec 7, 2021, 10:52 AM
Post #8 of 27 (2513 views)
Permalink
On Tue, 2021-12-07 at 13:54 -0300, Joao S. O. Bueno wrote:
> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing
> with
> what is met in the "real world".
>
> But if this update can be done by a simple search/replace on the C
> source
> of projects,
> I can only perceive two scenarios this will affect: well maintained
> projects,
>  for which it is fixable in minutes, and  stale packages, no longer
> released
> that "happen to work" when someone downloads and builds for new
> Python versions. In these cases, the build will fail. If the person
> trying
> the build can't fix it, but can take the error to a proper, or high
> visibility,
> forum, someone will be able to come to the fix, leading to renewed
> visibility for the otherwise stale package.
>

The problem are really less-maintained projects that may miss it or
take very long to react. And that may be very frustrating for their
users (who will not have a work-around beyond patching the project!).

So the question is whether these are so few users, that it is OK to
break them (even many bug fixes will break someone, after all).
The other consideration may be that documenting the change for a 1-2
years may achieve almost nothing except frustrating Victor ;).


One thing we once did in NumPy (for a runtime problem), was to
intentionally break everyone at pre-release/dev time to point out what
code needed fixing. Then flip the switch back at release time as to
not break production.
After a long enough time we enabled it for release mode.

Not saying that it was nice, but it was the only alternative would have
been to never fix it.

A similar switch could be worthwhile if it helps Victor with
experimenting on the dev-branch or reach a useful amount of projects.
Of course, I am not sure it would do either...

Cheers,

Sebastian



>
>
> On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou <antoine@python.org>
> wrote:
>
> > On Tue, 7 Dec 2021 15:39:25 +0100
> > Petr Viktorin <encukou@gmail.com> wrote:
> >
> > > On 30. 11. 21 19:52, Victor Stinner wrote:
> > > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum
> > > > <guido@python.org>
> > wrote:
> > > > > How about *not* asking for an exception and just following
> > > > > the PEP
> > 387 process? Is that really too burdensome?
> > > >
> > > > The Backward Compatibility section gives an explanation:
> > > >
> > > > "This change does not follow the PEP 387 deprecation process.
> > > > There is
> > no
> > > > known way to emit a deprecation warning when a macro is used as
> > > > a
> > > > l-value, but not when it's used differently (ex: r-value)."
> > > >
> > > > Apart of compiler warnings, one way to implement the PEP 387
> > > > "deprecation process" would be to announce the change in two
> > > > "What's
> > > > New in Python 3.X?" documents. But I expect that it will not be
> > > > efficient. Extract of the Rejected Idea section:
> > > >
> > > > "(...) only few developers read the documentation, and only a
> > > > minority
> > > > is tracking changes of the Python C API documentation."
> > > >
> > > > In my experience, even if a DeprecationWarning is emitted at
> > > > runtime,
> > > > developers miss or ignore it. See the recent "[Python-Dev] Do
> > > > we need
> > > > to remove everything that's deprecated?" discussion and
> > > > complains
> > > > about recent removal of deprecated features, like:
> > > >
> > > > * collections.MutableMapping was deprecated for 7 Python
> > > > versions
> > > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9
> > > > beta,
> > > > removed again in 3.11
> > > > * the "U" open() flag was deprecated for 10 Python versions
> > > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9
> > > > beta,
> > > > removed again in 3.11
> > > >
> > > > For this specific PEP changes, I consider that the number of
> > > > impacted
> > > > projects is low enough to skip a deprecation process: only 4
> > > > projects
> > > > are known to be impacted. One year ago (Python 3.10), 16 were
> > > > impacted, and 12 have already been updated in the meanwhile.
> > > > I'm
> > > > talking especially about Py_TYPE() and Py_SIZE() changes which,
> > > > again,
> > > > has been approved by the Steering Council.
> > >
> > >
> > > The current version of the PEP looks nice, but I don't think the
> > > rationale is strong enough.
> > > I believe we should:
> > > - Mark the l-value usage as deprecated in the docs,
> > > - And then do nothing until we find an actual case where this
> > > issue
> > > blocks development (or is actively dangerous for users).
> >
> > Is there a way to emit a compilation warning when those macros are
> > used
> > as l-values? Even if only enabled on some compilers.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-leave@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/WNTZZYLT3VK6REJ3BEBVGNDGZCIXWFCA/
> Code of Conduct: http://python.org/psf/codeofconduct/

Attached Files:

Re: PEP 674: Disallow using macros as l-value [ In reply to ]
guido at python
Dec 7, 2021, 11:54 AM
Post #9 of 27 (2513 views)
Permalink
On Tue, Dec 7, 2021 at 9:00 AM Joao S. O. Bueno <jsbueno@python.org.br>
wrote:

> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing with
> what is met in the "real world".
>
> But if this update can be done by a simple search/replace on the C source
> of projects,
> I can only perceive two scenarios this will affect: well maintained
> projects,
> for which it is fixable in minutes, and stale packages, no longer
> released
> that "happen to work" when someone downloads and builds for new
> Python versions. In these cases, the build will fail. If the person trying
> the build can't fix it, but can take the error to a proper, or high
> visibility,
> forum, someone will be able to come to the fix, leading to renewed
> visibility for the otherwise stale package.
>

It sounds like you (like probably many others) don't quite understand why
"the rules" exist.

Please trust me that what you perceive as simple updates, is not so simple
when you take the whole ecosystem of dependencies into account. You may
recall the long transition from Python 2 to 3. This was so painful in part
because the core dev team (led by myself) was similarly optimistic about
"oh, people can easily fix the few small things that will crop up". In
practice, things like this take a very long time to fix everywhere, as
people are waiting for their dependencies to solve the issue first before
they can even start testing, and so on. (For example, many libraries still
don't have wheels for Python 3.10, even though it was released over two
months ago and 3.10.1 is already out.)

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
steve.dower at python
Dec 7, 2021, 1:04 PM
Post #10 of 27 (2513 views)
Permalink
On 12/7/2021 6:52 PM, Sebastian Berg wrote:
> One thing we once did in NumPy (for a runtime problem), was to
> intentionally break everyone at pre-release/dev time to point out what
> code needed fixing. Then flip the switch back at release time as to
> not break production.
> After a long enough time we enabled it for release mode.
>
> Not saying that it was nice, but it was the only alternative would have
> been to never fix it.

I like this idea. We'd have to turn it back for RC, and ensure that it's
possible to have working code both before/after the change. We may be
getting enough usage during beta for it to be worthwhile, though we
still have the problem of knock-on effects (where e.g. until NumPy
works, nothing that depends on it can even begin testing).

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GQ7KF656QIHPKLT5Z23P3W5OYOGH4J5K/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
guido at python
Dec 7, 2021, 1:14 PM
Post #11 of 27 (2513 views)
Permalink
On Tue, Dec 7, 2021 at 1:09 PM Steve Dower <steve.dower@python.org> wrote:

> On 12/7/2021 6:52 PM, Sebastian Berg wrote:
> > One thing we once did in NumPy (for a runtime problem), was to
> > intentionally break everyone at pre-release/dev time to point out what
> > code needed fixing. Then flip the switch back at release time as to
> > not break production.
> > After a long enough time we enabled it for release mode.
> >
> > Not saying that it was nice, but it was the only alternative would have
> > been to never fix it.
>
> I like this idea. We'd have to turn it back for RC, and ensure that it's
> possible to have working code both before/after the change. We may be
> getting enough usage during beta for it to be worthwhile, though we
> still have the problem of knock-on effects (where e.g. until NumPy
> works, nothing that depends on it can even begin testing).
>

Yeah, this sounds like a good approach *for things where the alternative is
never to fix it*.

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Dec 7, 2021, 3:47 PM
Post #12 of 27 (2513 views)
Permalink
On Tue, Dec 7, 2021 at 3:43 PM Petr Viktorin <encukou@gmail.com> wrote:
> If we would deprecate using Py_REFCNT as l-value in the docs, but wait
> with the conversion until it was *actually* needed, we would not lose
> anything:
> (...)
> ## CPython nogil fork
>
> In CPython, we cannot change structs that are part of the stable ABI --
> such as PyObject.ob_refcnt. IMO, getting rid of the macros that access
> ob_refcnt is a relatively small part of this issue.

The question is if the nogil optimization is relevant for CPython?
It's a trade-off between backward compatibility and performance. It
seems like some core devs are enthusiast by the idea of getting this
work into CPython: count it in ;-) Getting rid of the GIL is an old
dream which became reality: the abiltiy of using all CPU cores at the
same time!

But not all technical issues have to be solved at once. First, the
*API* can be made compatible with nogil: the Python 3.10 Py_RECNT()
change made this function compatible with nogil. Later, the question
of the stable *ABI* can be studied.


> ## GraalVM Python
>
> This PEP is not enough to get rid of wrappers in GraalVM, yet it forces
> users of CPython to adapt.

Currently, it's considered as second class citizen because of the C
API issues. The idea is to put GraalVM Python at the same level as
CPython.

I would like to put all Python implementation in a fair competition.
The competition is great for innovating and should benefit to
everybody!

The PEP 674 is part of this goal. You're right that this PEP alone is
not enough to remove all wrappers: it doesn't solve all problems.

On the other side, a large PEP like PEP 620 cannot be accepted...
because it requires too many changes at once, it's not technically
possible to implement all changes in a single Python. It must be done
incrementally over multiple Python versions. That's why I'm trying to
split the PEP 620 into smaller PEPs (now: PEP 670, PEP 674), write a
better rationale for each incremental change, and better study the
backward compatibility issues of each incompatible change.


> Until disallowing macros as l-values allows concrete improvements in
> CPython, it should be the job of linters.

Do you know existing linters for C extensions? I don't know any :-(


> FWIW, I do encourage alternative implementations to just not support
> l-value macros. There are only few projects doing this, and the fix is
> often (but not always!) easy. This should be a very small part of
> porting something to a different Python implementation (but I could
> definitely be wrong here, please correct me).

First, PyPy tried to only implement a subset of the C API and promote
cffi for incompatible C extensions. This approach failed.

So PyPy decided to be as compatible as possible with CPython for the C
API. Now PyPy basically supports the whole C API and doesn't want to
drop support for the C extensions "abusing" the C API, like using
macros as l-value.

The problem is that supporting all C API "abuses" requires to
basically copy everything from CPython and it's inefficient. I didn't
mention PyPy in the PEP 674 since this PEP alone doesn't help PyPy to
avoid converting efficient PyPy objects to inefficient CPython objects
when they are accessed by the C API (PyPy cpyext module).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UFPN4VUV3Y4ZV4TYJAEAUQUPKOB7SLUU/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Dec 7, 2021, 3:51 PM
Post #13 of 27 (2513 views)
Permalink
Extract of the Backward Compatibility section of the PEP:

"This change does not follow the PEP 387 deprecation process. There is
no known way to emit a deprecation warning only when a macro is used
as a l-value, but not when it's used differently (ex: as a r-value)."

Victor

On Tue, Dec 7, 2021 at 4:41 PM Antoine Pitrou <antoine@python.org> wrote:
>
> On Tue, 7 Dec 2021 15:39:25 +0100
> Petr Viktorin <encukou@gmail.com> wrote:
>
> > On 30. 11. 21 19:52, Victor Stinner wrote:
> > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
> > >> How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
> > >
> > > The Backward Compatibility section gives an explanation:
> > >
> > > "This change does not follow the PEP 387 deprecation process. There is no
> > > known way to emit a deprecation warning when a macro is used as a
> > > l-value, but not when it's used differently (ex: r-value)."
> > >
> > > Apart of compiler warnings, one way to implement the PEP 387
> > > "deprecation process" would be to announce the change in two "What's
> > > New in Python 3.X?" documents. But I expect that it will not be
> > > efficient. Extract of the Rejected Idea section:
> > >
> > > "(...) only few developers read the documentation, and only a minority
> > > is tracking changes of the Python C API documentation."
> > >
> > > In my experience, even if a DeprecationWarning is emitted at runtime,
> > > developers miss or ignore it. See the recent "[Python-Dev] Do we need
> > > to remove everything that's deprecated?" discussion and complains
> > > about recent removal of deprecated features, like:
> > >
> > > * collections.MutableMapping was deprecated for 7 Python versions
> > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > > * the "U" open() flag was deprecated for 10 Python versions
> > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
> > > removed again in 3.11
> > >
> > > For this specific PEP changes, I consider that the number of impacted
> > > projects is low enough to skip a deprecation process: only 4 projects
> > > are known to be impacted. One year ago (Python 3.10), 16 were
> > > impacted, and 12 have already been updated in the meanwhile. I'm
> > > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
> > > has been approved by the Steering Council.
> >
> >
> > The current version of the PEP looks nice, but I don't think the
> > rationale is strong enough.
> > I believe we should:
> > - Mark the l-value usage as deprecated in the docs,
> > - And then do nothing until we find an actual case where this issue
> > blocks development (or is actively dangerous for users).
>
> Is there a way to emit a compilation warning when those macros are used
> as l-values? Even if only enabled on some compilers.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YA3RN7Q2NHTRDMWI7Z6VWUZIAJ4JCD2D/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Dec 7, 2021, 3:58 PM
Post #14 of 27 (2513 views)
Permalink
An idea for unmaintained incompatible C extensions is to update the C
code when a C extension is installed, as part of the build process.
For example, replace "Py_TYPE(obj) = new_type" with "Py_SET_TYPE(obj,
new_type)".

Something similar to the old "2to3" option of setuptools. The
upgrade_pythoncapi.py of the pythoncapi_compat project is already a
partial implementation of idea: it can update the C code, but it's not
integrated in tools like setuptools.

It's just an idea, a full implementation should be designed and written.

Victor

On Tue, Dec 7, 2021 at 5:56 PM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
>
> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing with
> what is met in the "real world".
>
> But if this update can be done by a simple search/replace on the C source of projects,
> I can only perceive two scenarios this will affect: well maintained projects,
> for which it is fixable in minutes, and stale packages, no longer released
> that "happen to work" when someone downloads and builds for new
> Python versions. In these cases, the build will fail. If the person trying
> the build can't fix it, but can take the error to a proper, or high visibility,
> forum, someone will be able to come to the fix, leading to renewed
> visibility for the otherwise stale package.
>
>
>
> On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou <antoine@python.org> wrote:
>>
>> On Tue, 7 Dec 2021 15:39:25 +0100
>> Petr Viktorin <encukou@gmail.com> wrote:
>>
>> > On 30. 11. 21 19:52, Victor Stinner wrote:
>> > > On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
>> > >> How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
>> > >
>> > > The Backward Compatibility section gives an explanation:
>> > >
>> > > "This change does not follow the PEP 387 deprecation process. There is no
>> > > known way to emit a deprecation warning when a macro is used as a
>> > > l-value, but not when it's used differently (ex: r-value)."
>> > >
>> > > Apart of compiler warnings, one way to implement the PEP 387
>> > > "deprecation process" would be to announce the change in two "What's
>> > > New in Python 3.X?" documents. But I expect that it will not be
>> > > efficient. Extract of the Rejected Idea section:
>> > >
>> > > "(...) only few developers read the documentation, and only a minority
>> > > is tracking changes of the Python C API documentation."
>> > >
>> > > In my experience, even if a DeprecationWarning is emitted at runtime,
>> > > developers miss or ignore it. See the recent "[Python-Dev] Do we need
>> > > to remove everything that's deprecated?" discussion and complains
>> > > about recent removal of deprecated features, like:
>> > >
>> > > * collections.MutableMapping was deprecated for 7 Python versions
>> > > (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta,
>> > > removed again in 3.11
>> > > * the "U" open() flag was deprecated for 10 Python versions
>> > > (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta,
>> > > removed again in 3.11
>> > >
>> > > For this specific PEP changes, I consider that the number of impacted
>> > > projects is low enough to skip a deprecation process: only 4 projects
>> > > are known to be impacted. One year ago (Python 3.10), 16 were
>> > > impacted, and 12 have already been updated in the meanwhile. I'm
>> > > talking especially about Py_TYPE() and Py_SIZE() changes which, again,
>> > > has been approved by the Steering Council.
>> >
>> >
>> > The current version of the PEP looks nice, but I don't think the
>> > rationale is strong enough.
>> > I believe we should:
>> > - Mark the l-value usage as deprecated in the docs,
>> > - And then do nothing until we find an actual case where this issue
>> > blocks development (or is actively dangerous for users).
>>
>> Is there a way to emit a compilation warning when those macros are used
>> as l-values? Even if only enabled on some compilers.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVYIS6PJR3WTKLYDFY3XJ3UAKW3/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WNTZZYLT3VK6REJ3BEBVGNDGZCIXWFCA/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FEZN6IZO34IPYY52G5UDVIODT6YB5WI6/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
guido at python
Dec 7, 2021, 4:08 PM
Post #15 of 27 (2513 views)
Permalink
Hi Victor,

I wonder if there's a role for HPy in this context? What if instead of
evolving the stable ABI and the limited API, instead we were to focus on
first-class support for HPy? Surely 5 release cycles would be enough to
completely remove the stable ABI and perhaps even the limited API in favor
of HPy? Or am I misunderstanding the place of HPy in the ecosystem?

--Guido

PS. Eric wrote an analyzer for C code and checked it in under
Tools/c-analyzer. This currently focuses on finding globals, but perhaps it
forms a good starting point for a linter for C extensions?

On Tue, Dec 7, 2021 at 3:53 PM Victor Stinner <vstinner@python.org> wrote:

> On Tue, Dec 7, 2021 at 3:43 PM Petr Viktorin <encukou@gmail.com> wrote:
> > If we would deprecate using Py_REFCNT as l-value in the docs, but wait
> > with the conversion until it was *actually* needed, we would not lose
> > anything:
> > (...)
> > ## CPython nogil fork
> >
> > In CPython, we cannot change structs that are part of the stable ABI --
> > such as PyObject.ob_refcnt. IMO, getting rid of the macros that access
> > ob_refcnt is a relatively small part of this issue.
>
> The question is if the nogil optimization is relevant for CPython?
> It's a trade-off between backward compatibility and performance. It
> seems like some core devs are enthusiast by the idea of getting this
> work into CPython: count it in ;-) Getting rid of the GIL is an old
> dream which became reality: the abiltiy of using all CPU cores at the
> same time!
>
> But not all technical issues have to be solved at once. First, the
> *API* can be made compatible with nogil: the Python 3.10 Py_RECNT()
> change made this function compatible with nogil. Later, the question
> of the stable *ABI* can be studied.
>
>
> > ## GraalVM Python
> >
> > This PEP is not enough to get rid of wrappers in GraalVM, yet it forces
> > users of CPython to adapt.
>
> Currently, it's considered as second class citizen because of the C
> API issues. The idea is to put GraalVM Python at the same level as
> CPython.
>
> I would like to put all Python implementation in a fair competition.
> The competition is great for innovating and should benefit to
> everybody!
>
> The PEP 674 is part of this goal. You're right that this PEP alone is
> not enough to remove all wrappers: it doesn't solve all problems.
>
> On the other side, a large PEP like PEP 620 cannot be accepted...
> because it requires too many changes at once, it's not technically
> possible to implement all changes in a single Python. It must be done
> incrementally over multiple Python versions. That's why I'm trying to
> split the PEP 620 into smaller PEPs (now: PEP 670, PEP 674), write a
> better rationale for each incremental change, and better study the
> backward compatibility issues of each incompatible change.
>
>
> > Until disallowing macros as l-values allows concrete improvements in
> > CPython, it should be the job of linters.
>
> Do you know existing linters for C extensions? I don't know any :-(
>
>
> > FWIW, I do encourage alternative implementations to just not support
> > l-value macros. There are only few projects doing this, and the fix is
> > often (but not always!) easy. This should be a very small part of
> > porting something to a different Python implementation (but I could
> > definitely be wrong here, please correct me).
>
> First, PyPy tried to only implement a subset of the C API and promote
> cffi for incompatible C extensions. This approach failed.
>
> So PyPy decided to be as compatible as possible with CPython for the C
> API. Now PyPy basically supports the whole C API and doesn't want to
> drop support for the C extensions "abusing" the C API, like using
> macros as l-value.
>
> The problem is that supporting all C API "abuses" requires to
> basically copy everything from CPython and it's inefficient. I didn't
> mention PyPy in the PEP 674 since this PEP alone doesn't help PyPy to
> avoid converting efficient PyPy objects to inefficient CPython objects
> when they are accessed by the C API (PyPy cpyext module).
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UFPN4VUV3Y4ZV4TYJAEAUQUPKOB7SLUU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
greg.ewing at canterbury
Dec 7, 2021, 4:38 PM
Post #16 of 27 (2513 views)
Permalink
On 8/12/21 4:36 am, Antoine Pitrou wrote:
> Is there a way to emit a compilation warning when those macros are used
> as l-values? Even if only enabled on some compilers.

Maybe the macro could be written so that it expands to something
with a cast around it? I think gcc has an option for warning about
casts used as l-values.

--
Greg
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DG6GBX6OFCK4D2G6CGQFW5QX5JXQPAZO/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Dec 7, 2021, 4:47 PM
Post #17 of 27 (2513 views)
Permalink
For me, HPy is the only valid stable API and stable ABI in the long
term which is efficient on any Python implementation. Its design is
very different than the C API: HPy avoids all C API design mistakes,
it doesn't leak any implementation detail.

HPy can already be used today on CPython, even if it's not directly
provided by CPython.

Providing HPy as a first-class citizen in CPython, as already done in
PyPy, would be great to promote HPy! However, HPy evolves quickly and
so needs to be released more frequently than CPython. At least, we
could promote it more in the C API documentation, as we already
promote Cython.

Promoting the HPy usage doesn't solve any issue listed in PEP 620, 670
and 674 since CPython still has to continue supporting the C API. We
will only be fully free to make any change in Python internals without
having to care about breaking the C API once the *LAST* C extensions
using the C API will disappear... Look at Python 2.7 which is still
used in 2021. I bet that C extensions using the C API are not doing to
disappear soon.

For me, the question is:

internals because of the public C API?

The sub-question is:

until it will be possible again to evolve the Python internals?

Obviously, my answer is that we must change the C API as soon as
possible to allow again changing Python internals and help other
Python implementations to support the C API.

--

One option for CPython would be to have a native HPy support, and
emulate the legacy C API with something similar to what PyPy does with
its cpyext module. That may make C extensions using the C API slower
and may increase their memory usage.

Victor


On Wed, Dec 8, 2021 at 1:08 AM Guido van Rossum <guido@python.org> wrote:
>
> Hi Victor,
>
> I wonder if there's a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy? Surely 5 release cycles would be enough to completely remove the stable ABI and perhaps even the limited API in favor of HPy? Or am I misunderstanding the place of HPy in the ecosystem?
>
> --Guido
>
> PS. Eric wrote an analyzer for C code and checked it in under Tools/c-analyzer. This currently focuses on finding globals, but perhaps it forms a good starting point for a linter for C extensions?

--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GH7N6PKCM4775TRAPMCKJT4IRNG64KLP/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
vstinner at python
Dec 7, 2021, 5:07 PM
Post #18 of 27 (2513 views)
Permalink
Are you talking about gcc -Wignored-qualifiers? It seems like such
warning is only emitted where the function is *defined*, not where the
function is *called*. Example:
---
const int f(void) { return 1; }
int main() { return f(); }
---

Output:
---
$ gcc -Wextra y.c -o y
y.c:1:1: warning: type qualifiers ignored on function return type
[-Wignored-qualifiers]
1 | const int f(void) { return 1; }
| ^~~~~
---

I'm only able to get a compile *error* when a macro is used as a
l-value. Example:
---
struct Point { int x; int y; };
#define POINT_X(p) ((int)p.x)
int main()
{
struct Point p = {1, 2};
int x = POINT_X(p); // r-value ok
POINT_X(p) = 1; // l-value ERROR
return 0;
}
---

With "#define POINT_X(p) (p.x)", the macro can be used as l-value and r-value.

Victor

On Wed, Dec 8, 2021 at 1:43 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
>
> On 8/12/21 4:36 am, Antoine Pitrou wrote:
> > Is there a way to emit a compilation warning when those macros are used
> > as l-values? Even if only enabled on some compilers.
>
> Maybe the macro could be written so that it expands to something
> with a cast around it? I think gcc has an option for warning about
> casts used as l-values.
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DG6GBX6OFCK4D2G6CGQFW5QX5JXQPAZO/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SHQJR3S4WI2OWS6M3J3NFPRYZEHJV7ZC/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
encukou at gmail
Dec 8, 2021, 2:01 AM
Post #19 of 27 (2513 views)
Permalink
On 08. 12. 21 1:47, Victor Stinner wrote:
> For me, HPy is the only valid stable API and stable ABI in the long
> term which is efficient on any Python implementation. Its design is
> very different than the C API: HPy avoids all C API design mistakes,
> it doesn't leak any implementation detail.
>
> HPy can already be used today on CPython, even if it's not directly
> provided by CPython.
>
> Providing HPy as a first-class citizen in CPython, as already done in
> PyPy, would be great to promote HPy! However, HPy evolves quickly and
> so needs to be released more frequently than CPython. At least, we
> could promote it more in the C API documentation, as we already
> promote Cython.
>
> Promoting the HPy usage doesn't solve any issue listed in PEP 620, 670
> and 674 since CPython still has to continue supporting the C API. We
> will only be fully free to make any change in Python internals without
> having to care about breaking the C API once the *LAST* C extensions
> using the C API will disappear... Look at Python 2.7 which is still
> used in 2021. I bet that C extensions using the C API are not doing to
> disappear soon.
>
> For me, the question is:
>
> => Is it ok to no longer be able to make any change in Python
> internals because of the public C API?
>
> The sub-question is:
>
> => Is it ok to have a slow deprecation process and wait 5 to 10 years
> until it will be possible again to evolve the Python internals?

No, they should be evolved when they *need* to be evolved.
This PEP calls for breaking things because they *might* need evolving in
the future, but doesn't presemt any immediate benefit. In that case, I
think it's better document that we don't like some usage, but only do
the removals when they're helpful.

Especially in cases where there can't be a proper deprecation period.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WIEC7X4CTPL7PN7EIOJJJWCJJ5ZBYH7M/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
encukou at gmail
Dec 8, 2021, 2:12 AM
Post #20 of 27 (2513 views)
Permalink
On 07. 12. 21 17:54, Joao S. O. Bueno wrote:
> Sorry for stepping in - but I am seeing too many arguments in favour
> of the rules because "they are the rules", and just Victor arguing with
> what is met in the "real world".

OTOH, coming up with rules and then blatantly ignoring them is silly at
best.

If the rules are bad, they should definitely be changed. And if a case
is exceptional enough, we should make an exception -- but to make an
exception we need a very good understanding of why the rules are the way
they are (and in this case. I don't think any single person has the
proper understanding).


One of the roles the backwards compatibility policy serves is a promise
to our users. They can expect to not run into problems if they only
upgrade to every second Python version and fix deprecation warnings
(except for "extreme situations such as dangerously broken or insecure
features or features no one could reasonably be depending on").
That is, IMO, a pretty good reason to consider sticking to the rules.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NXOKMGCN3TOU7CA2YX3ZNRZMCPNJ2VUF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
matti.picus at gmail
Dec 8, 2021, 3:02 AM
Post #21 of 27 (2513 views)
Permalink
Victor Stinner wrote:
> First, PyPy tried to only implement a subset of the C API and promote
> cffi for incompatible C extensions. This approach failed.

Just to be clear: I don't think any of the PyPy devs assumed the huge Python c-extension ecosystem would suddenly run to adopt CFFI, so calling our effort to emulate the C-API a consequence of any failure is a bit of an exaggeration.

> So PyPy decided to be as compatible as possible with CPython for the C
> API. Now PyPy basically supports the whole C API and doesn't want to
> drop support for the C extensions "abusing" the C API, like using
> macros as l-value.

I think it would be more accurate to say that PyPy, as a small project in the huge Python ecosystem, is trying to support any C-API used widely by the community. If there is a common PyPy voice (after all, we are also a project with many opinions), we don't "not want to drop support" nor "want to drop support" for any commonly used C-API interfaces, rather we want to stay out of this argument and promote alternatives such as CFFI, cppyy, and HPy instead.

> Victor

Matti
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3HGX42QFURHCU6O6DOKBXLVTFIU6RDBO/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
tim.felgentreff at oracle
Dec 8, 2021, 6:56 AM
Post #22 of 27 (2513 views)
Permalink
Hi,

On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
> I wonder if there's a role for HPy in this context? What if instead of
> evolving the stable ABI and the limited API, instead we were to focus on
> first-class support for HPy?

The hope with the HPy project is indeed to provide a C API that is close to the
original API to make porting easy and have it perform as close to the existing
API as possible. At the same time, HPy is sufficently removed to be a good "C
extension API" (as opposed to a stable subset of the CPython implementation API)
that does not leak implementation details. To ensure this latter property is why
we try to develop everything in parallel for CPython, PyPy, and GraalVM Python.

On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
> Providing HPy as a first-class citizen in CPython, as already done in
> PyPy, would be great to promote HPy! However, HPy evolves quickly and
> so needs to be released more frequently than CPython. At least, we
> could promote it more in the C API documentation, as we already
> promote Cython.

Indeed, at this point HPy is still evolving very fast. We are still solving
issues while migrating NumPy, have begun adding support for HPy to Cython,
and will start working on pybind11 soon. I believe by the time we have these
users of the existing C API working, HPy should be in a state where it is
generally useful and can be deemed stable enough that further development can
follow a more stable process.

> For me, HPy is the only valid stable API and stable ABI in the long
> term which is efficient on any Python implementation. Its design is
> very different than the C API: HPy avoids all C API design mistakes,
> it doesn't leak any implementation detail.

In the long run the HPy project would like to become a promoted API to
write Python C extensions.


Tim Felgentreff
(on behalf of the HPy team)
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
tim.felgentreff at oracle
Dec 8, 2021, 8:33 AM
Post #23 of 27 (2513 views)
Permalink
Hi,

On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
> I wonder if there's a role for HPy in this context? What if instead of
> evolving the stable ABI and the limited API, instead we were to focus on
> first-class support for HPy?

The hope with the HPy project is indeed to provide a C API that is close to the
original API to make porting easy and have it perform as close to the existing
API as possible. At the same time, HPy is sufficently removed to be a good "C
extension API" (as opposed to a stable subset of the CPython implementation API)
that does not leak implementation details. To ensure this latter property is why
we try to develop everything in parallel for CPython, PyPy, and GraalVM Python.

On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
> Providing HPy as a first-class citizen in CPython, as already done in
> PyPy, would be great to promote HPy! However, HPy evolves quickly and
> so needs to be released more frequently than CPython. At least, we
> could promote it more in the C API documentation, as we already
> promote Cython.

Indeed, at this point HPy is still evolving very fast. We are still solving
issues while migrating NumPy, have begun adding support for HPy to Cython,
and will start working on pybind11 soon. I believe by the time we have these
users of the existing C API working, HPy should be in a state where it is
generally useful and can be deemed stable enough that further development can
follow a more stable process.

> For me, HPy is the only valid stable API and stable ABI in the long
> term which is efficient on any Python implementation. Its design is
> very different than the C API: HPy avoids all C API design mistakes,
> it doesn't leak any implementation detail.

In the long run the HPy project would like to become a promoted API to
write Python C extensions.

Tim Felgentreff
(on behalf of the HPy team)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2NFJC7UAVIQZ6BNV4NBTY3THI7OIJIMF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
matti.picus at gmail
Dec 8, 2021, 8:52 AM
Post #24 of 27 (2513 views)
Permalink
On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum guido@python.org wrote:

I wonder if there’s a role for HPy in this context? What if instead of
evolving the stable ABI and the limited API, instead we were to focus on
first-class support for HPy?

The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good “C extension API” (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why we try to develop everything in parallel for CPython, PyPy, and GraalVM Python.

On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner vstinner@python.org wrote:

Providing HPy as a first-class citizen in CPython, as already done in
PyPy, would be great to promote HPy! However, HPy evolves quickly and
so needs to be released more frequently than CPython. At least, we
could promote it more in the C API documentation, as we already
promote Cython.

Indeed, at this point HPy is still evolving very fast. We are still solving
issues while migrating NumPy, have begun adding support for HPy to Cython,
and will start working on pybind11 soon. I believe by the time we have these
users of the existing C API working, HPy should be in a state where it is
generally useful and can be deemed stable enough that further development can follow a more stable process.

For me, HPy is the only valid stable API and stable ABI in the long
term which is efficient on any Python implementation. Its design is
very different than the C API: HPy avoids all C API design mistakes,
it doesn’t leak any implementation detail.

In the long run the HPy project would like to become a promoted API to
write Python C extensions.

Tim Felgentreff, posted by Matti Picus
(on behalf of the HPy team)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GBDS2W575VED2KQN6DIN7E5JCMNKNQJF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: PEP 674: Disallow using macros as l-value [ In reply to ]
tim.felgentreff at oracle
Dec 8, 2021, 12:56 PM
Post #25 of 27 (2513 views)
Permalink
I am sorry for the multi-post, it seems the first message I sent and which I thought got lost somewhere along the way was simply routed via pidgeon. I promise to wait for the message to arrive more patiently next time.

Tim

________________________________
From: Tim Felgentreff <tim.felgentreff@oracle.com>
Sent: Wednesday, 8 December 2021, 18:21
To: python-dev@python.org
Subject: [External] : [Python-Dev] Re: PEP 674: Disallow using macros as l-value

Hi,

On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
> I wonder if there's a role for HPy in this context? What if instead of
> evolving the stable ABI and the limited API, instead we were to focus on
> first-class support for HPy?

The hope with the HPy project is indeed to provide a C API that is close to the
original API to make porting easy and have it perform as close to the existing
API as possible. At the same time, HPy is sufficently removed to be a good "C
extension API" (as opposed to a stable subset of the CPython implementation API)
that does not leak implementation details. To ensure this latter property is why
we try to develop everything in parallel for CPython, PyPy, and GraalVM Python.

On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
> Providing HPy as a first-class citizen in CPython, as already done in
> PyPy, would be great to promote HPy! However, HPy evolves quickly and
> so needs to be released more frequently than CPython. At least, we
> could promote it more in the C API documentation, as we already
> promote Cython.

Indeed, at this point HPy is still evolving very fast. We are still solving
issues while migrating NumPy, have begun adding support for HPy to Cython,
and will start working on pybind11 soon. I believe by the time we have these
users of the existing C API working, HPy should be in a state where it is
generally useful and can be deemed stable enough that further development can
follow a more stable process.

> For me, HPy is the only valid stable API and stable ABI in the long
> term which is efficient on any Python implementation. Its design is
> very different than the C API: HPy avoids all C API design mistakes,
> it doesn't leak any implementation detail.

In the long run the HPy project would like to become a promoted API to
write Python C extensions.

Tim Felgentreff
(on behalf of the HPy team)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/python-dev.python.org/__;!!ACWV5N9M2RV99hQ!aoUoshCDK0NCYcZl7KiFcC2WO7noi5LMeO6Fh_cwqn-y1PS8k_aqbBsSKvPRxq9nWh0S$
Message archived at https://urldefense.com/v3/__https://mail.python.org/archives/list/python-dev@python.org/message/2NFJC7UAVIQZ6BNV4NBTY3THI7OIJIMF/__;!!ACWV5N9M2RV99hQ!aoUoshCDK0NCYcZl7KiFcC2WO7noi5LMeO6Fh_cwqn-y1PS8k_aqbBsSKvPRxqoIdm36$
Code of Conduct: https://urldefense.com/v3/__http://python.org/psf/codeofconduct/__;!!ACWV5N9M2RV99hQ!aoUoshCDK0NCYcZl7KiFcC2WO7noi5LMeO6Fh_cwqn-y1PS8k_aqbBsSKvPRxhko9lh5$
1 2  View All

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal