Mailing List Archive

> So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11?
> Extension modules can use '#' format with ssize_t, without
> PY_SSIZE_T_CLEAN defined.
> Or should we wait one more version?

Hi, Inada,
I suggest we should wait until at least Python 3.12 or Python 4.0.

There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT().
Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MWMH6FWHJUUKI7OKBQR7OJJ6MZV6J35P/
Code of Conduct: http://python.org/psf/codeofconduct/

07.06.21 06:05, Inada Naoki ????:
> Since Python 3.8, PyArg_Parse*() APIs and Py_BuildValue() APIs emitted
> DeprecationWarning when
> '#' format is used without PY_SSIZE_T_CLEAN defined.
> In Python 3.10, they raise a RuntimeError, not a warning. Extension
> modules can not use '#' format with int.
>
> So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11?
> Extension modules can use '#' format with ssize_t, without
> PY_SSIZE_T_CLEAN defined.
>
> Or should we wait one more version?

Many users still use 3.6 or 3.7. Jumping from 3.7 to 3.11 could break
extensions in bad way (crash, truncated data, leaked sensitive
information, execution of arbitrary code). Also, deprecation warnings in
3.8 and 3.9 can be easily ignored.

I propose to wait until both of conditions became true:

* 3.7 no longer maintained
* 3.10 reaches security-only mode.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UXJAG63XCVRUC4ENDF7V2Q6FLLBFWM2H/
Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Jun 7, 2021 at 4:52 PM Serhiy Storchaka <storchaka@gmail.com> wrote:
>
> Many users still use 3.6 or 3.7. Jumping from 3.7 to 3.11 could break
> extensions in bad way (crash, truncated data, leaked sensitive
> information, execution of arbitrary code). Also, deprecation warnings in
> 3.8 and 3.9 can be easily ignored.
>
> I propose to wait until both of conditions became true:
>
> * 3.7 no longer maintained
> * 3.10 reaches security-only mode.
>

Makes sense.

Python 3.7 will get security fix until 2023-06.
https://www.python.org/dev/peps/pep-0537/#and-beyond-schedule

Python 3.12 will be released at 2023-10.
So we can change PY_SSIZE_T_CLEAN by default from 3.12.

Regards,

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QZSSHINMXXZ4I3ODPDFPGFC7MBSKIOVB/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jun 8, 2021 at 12:53 AM Hai Shi <shihai1992@gmail.com> wrote:
>
> > So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11?
> > Extension modules can use '#' format with ssize_t, without
> > PY_SSIZE_T_CLEAN defined.
> > Or should we wait one more version?
>
> Hi, Inada,
> I suggest we should wait until at least Python 3.12 or Python 4.0.
>

Serhiy and you suggest Python 3.12 and I agree with it.
Thank you for your reply.

> There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT().
> Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?

They are part of stable ABIs. So we can remove/merge them at Python 4.0.

Regards,
--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KNA7KVN6ZIBXWASUAJQRGT6OPCBDULFW/
Code of Conduct: http://python.org/psf/codeofconduct/

07.06.21 08:41, Hai Shi ????:
> There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT().
> Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?

We should support binary compatibility to some degree, so there should
be several steps:

* Make macro PyArg_Parse an alias of _PyArg_Parse_SizeT. Keep function
PyArg_Parse.
* Make function PyArg_Parse always raising an exception.
* Remove function PyArg_Parse.
* [Optionally] Now we can re-add function PyArg_Parse as an alias of
_PyArg_Parse_SizeT and remove macro PyArg_Parse.
* [Optionally in 4.0 or 5.0] Remove _PyArg_Parse_SizeT.

But we can squish several last steps in 4.0 which do not need to support
binary compatibility with 3.x.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/J2D5VBSOQTZOK42AUM5AHPPBA45VNJK4/
Code of Conduct: http://python.org/psf/codeofconduct/

On 08. 06. 21 10:05, Serhiy Storchaka wrote:
> 07.06.21 08:41, Hai Shi ????:
>> There have an another question. There have many C API defined under PY_SSIZE_T_CLEAN, for example: _PyArg_Parse_SizeT().
>> Could we remove or merge them after making PY_SSIZE_T_CLEAN not mandatory?
>
> We should support binary compatibility to some degree, so there should
> be several steps:
>
> * Make macro PyArg_Parse an alias of _PyArg_Parse_SizeT. Keep function
> PyArg_Parse.

One more thing about the stable ABI: in the future, I'd like to make it
more useful in languages other than C. This usually means avoiding macros.
Would it make sense to expose _PyArg_Parse_SizeT as a public function,
like PyArgT_Parse?
(The macro redirecting PyArg_Parse to this function could of course
stay, to help C users.)

> * Make function PyArg_Parse always raising an exception.

This would break extensions that use the stable ABI.
(Yes, even starting to raise RuntimeError in 3.10 broke things. And yes,
it's not strictly an ABI issue, but it has the same effect for users:
they still need to recompile extensions to keep them working.)

> * Remove function PyArg_Parse.
> * [Optionally] Now we can re-add function PyArg_Parse as an alias of
> _PyArg_Parse_SizeT and remove macro PyArg_Parse.
> * [Optionally in 4.0 or 5.0] Remove _PyArg_Parse_SizeT.
>
> But we can squish several last steps in 4.0 which do not need to support
> binary compatibility with 3.x.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZLS2AUQDGT3NHSBM63XEPN3TEUGRAP4Z/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jun 8, 2021 at 6:02 PM Petr Viktorin <encukou@gmail.com> wrote:
>
>
> > * Make function PyArg_Parse always raising an exception.
>
> This would break extensions that use the stable ABI.
> (Yes, even starting to raise RuntimeError in 3.10 broke things. And yes,
> it's not strictly an ABI issue, but it has the same effect for users:
> they still need to recompile extensions to keep them working.)
>

I think we can skip this step.
Extension modules using # format is already broken since Python 3.10.

Adding # format support with size_t won't break so much.
We can do it in Python 3.12 or 3.13.

Regards,

--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DN6OAOEP6MF75VJC6O6ATRQREPXE6CSU/
Code of Conduct: http://python.org/psf/codeofconduct/

> On 7 Jun 2021, at 05:05, Inada Naoki <songofacandy@gmail.com> wrote:
>
> Hi, folks,
>
> Since Python 3.8, PyArg_Parse*() APIs and Py_BuildValue() APIs emitted
> DeprecationWarning when
> '#' format is used without PY_SSIZE_T_CLEAN defined.
> In Python 3.10, they raise a RuntimeError, not a warning. Extension
> modules can not use '#' format with int.
>
> So how about making PY_SSIZE_T_CLEAN not mandatory in Python 3.11?
> Extension modules can use '#' format with ssize_t, without
> PY_SSIZE_T_CLEAN defined.
>
> Or should we wait one more version?

Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot.

I guess this depends a little on what promises the stable ABI makes, the functions are still there but behave differently than before.

P.S. I’d be in favour of just dropping PY_SSIZE_T_CLEAN completely (that is use Py_ssize_t unconditionally) to simplify the code base, apart from being slightly worried about the impact on the stable ABI. AFAIK The define was meant as a temporary transition mechanism when Py_ssize_t was introduced in the, by now, ancient past.

Ronald

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev
<python-dev@python.org> wrote:
> Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot.

Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API*
change. At the ABI level, what changed if that C extensions built
(with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an
error on Python 3.10 (for a few specific argument formats using "#").
Ah you are right, it's an incompatible ABI change.

It might be possible to keep the backward compatibility at the ABI
level by adding a 3rd flavor of "parse" functions:

* parse with size_t: no change
* parse without size_t: stable ABI
* parse without size_t which raises an exception on "#" formats: new
Python 3.10 functions

It's already painful to have 2 flavors of each functions. Adding a 3rd
flavor would make the maintenance burden even worse, whereas Inada-san
wants to opposite (remove the 2nd flavor to only have one!).

A more general question is: do we still want to keep backward
compatibility with Python 3.2 released 10 years ago, or is it ok to
start with a new stable ABI which drops backward compatibility with
Python 3.5 (for example)?

It's maybe time to replace "abi3" with "abi4" in Python 3.10?

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ENFYNPCPTEZQSBVB5BA5XYNVFUDUA4GF/
Code of Conduct: http://python.org/psf/codeofconduct/

> On 9 Jun 2021, at 11:13, Victor Stinner <vstinner@python.org> wrote:
>
> On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev
> <python-dev@python.org> wrote:
>> Its a bit late to complain (and I’m not affected by this myself), but those functions are part of the stable ABI. The change in 3.10 will break any extensions that use the stable ABI, use these functions and don’t use PY_SSIZE_T_CLEAN. I have no idea how many of those exist, especially given that the stable ABI doesn’t seem to be used a lot.
>
> Require PY_SSIZE_T_CLEAN macro to be defined in an incompatible *API*
> change. At the ABI level, what changed if that C extensions built
> (with Python 3.9 and older) without PY_SSIZE_T_CLEAN now raise an
> error on Python 3.10 (for a few specific argument formats using "#").
> Ah you are right, it's an incompatible ABI change.

:-(

>
>
> It might be possible to keep the backward compatibility at the ABI
> level by adding a 3rd flavor of "parse" functions:
>
> * parse with size_t: no change
> * parse without size_t: stable ABI
> * parse without size_t which raises an exception on "#" formats: new
> Python 3.10 functions
>
> It's already painful to have 2 flavors of each functions. Adding a 3rd
> flavor would make the maintenance burden even worse, whereas Inada-san
> wants to opposite (remove the 2nd flavor to only have one!).

I don’t think it is necessary to introduce a 3th variant, for 3.11+ we could do something like this:

* [3.11] Add deprecation warnings in the C headers to the few functions with a “PY_SSIZE_T_CLEAN” variant when “PY_SSIZE_T_CLEAN” is not defined
* [3.12+] Change the headers to behave as if “PY_SSIZE_T_CLEAN” is defined, and only keep the non-PY_SSIZE_T_CLEAN variants for the stable ABI (which would include dropping non-PY_SSIZE_T_CLEAN variants for private functions). The PY_SSIZE_T_CLEAN variants would keep their "_PysizeT” suffix in the (stable or unstable) ABI.

This wouldn’t allow dropping the non-PY_SSIZE_T_CLEAN variants entirely, at least not until we’re fine with breaking the stable ABI. Another disadvantage is that this might require changes in code that doesn’t even use “#” in format strings in 3.11.

>
> A more general question is: do we still want to keep backward
> compatibility with Python 3.2 released 10 years ago, or is it ok to
> start with a new stable ABI which drops backward compatibility with
> Python 3.5 (for example)?
>
> It's maybe time to replace "abi3" with "abi4" in Python 3.10?

Personally I’d say it is too soon for that, especially when the CPython speedup project (Guido, Mark, et.al.) is just started and HPy is far from finished. Either project might teach us what changes are needed for a long term stable ABI.

Ronald
—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

I think stable ABI keeps symbols, signatures, and memory layouts.
I don't think stable ABI keeps all behaviors.

For example, Py_CompileString() is stable ABI.
When we add `async` keyword, Py_CompileString() starts raising an
Error for source code using `async` name.
Is it ABI change? I don't think so.

I want to drop Py_UNICODE support in Python 3.12. It is another
incompatible change in PyArg_Parse*() *API*.
Users can not use "u" format after it. It is an incompatible *API*
change, but not an *ABI* change.

I suspect we had made many incompatible *API* changes in stable ABIs already.

If I am wrong, can we stop keeping stable ABI at Python 3.12?
Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.

Regards,
--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T7CPD4LHAVU5TMMCZ7CXNMOUL3D7ZR5O/
Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, 9 Jun 2021 at 11:36, Inada Naoki <songofacandy@gmail.com> wrote:
> If I am wrong, can we stop keeping stable ABI at Python 3.12?
> Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.

Conversely, the stable ABI allows projects to build cross-version
binary wheels. Not many projects do that yet, but it's definitely
something we'd like to see more of. Needing new binary builds every
version blocks users from testing new versions of Python in advance of
the release. Also, I often use the stable ABI when embedding, so that
I can replace the Python interpreter without needing to recompile my
application and redeploy new binaries everywhere (my use case is
pretty niche, though, so I wouldn't like to claim I represent a
typical user...).

But I do agree that we should either start keeping to the commitments
that we made around the stability of the stable ABI, or we should
abandon it properly and declare it no longer supported. Having
something that sort of works except when we accidentally broke it
doesn't help anyone.

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HXOMSUACA5GOH2SFU526R54AJSZGRNXW/
Code of Conduct: http://python.org/psf/codeofconduct/

On 09. 06. 21 13:09, Paul Moore wrote:
> On Wed, 9 Jun 2021 at 11:36, Inada Naoki <songofacandy@gmail.com> wrote:
>> If I am wrong, can we stop keeping stable ABI at Python 3.12?
>> Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.
>
> Conversely, the stable ABI allows projects to build cross-version
> binary wheels. Not many projects do that yet, but it's definitely
> something we'd like to see more of. Needing new binary builds every
> version blocks users from testing new versions of Python in advance of
> the release. [...]
>
> But I do agree that we should either start keeping to the commitments
> that we made around the stability of the stable ABI, or we should
> abandon it properly and declare it no longer supported. Having
> something that sort of works except when we accidentally broke it
> doesn't help anyone.

I don't think I made actual commitments regarding the API. The docs do
say: "we recommend testing an extension with all minor Python versions
it supports".
Also, when the API breaks, you get a Python exception; if the ABI does,
you get segfaults.

So breaking the API is much less severe, but still -- please think about
the effect on users who just want their compiled extensions to keep working.


On 09. 06. 21 13:09, Paul Moore wrote:
> Also, I often use the stable ABI when embedding, so that
> I can replace the Python interpreter without needing to recompile my
> application and redeploy new binaries everywhere (my use case is
> pretty niche, though, so I wouldn't like to claim I represent a
> typical user...).

I hope this use case becomes non-niche. I would love it if embedders
tell people to just use any Python they have lying around, instead of
vendoring it (or more realistically, embedding JS or Lua instead).
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3VYTTABB2UB6HVQHASYONSYQDBHDL3OU/
Code of Conduct: http://python.org/psf/codeofconduct/

> On 9 Jun 2021, at 12:28, Inada Naoki <songofacandy@gmail.com> wrote:
>
> I think stable ABI keeps symbols, signatures, and memory layouts.
> I don't think stable ABI keeps all behaviors.

As often “it depends”. Behaviour is IMHO part of the API/ABI contract. That said, that does not necessarily mean that we cannot change behaviour at all, but that the cost to users for such changes should be taken into account.

>
> For example, Py_CompileString() is stable ABI.
> When we add `async` keyword, Py_CompileString() starts raising an
> Error for source code using `async` name.
> Is it ABI change? I don't think so.

I agree. But its not as easy as “it is not an ABI change because it only changes functionality of a function”. The interface contract of Py_CompileString is that it compiles Python code. If the rules for what valid Python code is change (such as the introduction of ‘async’ as a hard keyword) the function can start to reject input that was accepted earlier. That’s IMHO different from a change to Py_CompileString that starts raising an error unconditionally because we no longer want to expose the API.

>
> I want to drop Py_UNICODE support in Python 3.12. It is another
> incompatible change in PyArg_Parse*() *API*.
> Users can not use "u" format after it. It is an incompatible *API*
> change, but not an *ABI* change.

It is an ABI change: an extensions targetting the stable ABI no longer works due to a change in implementation. That doesn’t necessarily mean the change cannot be made, especially when a deprecation warning is emitted before the feature is removed.

>
> I suspect we had made many incompatible *API* changes in stable ABIs already.
>
> If I am wrong, can we stop keeping stable ABI at Python 3.12?
> Python 4.0 won't come in foreseeable future. Stable ABI blocks Python evolution.

The annoying part of the stable ABI is that it still exposes some implementation details and behaviour that make it harder to write correct code (such as borrowed references, these can be very convenient but are also easy to misuse). That’s one reason why HPy is an interesting project, even when only targeting CPython.

And to be clear: I’m not opposed to the change for the “#” format character and the removal of the “u” format you mention earlier.

Ronald
>
> Regards,
> --
> Inada Naoki <songofacandy@gmail.com>

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

On 6/9/2021 2:20 PM, Petr Viktorin wrote:
> On 09. 06. 21 13:09, Paul Moore wrote:
>> Also, I often use the stable ABI when embedding, so that
>> I can replace the Python interpreter without needing to recompile my
>> application and redeploy new binaries everywhere (my use case is
>> pretty niche, though, so I wouldn't like to claim I represent a
>> typical user...).
>
> I hope this use case becomes non-niche. I would love it if embedders
> tell people to just use any Python they have lying around, instead of
> vendoring it (or more realistically, embedding JS or Lua instead).

I also hope it becomes non-niche, but I'd rather you started
embedding/vendoring CPython rather than using anything that just happens
to be laying around.

The number one issue that *all* of my customers (and their customers)
have is installation. For most of them, the best way to solve it is to
not make them install Python themselves, which in many cases means
vendoring. The more acceptable and easy we can make this process, the
more Python will be a viable choice against JS or Lua (though with all
the other C API, threading and initialization issues, it's unlikely that
embedding CPython is going to become significantly more attractive than
those two - even IronPython still lives on for embedding because it
works so well).

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/36MZXQC3WGDXO6SFTPGT7RC34EMFPP6E/
Code of Conduct: http://python.org/psf/codeofconduct/

I don’t see how the stable ABI works as a substitute for vendoring Python.
A lot of other things can still vary even when the C API remains the same!
(E.g. syntax, and stdlib behavior.)

On Thu, Jun 17, 2021 at 11:49 Steve Dower <steve.dower@python.org> wrote:

> On 6/9/2021 2:20 PM, Petr Viktorin wrote:
> > On 09. 06. 21 13:09, Paul Moore wrote:
> >> Also, I often use the stable ABI when embedding, so that
> >> I can replace the Python interpreter without needing to recompile my
> >> application and redeploy new binaries everywhere (my use case is
> >> pretty niche, though, so I wouldn't like to claim I represent a
> >> typical user...).
> >
> > I hope this use case becomes non-niche. I would love it if embedders
> > tell people to just use any Python they have lying around, instead of
> > vendoring it (or more realistically, embedding JS or Lua instead).
>
> I also hope it becomes non-niche, but I'd rather you started
> embedding/vendoring CPython rather than using anything that just happens
> to be laying around.
>
> The number one issue that *all* of my customers (and their customers)
> have is installation. For most of them, the best way to solve it is to
> not make them install Python themselves, which in many cases means
> vendoring. The more acceptable and easy we can make this process, the
> more Python will be a viable choice against JS or Lua (though with all
> the other C API, threading and initialization issues, it's unlikely that
> embedding CPython is going to become significantly more attractive than
> those two - even IronPython still lives on for embedding because it
> works so well).
>
> Cheers,
> Steve
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/36MZXQC3WGDXO6SFTPGT7RC34EMFPP6E/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)

On Fri, 18 Jun 2021 at 01:57, Guido van Rossum <guido@python.org> wrote:
>
> I don’t see how the stable ABI works as a substitute for vendoring Python. A lot of other things can still vary even when the C API remains the same! (E.g. syntax, and stdlib behavior.)

IMO it doesn't. However for certain applications (the sort of thing I
was referring to) - where the user is writing their own scripts and
the embedding API is used merely to expose an interface to the Python
language, dynamically linking to whatever version of Python the user
has installed can be precisely the right thing to do - the user gets
access to the version of the language they expect, the installed
packages they expect to see, etc.

Consider something like the vim integration with Python (which
*doesn't* use the stable ABI, so you have to use a distribution built
for precisely the Python version you have installed - the stable ABI
would be better from the POV of supporting fewer binary builds).
Vendoring is a viable alternative here as well, but as I say, it just
has different trade-offs.

As I said, it's a niche usage, but it has its place. I don't
personally have an opinion on promoting the option of linking to the
system Python - it just suits my use cases.

Paul
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5LOUHPH6VHNA5KFU4ZE6QC4GKCXSSAHX/
Code of Conduct: http://python.org/psf/codeofconduct/

Le 18/06/2021 à 08:50, Paul Moore a écrit :
>
> IMO it doesn't. However for certain applications (the sort of thing I
> was referring to) - where the user is writing their own scripts and
> the embedding API is used merely to expose an interface to the Python
> language, dynamically linking to whatever version of Python the user
> has installed can be precisely the right thing to do - the user gets
> access to the version of the language they expect, the installed
> packages they expect to see, etc.

As a user, I second this. When trying to drive applications from the
outside (as opposed to extending them through plugins), it is annoying
when two applications won't work together because each one insists on
using its own vendored python.

Of course, there are often real blockers, such as incompatible event
loops. But not always…

Cheers,
Baptiste
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/
Code of Conduct: http://python.org/psf/codeofconduct/

Can you elaborate on that use case? Which two applications are you thinking
of, and what was your goal in driving them? This sounds interesting but I
haven’t encountered this myself.

On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello <
devel2021@baptiste-carvello.net> wrote:

> Le 18/06/2021 à 08:50, Paul Moore a écrit :
> >
> > IMO it doesn't. However for certain applications (the sort of thing I
> > was referring to) - where the user is writing their own scripts and
> > the embedding API is used merely to expose an interface to the Python
> > language, dynamically linking to whatever version of Python the user
> > has installed can be precisely the right thing to do - the user gets
> > access to the version of the language they expect, the installed
> > packages they expect to see, etc.
>
> As a user, I second this. When trying to drive applications from the
> outside (as opposed to extending them through plugins), it is annoying
> when two applications won't work together because each one insists on
> using its own vendored python.
>
> Of course, there are often real blockers, such as incompatible event
> loops. But not always…
>
> Cheers,
> Baptiste
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)

Hi,

Le 18/06/2021 à 21:00, Guido van Rossum a écrit :
> Can you elaborate on that use case? Which two applications are you
> thinking of, and what was your goal in driving them? This sounds
> interesting but I haven’t encountered this myself.

Well, I'm not sure the case I was thinking of is still relevant to
anything: that was plotting 3D crystal models using crystallography
library CCTBX [1] and visualization application Mayavi [2], some 15-20
years ago. BTW, I misremembered a bit: only CCTBX insisted on using a
vendored python ("libtbx.python"), Mayavi used the system python.
Anyway, it was more pain to make Mayavi use libtbx.python, than to make
CCTBX work with the system python.

Also, I must admit that even applications embedding the system python
can have some limitations. For example, GIMP and GDB can execute python
scripts, but their API can't be "imported" from the outside. Which means
no arguments passed to the script over the command line ("sys.argv"), no
venvs, no REPL. But at least you can install additional packages (pip /
distro package manager) and limitations can be more or less hacked
around. For a sophisticated example, the debugger extension Voltron [3]
provides REPL access to GDB objects over a client-server connexion.

Cheers,
Baptiste

[1] https://cci.lbl.gov/docs/cctbx/
[2] https://docs.enthought.com/mayavi/mayavi/
[3] https://github.com/snare/voltron

> On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello
> <devel2021@baptiste-carvello.net
> <mailto:devel2021@baptiste-carvello.net>> wrote:
>
> Le 18/06/2021 à 08:50, Paul Moore a écrit :
> >
> > IMO it doesn't. However for certain applications (the sort of thing I
> > was referring to) - where the user is writing their own scripts and
> > the embedding API is used merely to expose an interface to the Python
> > language, dynamically linking to whatever version of Python the user
> > has installed can be precisely the right thing to do - the user gets
> > access to the version of the language they expect, the installed
> > packages they expect to see, etc.
>
> As a user, I second this. When trying to drive applications from the
> outside (as opposed to extending them through plugins), it is annoying
> when two applications won't work together because each one insists on
> using its own vendored python.
>
> Of course, there are often real blockers, such as incompatible event
> loops. But not always…
>
> Cheers,
> Baptiste
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
> --Guido (mobile)
>

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UWSV3VDCJOWCXUZIBPDWFJHRLU/
Code of Conduct: http://python.org/psf/codeofconduct/

Okay, I think your evidence can then be discounted. Really, any app that
relies on the publicly installed Python runs a serious risk of breaking
when that Python gets updated, regardless of whether the ABI changes or not.

On Mon, Jun 21, 2021 at 2:46 AM Baptiste Carvello <
devel2021@baptiste-carvello.net> wrote:

> Hi,
>
> Le 18/06/2021 à 21:00, Guido van Rossum a écrit :
> > Can you elaborate on that use case? Which two applications are you
> > thinking of, and what was your goal in driving them? This sounds
> > interesting but I haven’t encountered this myself.
>
> Well, I'm not sure the case I was thinking of is still relevant to
> anything: that was plotting 3D crystal models using crystallography
> library CCTBX [1] and visualization application Mayavi [2], some 15-20
> years ago. BTW, I misremembered a bit: only CCTBX insisted on using a
> vendored python ("libtbx.python"), Mayavi used the system python.
> Anyway, it was more pain to make Mayavi use libtbx.python, than to make
> CCTBX work with the system python.
>
> Also, I must admit that even applications embedding the system python
> can have some limitations. For example, GIMP and GDB can execute python
> scripts, but their API can't be "imported" from the outside. Which means
> no arguments passed to the script over the command line ("sys.argv"), no
> venvs, no REPL. But at least you can install additional packages (pip /
> distro package manager) and limitations can be more or less hacked
> around. For a sophisticated example, the debugger extension Voltron [3]
> provides REPL access to GDB objects over a client-server connexion.
>
> Cheers,
> Baptiste
>
> [1] https://cci.lbl.gov/docs/cctbx/
> [2] https://docs.enthought.com/mayavi/mayavi/
> [3] https://github.com/snare/voltron
>
> > On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello
> > <devel2021@baptiste-carvello.net
> > <mailto:devel2021@baptiste-carvello.net>> wrote:
> >
> > Le 18/06/2021 à 08:50, Paul Moore a écrit :
> > >
> > > IMO it doesn't. However for certain applications (the sort of
> thing I
> > > was referring to) - where the user is writing their own scripts and
> > > the embedding API is used merely to expose an interface to the
> Python
> > > language, dynamically linking to whatever version of Python the
> user
> > > has installed can be precisely the right thing to do - the user
> gets
> > > access to the version of the language they expect, the installed
> > > packages they expect to see, etc.
> >
> > As a user, I second this. When trying to drive applications from the
> > outside (as opposed to extending them through plugins), it is
> annoying
> > when two applications won't work together because each one insists on
> > using its own vendored python.
> >
> > Of course, there are often real blockers, such as incompatible event
> > loops. But not always…
> >
> > Cheers,
> > Baptiste
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > <mailto:python-dev@python.org>
> > To unsubscribe send an email to python-dev-leave@python.org
> > <mailto:python-dev-leave@python.org>
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
> > --
> > --Guido (mobile)
> >
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UWSV3VDCJOWCXUZIBPDWFJHRLU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 21. 06. 21 20:20, Guido van Rossum wrote:
> Okay, I think your evidence can then be discounted. Really, any app that
> relies on the publicly installed Python runs a serious risk of breaking
> when that Python gets updated, regardless of whether the ABI changes or not.

Unfortunately, this includes scripts for any extensible software
(Mayavi, GIMP, etc) -- even if Python is bundled with that software,
upgrading the software risks breaking the scripts.


> On Mon, Jun 21, 2021 at 2:46 AM Baptiste Carvello
> <devel2021@baptiste-carvello.net
> <mailto:devel2021@baptiste-carvello.net>> wrote:
>
> Hi,
>
> Le 18/06/2021 à 21:00, Guido van Rossum a écrit :
> > Can you elaborate on that use case? Which two applications are you
> > thinking of, and what was your goal in driving them? This sounds
> > interesting but I haven’t encountered this myself.
>
> Well, I'm not sure the case I was thinking of is still relevant to
> anything: that was plotting 3D crystal models using crystallography
> library CCTBX [1] and visualization application Mayavi [2], some 15-20
> years ago. BTW, I misremembered a bit: only CCTBX insisted on using a
> vendored python ("libtbx.python"), Mayavi used the system python.
> Anyway, it was more pain to make Mayavi use libtbx.python, than to make
> CCTBX work with the system python.
>
> Also, I must admit that even applications embedding the system python
> can have some limitations. For example, GIMP and GDB can execute python
> scripts, but their API can't be "imported" from the outside. Which means
> no arguments passed to the script over the command line ("sys.argv"), no
> venvs, no REPL. But at least you can install additional packages (pip /
> distro package manager) and limitations can be more or less hacked
> around. For a sophisticated example, the debugger extension Voltron [3]
> provides REPL access to GDB objects over a client-server connexion.
>
> Cheers,
> Baptiste
>
> [1] https://cci.lbl.gov/docs/cctbx/ <https://cci.lbl.gov/docs/cctbx/>
> [2] https://docs.enthought.com/mayavi/mayavi/
> <https://docs.enthought.com/mayavi/mayavi/>
> [3] https://github.com/snare/voltron <https://github.com/snare/voltron>
>
> > On Fri, Jun 18, 2021 at 09:44 Baptiste Carvello
> > <devel2021@baptiste-carvello.net
> <mailto:devel2021@baptiste-carvello.net>
> > <mailto:devel2021@baptiste-carvello.net
> <mailto:devel2021@baptiste-carvello.net>>> wrote:
> >
> >     Le 18/06/2021 à 08:50, Paul Moore a écrit :
> >     >
> >     > IMO it doesn't. However for certain applications (the sort
> of thing I
> >     > was referring to) - where the user is writing their own
> scripts and
> >     > the embedding API is used merely to expose an interface to
> the Python
> >     > language, dynamically linking to whatever version of Python
> the user
> >     > has installed can be precisely the right thing to do - the
> user gets
> >     > access to the version of the language they expect, the
> installed
> >     > packages they expect to see, etc.
> >
> >     As a user, I second this. When trying to drive applications
> from the
> >     outside (as opposed to extending them through plugins), it is
> annoying
> >     when two applications won't work together because each one
> insists on
> >     using its own vendored python.
> >
> >     Of course, there are often real blockers, such as
> incompatible event
> >     loops. But not always…
> >
> >     Cheers,
> >     Baptiste
> >     _______________________________________________
> >     Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> >     <mailto:python-dev@python.org <mailto:python-dev@python.org>>
> >     To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> >     <mailto:python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>>
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> >     Message archived at
> >
> https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/
> <https://mail.python.org/archives/list/python-dev@python.org/message/PPKL7466BIG6DPCUIJURLE5ZGFNHBNSM/>
> >     Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
> >
> > --
> > --Guido (mobile)
> >
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UWSV3VDCJOWCXUZIBPDWFJHRLU/
> <https://mail.python.org/archives/list/python-dev@python.org/message/KP3SE6UWSV3VDCJOWCXUZIBPDWFJHRLU/>
> Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> /Pronouns: he/him //(why is my pronoun here?)/
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GC6PMZP3CI5TAZTXJ67GGEUDRZ4IZ7OJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/37FUSJCCQGONCCFSCJFXCXGYICZ7HXMJ/
Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 22 Jun 2021 at 10:06, Petr Viktorin <encukou@gmail.com> wrote:

> On 21. 06. 21 20:20, Guido van Rossum wrote:
> > Okay, I think your evidence can then be discounted. Really, any app that
> > relies on the publicly installed Python runs a serious risk of breaking
> > when that Python gets updated, regardless of whether the ABI changes or
> not.
>
> Unfortunately, this includes scripts for any extensible software
> (Mayavi, GIMP, etc) -- even if Python is bundled with that software,
> upgrading the software risks breaking the scripts.
>
>
I'm confused by what you mean by this, or why it is a problem?

If I upgrade GIMP (and it vendors some version/variant of Python), it is
not unreasonable that this would break a script that I have written in
GIMPython? (GIMP should probably mention that it has changed its Python and
how in the changelog/release notes)

If I upgrade my OS, and I use the system Python, scripts I have written
might break too.

(Of course, GIMP is a placeholder here, I do not actually know what it does
in terms of Python (vendoring), if at all.

On June 22, 2021 11:18:46 AM GMT+02:00, Henk-Jaap Wagenaar <wagenaarhenkjaap@gmail.com> wrote:
>On Tue, 22 Jun 2021 at 10:06, Petr Viktorin <encukou@gmail.com> wrote:
>
>> On 21. 06. 21 20:20, Guido van Rossum wrote:
>> > Okay, I think your evidence can then be discounted. Really, any app
>that
>> > relies on the publicly installed Python runs a serious risk of
>breaking
>> > when that Python gets updated, regardless of whether the ABI
>changes or
>> not.
>>
>> Unfortunately, this includes scripts for any extensible software
>> (Mayavi, GIMP, etc) -- even if Python is bundled with that software,
>> upgrading the software risks breaking the scripts.
>>
>>
>I'm confused by what you mean by this, or why it is a problem?

Not necessarily a problem, I just want to point out that there are situations where you need to depend on Python managed by someone else.

>If I upgrade GIMP (and it vendors some version/variant of Python), it
>is
>not unreasonable that this would break a script that I have written in
>GIMPython? (GIMP should probably mention that it has changed its Python
>and
>how in the changelog/release notes)
>
>If I upgrade my OS, and I use the system Python, scripts I have written
>might break too.
>
>(Of course, GIMP is a placeholder here, I do not actually know what it
>does
>in terms of Python (vendoring), if at all.

GIMP itself doesn't, but it's sometimes distributed in flatpak/appimage with its own bundled/pinned Python. (AFAIK it's usually Python 2.7, for a few reasons; one of them being that upgrading would break scripts)

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.