Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Python>
  3. Dev
1 2  View All
C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API
vstinner at python
Mar 22, 2022, 11:07 AM
Post #1 of 47 (1994 views)
Permalink
Hi,

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):

* https://github.com/python/cpython/pull/32052
* https://github.com/python/cpython/pull/32054

API:

* _PyFrameEvalFunction type
* _PyInterpreterState_GetEvalFrameFunc()
* _PyInterpreterState_SetEvalFrameFunc()
* (undocumented) _PyEval_EvalFrameDefault()

The private API to get/set the eval function *is* documented at:
https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc

I added the Get/Set functions so debuggers don't have to access
directly to the PyInterpreterState structure which has been moved to
the internal C API in Python 3.8.

This API causes me multiple issues:

* It's a private API and I'm trying to remove the private API from the
public C API header files.
* The _PyFrameEvalFunction type is not stable: it got a new "tstate"
parameter in Python 3.9 and the type of the second parameter changed
from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
* These functions use the _PyInterpreterFrame type which is part of
the internal C API.

While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.

I propose to move theses API to the internal header files
(Include/internals/) to clarify that it's not part of the public C API
and that there is no backward compatibility warranty.

The change is being discussed at:
https://bugs.python.org/issue46850

--

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild. I'm not
sure if these ones should be moved to the internal C API. They can be
left unchanged, since they don't use a type only defined by the
internal C API.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MA4FQ7G6F35NG3TUN6RQPXRGXTYMFMDY/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
steve.dower at python
Mar 22, 2022, 11:26 AM
Post #2 of 47 (1994 views)
Permalink
On 3/22/2022 6:07 PM, Victor Stinner wrote:
> I proposed two PRs to move the private C API (Include/cpython/) of PEP
> 523 "Adding a frame evaluation API to CPython" to the internal C API
> (Include/internals/):

After a normal deprecation period, yes?

> While Pyston didn't bring a JIT compiler to Python with PEP 523,
> debuggers were made faster by using this API. Debuggers like pydevd,
> debugpy and ptvsd use it.

Pyjion (https://github.com/tonybaloney/Pyjion) uses it, and ptvsd never
did. I also understand that pydevd - and implicitly debugpy - disabled
its use of the API by default because of reliability, but that was a few
years back so maybe they fixed it. So it's likely only being used by the
project that requested it, though I don't think that's enough to justify
skipping normal deprecation.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/X6EC7KW4B5CB4R5CS4WSQYWSXIYFV4J3/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
vstinner at python
Mar 22, 2022, 4:28 PM
Post #3 of 47 (1993 views)
Permalink
On Tue, Mar 22, 2022 at 7:33 PM Steve Dower <steve.dower@python.org> wrote:
> After a normal deprecation period, yes?

There is no backward compatibility warranty and no deprecation process
for private APIs.

Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GTHWKT3QPPMU4VEC6SNO5HLRNNAZI6US/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
encukou at gmail
Mar 23, 2022, 2:48 AM
Post #4 of 47 (1993 views)
Permalink
On 22. 03. 22 19:07, Victor Stinner wrote:
> Hi,
>
> I proposed two PRs to move the private C API (Include/cpython/) of PEP
> 523 "Adding a frame evaluation API to CPython" to the internal C API
> (Include/internals/):
>
> * https://github.com/python/cpython/pull/32052
> * https://github.com/python/cpython/pull/32054
>
> API:
>
> * _PyFrameEvalFunction type
> * _PyInterpreterState_GetEvalFrameFunc()
> * _PyInterpreterState_SetEvalFrameFunc()
> * (undocumented) _PyEval_EvalFrameDefault()
>
> The private API to get/set the eval function *is* documented at:
> https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc
>
> I added the Get/Set functions so debuggers don't have to access
> directly to the PyInterpreterState structure which has been moved to
> the internal C API in Python 3.8.
>
> This API causes me multiple issues:
>
> * It's a private API and I'm trying to remove the private API from the
> public C API header files.
> * The _PyFrameEvalFunction type is not stable: it got a new "tstate"
> parameter in Python 3.9 and the type of the second parameter changed
> from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
> * These functions use the _PyInterpreterFrame type which is part of
> the internal C API.
>
> While Pyston didn't bring a JIT compiler to Python with PEP 523,
> debuggers were made faster by using this API. Debuggers like pydevd,
> debugpy and ptvsd use it.
>
> I propose to move theses API to the internal header files
> (Include/internals/) to clarify that it's not part of the public C API
> and that there is no backward compatibility warranty.
>
> The change is being discussed at:
> https://bugs.python.org/issue46850
>
> --
>
> PEP 523 API added more private functions for code objects:
>
> * _PyEval_RequestCodeExtraIndex()
> * _PyCode_GetExtra()
> * _PyCode_SetExtra()
>
> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> pydevd debugger. The two others seem to be unused in the wild. I'm not
> sure if these ones should be moved to the internal C API. They can be
> left unchanged, since they don't use a type only defined by the
> internal C API.

PEP 523 clearly meant for these to be used by external tools, but made
them private. That sounds like a contradiction.

Brett/Dino, what was the intent here?

IMO, if external code should use these, they should lose the leading
underscore.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CW3EKBNZCPYPCQMNEXPBV2BKTKI3Y6TX/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
steve.dower at python
Mar 23, 2022, 9:46 AM
Post #5 of 47 (1993 views)
Permalink
On 3/22/2022 11:28 PM, Victor Stinner wrote:
> On Tue, Mar 22, 2022 at 7:33 PM Steve Dower <steve.dower@python.org> wrote:
>> After a normal deprecation period, yes?
>
> There is no backward compatibility warranty and no deprecation process
> for private APIs.

And yet you're asking the question, which means you know these are
special ;)

The PEP says: "The API is purposefully listed as private to communicate
the fact that there are no semantic guarantees of the API between Python
releases."

Absence/presence isn't a semantic guarantee, it's an availability
guarantee. Code using them should be able to rely on their presence, and
ideally their prototype (though it seems we've messed that up in the
past), but shouldn't expect code that worked against 3.8 to also work
against 3.9 or 3.10.

Perhaps in hindsight, we could have not used the underscore and just
explicitly described them as being behaviorally unstable between major
versions. I guess that would have raised exactly the same question though.

The point is, it's a documented API that we've told people they can use.
We can't simply revoke that without telling people that it's going to
happen, even if we covered ourselves for there being version changes
that affect how they need to be used.

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YF75IDAHS66BKJZDAAQDGOW2IGU2KME5/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
brett at python
Mar 23, 2022, 12:58 PM
Post #6 of 47 (1993 views)
Permalink
On Wed, Mar 23, 2022 at 2:48 AM Petr Viktorin <encukou@gmail.com> wrote:

> On 22. 03. 22 19:07, Victor Stinner wrote:
> > Hi,
> >
> > I proposed two PRs to move the private C API (Include/cpython/) of PEP
> > 523 "Adding a frame evaluation API to CPython" to the internal C API
> > (Include/internals/):
> >
> > * https://github.com/python/cpython/pull/32052
> > * https://github.com/python/cpython/pull/32054
> >
> > API:
> >
> > * _PyFrameEvalFunction type
> > * _PyInterpreterState_GetEvalFrameFunc()
> > * _PyInterpreterState_SetEvalFrameFunc()
> > * (undocumented) _PyEval_EvalFrameDefault()
> >
> > The private API to get/set the eval function *is* documented at:
> >
> https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc
> >
> > I added the Get/Set functions so debuggers don't have to access
> > directly to the PyInterpreterState structure which has been moved to
> > the internal C API in Python 3.8.
> >
> > This API causes me multiple issues:
> >
> > * It's a private API and I'm trying to remove the private API from the
> > public C API header files.
> > * The _PyFrameEvalFunction type is not stable: it got a new "tstate"
> > parameter in Python 3.9 and the type of the second parameter changed
> > from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
> > * These functions use the _PyInterpreterFrame type which is part of
> > the internal C API.
> >
> > While Pyston didn't bring a JIT compiler to Python with PEP 523,
> > debuggers were made faster by using this API. Debuggers like pydevd,
> > debugpy and ptvsd use it.
> >
> > I propose to move theses API to the internal header files
> > (Include/internals/) to clarify that it's not part of the public C API
> > and that there is no backward compatibility warranty.
> >
> > The change is being discussed at:
> > https://bugs.python.org/issue46850
> >
> > --
> >
> > PEP 523 API added more private functions for code objects:
> >
> > * _PyEval_RequestCodeExtraIndex()
> > * _PyCode_GetExtra()
> > * _PyCode_SetExtra()
> >
> > The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> > pydevd debugger. The two others seem to be unused in the wild. I'm not
> > sure if these ones should be moved to the internal C API. They can be
> > left unchanged, since they don't use a type only defined by the
> > internal C API.
>
> PEP 523 clearly meant for these to be used by external tools, but made
> them private. That sounds like a contradiction.
>
> Brett/Dino, what was the intent here?
>

From the PEP <https://peps.python.org/pep-0523/#expanding-pycodeobject>:
"The API is purposefully listed as private to communicate the fact that
there are no semantic guarantees of the API between Python releases."

-Brett


>
> IMO, if external code should use these, they should lose the leading
> underscore.
>
>
>
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
fabiofz at gmail
Mar 24, 2022, 11:39 AM
Post #7 of 47 (1992 views)
Permalink
>
> PEP 523 API added more private functions for code objects:
>
> * _PyEval_RequestCodeExtraIndex()
> * _PyCode_GetExtra()
> * _PyCode_SetExtra()
>
> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> pydevd debugger. The two others seem to be unused in the wild. I'm not
> sure if these ones should be moved to the internal C API. They can be
> left unchanged, since they don't use a type only defined by the
> internal C API.
>
Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.

i.e.:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311

The debugger already has workarounds because of changes to evaluation api
over time (see:
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
and I know 3.11 won't be different.

I'm ok with changes as I understand that this is a special API -- as long
as there's still a way to use it and get the information needed (the
debugger already goes through many hops because it needs to use many
internals of CPython -- in every new release it's a **really** big task to
update to the latest version as almost everything that the debugger relies
to make debugging fast changes across versions and I never really know if
it'll be possible to support it until I really try to do the port -- I
appreciate having less things in a public API so it's easier to have
extensions work in other interpreters/not recompiling on newer versions,
but please keep it possible to use private APIs which provides the same
access that CPython has to access things internally for special cases such
as the debugger).

Maybe later on that PEP from mark which allows a better debugger API could
alleviate that (but until then, if possible I appreciate it if there's some
effort not to break things unless really needed -- ideally with
instructions on how to port).

Anyways, to wrap up, the debugger already needs to be built with
`Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API
(as long as it's still accessible in that case) is probably not a big issue
for the debugger and having setters/getters to set it instead of relying on
`state.interp.eval_frame` seems good to me.

Cheers,

Fabio
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
fabiofz at gmail
Mar 24, 2022, 12:06 PM
Post #8 of 47 (1992 views)
Permalink
Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny <fabiofz@gmail.com>
escreveu:

> PEP 523 API added more private functions for code objects:
>>
>> * _PyEval_RequestCodeExtraIndex()
>> * _PyCode_GetExtra()
>> * _PyCode_SetExtra()
>>
>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
>> pydevd debugger. The two others seem to be unused in the wild. I'm not
>> sure if these ones should be moved to the internal C API. They can be
>> left unchanged, since they don't use a type only defined by the
>> internal C API.
>>
> Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.
>
> i.e.:
>
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
>
> The debugger already has workarounds because of changes to evaluation api
> over time (see:
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
> and I know 3.11 won't be different.
>
> I'm ok with changes as I understand that this is a special API -- as long
> as there's still a way to use it and get the information needed (the
> debugger already goes through many hops because it needs to use many
> internals of CPython -- in every new release it's a **really** big task to
> update to the latest version as almost everything that the debugger relies
> to make debugging fast changes across versions and I never really know if
> it'll be possible to support it until I really try to do the port -- I
> appreciate having less things in a public API so it's easier to have
> extensions work in other interpreters/not recompiling on newer versions,
> but please keep it possible to use private APIs which provides the same
> access that CPython has to access things internally for special cases such
> as the debugger).
>
> Maybe later on that PEP from mark which allows a better debugger API could
> alleviate that (but until then, if possible I appreciate it if there's some
> effort not to break things unless really needed -- ideally with
> instructions on how to port).
>
> Anyways, to wrap up, the debugger already needs to be built with
> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API
> (as long as it's still accessible in that case) is probably not a big issue
> for the debugger and having setters/getters to set it instead of relying on
> `state.interp.eval_frame` seems good to me.
>
> Cheers,
>
> Fabio
>
>

I think the main issue here is the compatibility across the same version
though... is it possible to have some kind of guarantee on private APIs
that something won't change across micro-releases?

I.e.: having the frame evaluation function change across major releases and
having them be reworked seems reasonable, but then having the frame
evaluation be changed across micro-releases wouldn't be.

So, I'm ok in pushing things to the internal API, but then I still would
like guarantees about the compatibility of that API in the same major
release (otherwise those setters/getters/frame evaluation should probably
remain on the public API if the related structure was moved to the internal
API).

Cheers,

Fabio
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
encukou at gmail
Mar 28, 2022, 4:44 AM
Post #9 of 47 (1974 views)
Permalink
On 24. 03. 22 20:06, Fabio Zadrozny wrote:
>
> Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny <fabiofz@gmail.com
> <mailto:fabiofz@gmail.com>> escreveu:
>
> PEP 523 API added more private functions for code objects:
>
> * _PyEval_RequestCodeExtraIndex()
> * _PyCode_GetExtra()
> * _PyCode_SetExtra()
>
> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> pydevd debugger. The two others seem to be unused in the wild.
> I'm not
> sure if these ones should be moved to the internal C API. They
> can be
> left unchanged, since they don't use a type only defined by the
> internal C API.
>
> Just to note, the pydevd/debugpy debuggers actually uses all of
> those APIs.
>
> i.e.:
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
> <https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
> <https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
> <https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311>
>
> The debugger already has workarounds because of changes to
> evaluation api over time (see:
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491
> <https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491>)
> and I know 3.11 won't be different.
>
> I'm ok with changes as I understand that this is a special API -- as
> long as there's still a way to use it and get the information needed
> (the debugger already goes through many hops because it needs to use
> many internals of CPython -- in every new release it's a **really**
> big task to update to the latest version as almost everything that
> the debugger relies to make debugging fast changes across versions
> and I never really know if it'll be possible to support it until I
> really try to do the port -- I appreciate having less things in a
> public API so it's easier to have extensions work in other
> interpreters/not recompiling on newer versions, but please keep it
> possible to use private APIs which provides the same access that
> CPython has to access things internally for special cases such as
> the debugger).
>
> Maybe later on that PEP from mark which allows a better debugger API
> could alleviate that (but until then, if possible I appreciate it if
> there's some effort not to break things unless really needed --
> ideally with instructions on how to port).
>
> Anyways, to wrap up, the debugger already needs to be built with
> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private
> API (as long as it's still accessible in that case) is probably not
> a big issue for the debugger and having setters/getters to set it
> instead of relying on `state.interp.eval_frame` seems good to me.
>
> Cheers,
>
> Fabio
>
>
> I think the main issue here is the compatibility across the same version
> though... is it possible to have some kind of guarantee on private APIs
> that something won't change across micro-releases?

Currently we don't really have that (except that in practice, bugfix
releases tend to not need internal API changes.)


> I.e.: having the frame evaluation function change across major releases
> and having them be reworked seems reasonable, but then having the frame
> evaluation be changed across micro-releases wouldn't be.
>
> So, I'm ok in pushing things to the internal API, but then I still would
> like guarantees about the compatibility of that API in the same major
> release (otherwise those setters/getters/frame evaluation should
> probably remain on the public API if the related structure was moved to
> the internal API).

Perhaps we need a new "tier" of C API for debuggers -- API that's
guaranteed stable for a major release, and if it's changed it should
always break with compile errors (e.g. the function gets a new
argument), rather than silently change semantics.
The internal API Cython & greenlet need might go it this category too.


_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OF4UNE22IDDJDJVQZMOUXHELAYEFCJ45/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
python-dev at python
Mar 28, 2022, 2:44 PM
Post #10 of 47 (1974 views)
Permalink
The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.

The related project is TorchDynamo, which can be found here:
https://github.com/facebookresearch/torchdynamo

We will likely move this into the core of PyTorch closer to release.

If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
tjreedy at udel
Mar 29, 2022, 3:24 PM
Post #11 of 47 (1974 views)
Permalink
On 3/28/2022 5:44 PM, Jason Ansel via Python-Dev wrote:
> The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
>
> The related project is TorchDynamo, which can be found here:
> https://github.com/facebookresearch/torchdynamo
>
> We will likely move this into the core of PyTorch closer to release.
>
> If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

I believe that you will just have to add a new or different #include
statement. I recommend that you add a comment to the issue,
https://bugs.python.org/issue46850,
and even more, that you create a branch with the diffs in

https://github.com/python/cpython/pull/32052
https://github.com/python/cpython/pull/32054

applied so that you can experiment and verify what I believe.


--
Terry Jan Reedy

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JS7YMJGEWCY5PBF4ATMZPWXVY6PYLPKK/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
steve.dower at python
Mar 29, 2022, 5:25 PM
Post #12 of 47 (1974 views)
Permalink
On 3/28/2022 10:44 PM, Jason Ansel via Python-Dev wrote:
> The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
>
> The related project is TorchDynamo, which can be found here:
> https://github.com/facebookresearch/torchdynamo
>
> We will likely move this into the core of PyTorch closer to release.
>
> If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?

You'd be able to use it, but if we don't nail down the compatibility
guarantees then you might find PyTorch doesn't work properly against
3.11.(n+1) while 3.11.n was fine.

Right now, the API is allowed/expected to change between 3.x releases
(which normally we don't allow without a deprecation period) but it
still has to remain compatible within a single 3.x release. Making it
fully internal *without adding a stability guarantee* means it could
change more frequently, which you wouldn't be able to handle as an
installable package.

It's *unlikely* that it'll change that often, because there are still
other public interfaces that cannot. But, the plan behind this is to
make more stuff internal so that it can be modified more freely, so we
may see that rate of change increase.

In this world, your best bet is for TorchDynamo to become a full CPython
fork. And when that's your best bet, it should set off alarms everywhere
;) But that is what relying on internal APIs implies, and is certainly
what people would have to do if we were to remove the existing PEP's
interface. (It's what Pyjion was doing before the PEP was written, and
it's why Pyjion doesn't have to be a full fork these days.)

So you probably want to state explicit support for either keeping the
APIs public and slightly-flexible, or making them internal but stable.
(Public and stable won't work at all for us, and normal internal won't
work at all for you.)

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MU56YBMPWSVMK3HAUJOBRW4KEN2ZLVCG/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
vstinner at python
Mar 30, 2022, 7:56 AM
Post #13 of 47 (1974 views)
Permalink
On Wed, Mar 30, 2022 at 2:26 AM Steve Dower <steve.dower@python.org> wrote:
> Right now, the API is allowed/expected to change between 3.x releases
> (which normally we don't allow without a deprecation period) but it
> still has to remain compatible within a single 3.x release. Making it
> fully internal *without adding a stability guarantee* means it could
> change more frequently, which you wouldn't be able to handle as an
> installable package.
>
> It's *unlikely* that it'll change that often, because there are still
> other public interfaces that cannot. But, the plan behind this is to
> make more stuff internal so that it can be modified more freely, so we
> may see that rate of change increase.

Well, my plan is not break these internal C API just for fun. It's
only to better "advertise" the separation between the "stable" public
C API (backward compatibility warranty) and the "unstable"
private/internal C API (can change any time, changes are not
documented).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OMGUI5N33BG3EU4OG3IUZXJQF6XU7X2B/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
vstinner at python
Mar 30, 2022, 8:01 AM
Post #14 of 47 (1974 views)
Permalink
The internal C API can be used on purpose. But there is no backward
compatibility warranty and it can change anytime. In practice, usually
it only changes in 3.x.0 releases. For example, these private C API
changed in Python 3.9 and Python 3.11 (see my first email in the other
PEP 523 thread).

To use the internal C API, you have to declare the Py_BUILD_CORE macro
and include an internal C API header file. For
_PyInterpreterState_SetEvalFrameFunc(), it should be:

#ifndef Py_BUILD_CORE_MODULE
# define Py_BUILD_CORE_MODULE
#endif
#include <Python.h>
#include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc()
#include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault

Victor

On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
<python-dev@python.org> wrote:
>
> The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
>
> The related project is TorchDynamo, which can be found here:
> https://github.com/facebookresearch/torchdynamo
>
> We will likely move this into the core of PyTorch closer to release.
>
> If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
python-dev at python
Mar 30, 2022, 8:33 AM
Post #15 of 47 (1974 views)
Permalink
Got it, thanks for the clarifications!

Tracking 3.x Python versions is fine. We already need to do that to support things like new bytecodes, and PyTorch supports an explicit list of 3.x Python versions with separate builds for each.

Tracking 3.x.y Python versions would be much more painful, and make us need to rethink our approach.

So what Steve Downer described as "remain compatible within a single 3.x release", seems like exactly what we want. I support that level of compatibility guarantee. Could we keep that guarantee with this change?

Thanks,
Jason





________________________________________
From: Victor Stinner <vstinner@python.org>
Sent: Wednesday, March 30, 2022 7:56 AM
To: Steve Dower
Cc: Jason Ansel; python-dev@python.org
Subject: Re: [Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

On Wed, Mar 30, 2022 at 2:26 AM Steve Dower <steve.dower@python.org> wrote:
> Right now, the API is allowed/expected to change between 3.x releases
> (which normally we don't allow without a deprecation period) but it
> still has to remain compatible within a single 3.x release. Making it
> fully internal *without adding a stability guarantee* means it could
> change more frequently, which you wouldn't be able to handle as an
> installable package.
>
> It's *unlikely* that it'll change that often, because there are still
> other public interfaces that cannot. But, the plan behind this is to
> make more stuff internal so that it can be modified more freely, so we
> may see that rate of change increase.

Well, my plan is not break these internal C API just for fun. It's
only to better "advertise" the separation between the "stable" public
C API (backward compatibility warranty) and the "unstable"
private/internal C API (can change any time, changes are not
documented).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U7M65SSDZMM7LNAKEDZZ4KKQIFTCOQ2M/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
guido at python
Mar 30, 2022, 8:42 AM
Post #16 of 47 (1974 views)
Permalink
In the not so distant past I have proposed to introduce a new category,
"Unstable APIs". These are public but are not guaranteed to be backwards
compatible in feature releases (though I feel they should remain so in
bugfix releases).

I'm not sure whether those should have a leading underscore or not. Perhaps
(like some other languages do and like maybe we've used in a few places)
the name could just include the word "Unstable"?

On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner <vstinner@python.org> wrote:

> The internal C API can be used on purpose. But there is no backward
> compatibility warranty and it can change anytime. In practice, usually
> it only changes in 3.x.0 releases. For example, these private C API
> changed in Python 3.9 and Python 3.11 (see my first email in the other
> PEP 523 thread).
>
> To use the internal C API, you have to declare the Py_BUILD_CORE macro
> and include an internal C API header file. For
> _PyInterpreterState_SetEvalFrameFunc(), it should be:
>
> #ifndef Py_BUILD_CORE_MODULE
> # define Py_BUILD_CORE_MODULE
> #endif
> #include <Python.h>
> #include <internal/pycore_interp.h> //
> _PyInterpreterState_SetEvalFrameFunc()
> #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault
>
> Victor
>
> On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
> <python-dev@python.org> wrote:
> >
> > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this
> proposal may break the next major release of PyTorch.
> >
> > The related project is TorchDynamo, which can be found here:
> > https://github.com/facebookresearch/torchdynamo
> >
> > We will likely move this into the core of PyTorch closer to release.
> >
> > If the changed happens, would PyTorch still be able to use the eval
> frame API? Or would it prevent from being used entirely?
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-leave@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
encukou at gmail
Mar 30, 2022, 8:51 AM
Post #17 of 47 (1974 views)
Permalink
On 30. 03. 22 17:42, Guido van Rossum wrote:
> In the not so distant past I have proposed to introduce a new category,
> "Unstable APIs". These are public but are not guaranteed to be backwards
> compatible in feature releases (though I feel they should remain so in
> bugfix releases).
>
> I'm not sure whether those should have a leading underscore or not.
> Perhaps (like some other languages do and like maybe we've used in a few
> places) the name could just include the word "Unstable"?

IMO, the underscore should mark an API as internal: it can change at any
time (though in practice it often doesn't, e.g. to accommodate projects
that used it before a policy is written down).

This is useful e.g. for macros/static functions that wrap access to
something private, where the definition needs to be available but marked
"keep off".


> On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner <vstinner@python.org
> <mailto:vstinner@python.org>> wrote:
>
> The internal C API can be used on purpose. But there is no backward
> compatibility warranty and it can change anytime. In practice, usually
> it only changes in 3.x.0 releases. For example, these private C API
> changed in Python 3.9 and Python 3.11 (see my first email in the other
> PEP 523 thread).
>
> To use the internal C API, you have to declare the Py_BUILD_CORE macro
> and include an internal C API header file. For
> _PyInterpreterState_SetEvalFrameFunc(), it should be:
>
> #ifndef Py_BUILD_CORE_MODULE
> #  define Py_BUILD_CORE_MODULE
> #endif
> #include <Python.h>
> #include <internal/pycore_interp.h> //
> _PyInterpreterState_SetEvalFrameFunc()
> #include <internal/pycore_ceval.h>  // _PyEval_EvalFrameDefault
>
> Victor
>
> On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
> <python-dev@python.org <mailto:python-dev@python.org>> wrote:
> >
> > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0,
> so this proposal may break the next major release of PyTorch.
> >
> > The related project is TorchDynamo, which can be found here:
> > https://github.com/facebookresearch/torchdynamo
> <https://github.com/facebookresearch/torchdynamo>
> >
> > We will likely move this into the core of PyTorch closer to release.
> >
> > If the changed happens, would PyTorch still be able to use the
> eval frame API?  Or would it prevent from being used entirely?
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> > To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> <https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/>
> > Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org
> <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> <https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/>
> Code of Conduct: http://python.org/psf/codeofconduct/
> <http://python.org/psf/codeofconduct/>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> /Pronouns: he/him //(why is my pronoun here?)/
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EIK3XLPV7A7WVB4FA47PM2G2SJ42RERO/
> Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MBHN5AR5C7S3SP3OBSFV3ES26PDM746R/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
steve.dower at python
Mar 30, 2022, 8:58 AM
Post #18 of 47 (1974 views)
Permalink
On 3/30/2022 4:42 PM, Guido van Rossum wrote:
> In the not so distant past I have proposed to introduce a new category,
> "Unstable APIs". These are public but are not guaranteed to be backwards
> compatible in feature releases (though I feel they should remain so in
> bugfix releases).

Agreed. This is definitely a new category, and it seems the only thing
we're debating now is whether or not to add/omit the underscore.

> I'm not sure whether those should have a leading underscore or not.
> Perhaps (like some other languages do and like maybe we've used in a few
> places) the name could just include the word "Unstable"?

I don't think we have "Unstable" anywhere, though we do have "Unsafe"
(which is more about threading than stability, so not right for this).
But I'd be okay with that as a compromise.

I'd prefer to not have public-unstable APIs hidden behind the same
preprocessor directive as internal APIs. That's a big switch to throw
that may also activate other settings - for example, on Windows we will
set the minimum Windows version in our headers if you enable internal
APIs, and disable automatic linking of the Python DLL. Easy enough
things to work around, but they probably need to be explicitly
documented as well if we're going to document public APIs as requiring
Py_BUILD_CORE (and I don't want to have to document that kind of stuff).

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZEPJSIJNXQKXTOE2MRNS6GJRP52WLDEF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
jpe at wingware
Mar 30, 2022, 9:02 AM
Post #19 of 47 (1974 views)
Permalink
As someone who maintains a debugger that uses private api’s, I’d like to see some commitment to seeing them not change in micro releases such as 3.11.1 -> 3.11.2. Micro releases should be compatible with other micro releases of the same major.minor release such as 3.11 so that an extension compiled against 3.11.1 will work with 3.11.2.

Compatibility was broken (at least) once in a 2.x series (maybe 2.4?) and we started getting reports of people using 2.x.0 were having problems. Fortunately, in that case compiling against 2.x.0 produced an extension that worked against all micro releases.

Note that I’m not asking for private api’s not change in major.micro releases such as 3.10 -> 3.11. I know that there can be very good reasons to change private api's because that I probably will have work to do in order to support a new major.micro release.

I also don’t think that exposing everything that every extension needs with a non-private api is a good idea because then the internals will be more difficult to change — you’d be signing up to support that api for all future major.micro versions until there’s a compatibility break.

John

> On Mar 30, 2022, at 11:01 AM, Victor Stinner <vstinner@python.org> wrote:
>
> The internal C API can be used on purpose. But there is no backward
> compatibility warranty and it can change anytime. In practice, usually
> it only changes in 3.x.0 releases. For example, these private C API
> changed in Python 3.9 and Python 3.11 (see my first email in the other
> PEP 523 thread).
>
> To use the internal C API, you have to declare the Py_BUILD_CORE macro
> and include an internal C API header file. For
> _PyInterpreterState_SetEvalFrameFunc(), it should be:
>
> #ifndef Py_BUILD_CORE_MODULE
> # define Py_BUILD_CORE_MODULE
> #endif
> #include <Python.h>
> #include <internal/pycore_interp.h> // _PyInterpreterState_SetEvalFrameFunc()
> #include <internal/pycore_ceval.h> // _PyEval_EvalFrameDefault
>
> Victor
>
> On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
> <python-dev@python.org> wrote:
>>
>> The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this proposal may break the next major release of PyTorch.
>>
>> The related project is TorchDynamo, which can be found here:
>> https://github.com/facebookresearch/torchdynamo
>>
>> We will likely move this into the core of PyTorch closer to release.
>>
>> If the changed happens, would PyTorch still be able to use the eval frame API? Or would it prevent from being used entirely?
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TK27HEVDUWQQZQ4XIKTQLDHWK52ZLHWN/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
guido at python
Mar 30, 2022, 11:29 AM
Post #20 of 47 (1974 views)
Permalink
I don't think such a guarantee (to not vary internal APIs from 3.x.y to
3.x.(y+1)) has ever before been made in writing, although in practice we've
been doing this (more so now than we were in the 2.x timeframe).

I think we should not lightly vary internal APIs between bugfix releases,
but at the same time I am not sure I want to absolutely guarantee that no
internal API ever changes in a bugfix release (because that might prevent
fixing certain bugs).

So I think we need to officially embrace a category of "unstable public
APIs" and set a policy specifically for those, before we can make progress.

I'd like the SC to take some initiative here.

On Wed, Mar 30, 2022 at 9:34 AM Jason Ansel via Python-Dev <
python-dev@python.org> wrote:

> Got it, thanks for the clarifications!
>
> Tracking 3.x Python versions is fine. We already need to do that to
> support things like new bytecodes, and PyTorch supports an explicit list of
> 3.x Python versions with separate builds for each.
>
> Tracking 3.x.y Python versions would be much more painful, and make us
> need to rethink our approach.
>
> So what Steve Downer described as "remain compatible within a single 3.x
> release", seems like exactly what we want. I support that level of
> compatibility guarantee. Could we keep that guarantee with this change?
>
> Thanks,
> Jason
>
>
>
>
>
> ________________________________________
> From: Victor Stinner <vstinner@python.org>
> Sent: Wednesday, March 30, 2022 7:56 AM
> To: Steve Dower
> Cc: Jason Ansel; python-dev@python.org
> Subject: Re: [Python-Dev] Re: C API: Move PEP 523 "Adding a frame
> evaluation API to CPython" private C API to the internal C API
>
> On Wed, Mar 30, 2022 at 2:26 AM Steve Dower <steve.dower@python.org>
> wrote:
> > Right now, the API is allowed/expected to change between 3.x releases
> > (which normally we don't allow without a deprecation period) but it
> > still has to remain compatible within a single 3.x release. Making it
> > fully internal *without adding a stability guarantee* means it could
> > change more frequently, which you wouldn't be able to handle as an
> > installable package.
> >
> > It's *unlikely* that it'll change that often, because there are still
> > other public interfaces that cannot. But, the plan behind this is to
> > make more stuff internal so that it can be modified more freely, so we
> > may see that rate of change increase.
>
> Well, my plan is not break these internal C API just for fun. It's
> only to better "advertise" the separation between the "stable" public
> C API (backward compatibility warranty) and the "unstable"
> private/internal C API (can change any time, changes are not
> documented).
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/U7M65SSDZMM7LNAKEDZZ4KKQIFTCOQ2M/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
sebastian at sipsolutions
Mar 30, 2022, 12:20 PM
Post #21 of 47 (1974 views)
Permalink
On Wed, 2022-03-30 at 17:51 +0200, Petr Viktorin wrote:
> On 30. 03. 22 17:42, Guido van Rossum wrote:
> > In the not so distant past I have proposed to introduce a new
> > category,
> > "Unstable APIs". These are public but are not guaranteed to be
> > backwards
> > compatible in feature releases (though I feel they should remain so
> > in
> > bugfix releases).
> >
> > I'm not sure whether those should have a leading underscore or not.
> > Perhaps (like some other languages do and like maybe we've used in
> > a few
> > places) the name could just include the word "Unstable"?
>
> IMO, the underscore should mark an API as internal: it can change at
> any
> time (though in practice it often doesn't, e.g. to accommodate
> projects
> that used it before a policy is written down).
>

That is fair, although there are documented underscored ones:
https://docs.python.org/3/search.html?q=_Py

I suppose that means all bets are off _unless_ it is documented or
later adopted as stable API (e.g. `_PyObject_Vectorcall`).

With that, the only "not obviously OK" use in NumPy that I am aware of
is `_Py_HashDouble` (it seems undocumented).

Maybe "unless documented" is just a clear enough distinction in
practice.
Although, to some degree, I think it would be clearer if symbols that
have a realistic chance of changing in bug-fix releases had an
additional safe-guard.

- Sebastian



> This is useful e.g. for macros/static functions that wrap access to
> something private, where the definition needs to be available but
> marked
> "keep off".
>
>
> > On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner
> > <vstinner@python.org 
> > <mailto:vstinner@python.org>> wrote:
> >
> >     The internal C API can be used on purpose. But there is no
> > backward
> >     compatibility warranty and it can change anytime. In practice,
> > usually
> >     it only changes in 3.x.0 releases. For example, these private C
> > API
> >     changed in Python 3.9 and Python 3.11 (see my first email in
> > the other
> >     PEP 523 thread).
> >
> >     To use the internal C API, you have to declare the
> > Py_BUILD_CORE macro
> >     and include an internal C API header file. For
> >     _PyInterpreterState_SetEvalFrameFunc(), it should be:
> >
> >     #ifndef Py_BUILD_CORE_MODULE
> >     #  define Py_BUILD_CORE_MODULE
> >     #endif
> >     #include <Python.h>
> >     #include <internal/pycore_interp.h> //
> >     _PyInterpreterState_SetEvalFrameFunc()
> >     #include <internal/pycore_ceval.h>  // _PyEval_EvalFrameDefault
> >
> >     Victor
> >
> >     On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
> >     <python-dev@python.org <mailto:python-dev@python.org>> wrote:
> >      >
> >      > The PyTorch team plans to use PEP 523 as a part of PyTorch
> > 2.0,
> >     so this proposal may break the next major release of PyTorch.
> >      >
> >      > The related project is TorchDynamo, which can be found here:
> >      > https://github.com/facebookresearch/torchdynamo
> >     <https://github.com/facebookresearch/torchdynamo>
> >      >
> >      > We will likely move this into the core of PyTorch closer to
> > release.
> >      >
> >      > If the changed happens, would PyTorch still be able to use
> > the
> >     eval frame API?  Or would it prevent from being used entirely?
> >      > _______________________________________________
> >      > Python-Dev mailing list -- python-dev@python.org
> >     <mailto:python-dev@python.org>
> >      > To unsubscribe send an email to python-dev-leave@python.org
> >     <mailto:python-dev-leave@python.org>
> >      >
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> >     <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> >      > Message archived at
> >    
> > https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> >    
> > <https://mail.python.org/archives/list/python-dev@python.org/messag
> > e/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/>
> >      > Code of Conduct: http://python.org/psf/codeofconduct/
> >     <http://python.org/psf/codeofconduct/>
> >
> >
> >
> >     --
> >     Night gathers, and now my watch begins. It shall not end until
> > my death.
> >     _______________________________________________
> >     Python-Dev mailing list -- python-dev@python.org
> >     <mailto:python-dev@python.org>
> >     To unsubscribe send an email to python-dev-leave@python.org
> >     <mailto:python-dev-leave@python.org>
> >     https://mail.python.org/mailman3/lists/python-dev.python.org/
> >     <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> >     Message archived at
> >    
> > https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> >    
> > <https://mail.python.org/archives/list/python-dev@python.org/messag
> > e/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/>
> >     Code of Conduct: http://python.org/psf/codeofconduct/
> >     <http://python.org/psf/codeofconduct/>
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
> > /Pronouns: he/him //(why is my pronoun here?)/
> > <
> > http://feministing.com/2015/02/03/how-using-they-as-a-singular-prono
> > un-can-change-the-world/>
> >
> > _______________________________________________
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-leave@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/python-dev@python.org/message/EIK3XLPV7A7WVB4FA47PM2G2SJ42RERO/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/MBHN5AR5C7S3SP3OBSFV3ES26PDM746R/
> Code of Conduct: http://python.org/psf/codeofconduct/

Attached Files:

Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
vstinner at python
Mar 31, 2022, 12:06 AM
Post #22 of 47 (1974 views)
Permalink
On Wed, Mar 30, 2022 at 9:26 PM Sebastian Berg
<sebastian@sipsolutions.net> wrote:
> That is fair, although there are documented underscored ones:
> https://docs.python.org/3/search.html?q=_Py
>
> I suppose that means all bets are off _unless_ it is documented or
> later adopted as stable API (e.g. `_PyObject_Vectorcall`).
>
> With that, the only "not obviously OK" use in NumPy that I am aware of
> is `_Py_HashDouble` (it seems undocumented).
>
> Maybe "unless documented" is just a clear enough distinction in
> practice.
> Although, to some degree, I think it would be clearer if symbols that
> have a realistic chance of changing in bug-fix releases had an
> additional safe-guard.

Since Python 3.7, there is a work-in-progress to (1) better hide
internal C API functions and to (2) promote *private* C API functions
being used by 3rd party projects as documented and well tested
*public* functions.

An example of (2) is the addition of float pack/unpack public
functions, like PyFloat_Pack8() and PyFloat_Unpack8():
https://docs.python.org/dev/c-api/float.html#pack-and-unpack-functions

There were previously known as private _PyFloat_Pack8() and
_PyFloat_Unpack8() functions. They are used by a few serialization
projects like msgpack. When they were made public, tests were added
and the existing comments were converted to documentation and
enhanced.

I discovered that these private functions were used when I moved them
to the internal C API in bpo-46906 (1) and it broke a few projects.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DT2PUGWPOSZOJZLR4VOMPS6QOS3PDRYR/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
vstinner at python
Apr 1, 2022, 1:33 AM
Post #23 of 47 (1974 views)
Permalink
On Mon, Mar 28, 2022 at 1:44 PM Petr Viktorin <encukou@gmail.com> wrote:
> Perhaps we need a new "tier" of C API for debuggers -- API that's
> guaranteed stable for a major release, and if it's changed it should
> always break with compile errors (e.g. the function gets a new
> argument), rather than silently change semantics.
> The internal API Cython & greenlet need might go it this category too.

We should provide public C API functions for Cython and greenlet
needs: well tested and documented functions. There is an on-going work
in Python 3.11:

* https://docs.python.org/dev/c-api/frame.html
* https://github.com/faster-cpython/ideas/issues/309
* https://bugs.python.org/issue40421

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H64KULVN5O4MXPWWYMO2VBJRTLNNWBYX/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
vstinner at python
Apr 1, 2022, 1:46 AM
Post #24 of 47 (1974 views)
Permalink
On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum <guido@python.org> wrote:
> In the not so distant past I have proposed to introduce a new category, "Unstable APIs". These are public but are not guaranteed to be backwards compatible in feature releases (though I feel they should remain so in bugfix releases).
>
> I'm not sure whether those should have a leading underscore or not. Perhaps (like some other languages do and like maybe we've used in a few places) the name could just include the word "Unstable"?

I recall discussions about PyCode_New(). IMO this API should not be
public at all. It leaks way too many implementation details: cell
variables, optimization for bytecode offset to line and column
numbers, exception table, etc. This API changed often and will
continue to change.

It's not the right abstraction level. We just exposed the function
because it was technically possible and it was convenient since Python
consumes its own C API. The internal C API was created to draw a line
between what API can be consumed outside Python (public) and what API
must not be used outside Python (internal) unless you're writing a
debugger or other uncommon very specific use case. The main difference
is the warranties provided (public) or not (internal) by Python:
tests, documentation, backward compatibility.

In Python, sadly the types.CodeType type also has a public constructor
and many projects break at each Python release because the API
changes. Hopefully, it seems like the new CodeType.replace() method
added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
better abstraction and closer to what developers need in practice.

I'm not convinced that advertising an API as being Unstable (in the
documentation?) is going to solve any kind of problem. People love to
use private APIs, and they do it simply because it's technically
possible :-) At the end of the day, we have to help them updating
their code, otherwise we (at least my Red Hat team) cannot update
Python.

I designed the internal C API to be more annoying to be used (need to
define a macro, need more specific #include) in the hope that people
will think twice before using it :-)

Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API [ In reply to ]
rosuav at gmail
Apr 1, 2022, 1:55 AM
Post #25 of 47 (1974 views)
Permalink
On Fri, 1 Apr 2022 at 19:51, Victor Stinner <vstinner@python.org> wrote:
> In Python, sadly the types.CodeType type also has a public constructor
> and many projects break at each Python release because the API
> changes. Hopefully, it seems like the new CodeType.replace() method
> added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> better abstraction and closer to what developers need in practice.

It certainly has been for me. When I want to do bytecode hackery, I
usually start by creating a function with def/lambda, then construct a
modified function using f.__code__.replace(). It's the easiest way to
ensure that all the little details are correct.

ChrisA
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JJO6LDWA4BE7SFTEF2VP6NLSIX6ZBCGD/
Code of Conduct: http://python.org/psf/codeofconduct/
1 2  View All

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal