Mailing List Archive

Please explain how to migrate when a function is removed, thanks ;-)
Hi,

We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
are facing many Python 3.10 incompatible changes. Most changes were
planned with a deprecation period, but, as usual, projects are not
tested with DeprecationWarning treated as errors, and so tons of
packages now fail to build.

I'm not here to talk about DeprecationWarning, but how we communicate
on incompatible changes made on purpose. For example, What's New in
Python 3.8 announces: "In asyncio, the explicit passing of a loop
argument has been deprecated and will be removed in version 3.10". As
expected, the parameter was removed in Python 3.10 (bpo-42392), but I
cannot see anything in What's New in Python 3.10. The problem is that
I don't know how to fix projects broken by this change. I'm not
complaining about this specific change, but more generally.

I strongly suggest to well document incompatible changes. The bare
minimum is to mention them in "Porting to Python 3.10" section:
https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10

A link to the bpo sometimes helps to understand how to port code. But
I would really appreciate if authors of incompatible changes would
explain how to add Python 3.10 support to existing projects, without
losing support for older Python versions. Not just "this function is
now removed, good luck!" :-)

I didn't touch asyncio for at least 1 year, so I don't know what
happens if I remove a loop argument. Does an application remain
compatible with Python 3.6 without passing loop?

I know that I made multiple incompatible changes myself and I'm sure
that the documentation should probably be enhanced, but I also expect
others to help on that! ;-)

Please think about people who have to port 4000 Python projects to Python 3.10!

---

The best would be to ship a tool which adds Python 3.10 support to
existing projects without losing support with Python 3.6. Maybe
something like https://github.com/asottile/pyupgrade could be used for
that? pyupgrade looks more specific to the Python syntax, than the
usage of the stdlib.

I wrote such tool to add Python 3.10 support to C extensions without
losing support with Python 2.7. It relies on a header file
(pythoncapi_compat.h) which provides new C API functions on old Python
versions.
https://github.com/pythoncapi/pythoncapi_compat

For example, it replaces "obj->ob_refcnt = refcnt;" with
"Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
3.8 and older (function added to Python 3.9)

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/O3T7SK3BGMFWMLCQXDODZJSBL42AUWTR/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
A PR was proposed to document the removal of the loop parameter:
https://github.com/python/cpython/pull/24256

Victor

On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner <vstinner@python.org> wrote:
>
> Hi,
>
> We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
> are facing many Python 3.10 incompatible changes. Most changes were
> planned with a deprecation period, but, as usual, projects are not
> tested with DeprecationWarning treated as errors, and so tons of
> packages now fail to build.
>
> I'm not here to talk about DeprecationWarning, but how we communicate
> on incompatible changes made on purpose. For example, What's New in
> Python 3.8 announces: "In asyncio, the explicit passing of a loop
> argument has been deprecated and will be removed in version 3.10". As
> expected, the parameter was removed in Python 3.10 (bpo-42392), but I
> cannot see anything in What's New in Python 3.10. The problem is that
> I don't know how to fix projects broken by this change. I'm not
> complaining about this specific change, but more generally.
>
> I strongly suggest to well document incompatible changes. The bare
> minimum is to mention them in "Porting to Python 3.10" section:
> https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
>
> A link to the bpo sometimes helps to understand how to port code. But
> I would really appreciate if authors of incompatible changes would
> explain how to add Python 3.10 support to existing projects, without
> losing support for older Python versions. Not just "this function is
> now removed, good luck!" :-)
>
> I didn't touch asyncio for at least 1 year, so I don't know what
> happens if I remove a loop argument. Does an application remain
> compatible with Python 3.6 without passing loop?
>
> I know that I made multiple incompatible changes myself and I'm sure
> that the documentation should probably be enhanced, but I also expect
> others to help on that! ;-)
>
> Please think about people who have to port 4000 Python projects to Python 3.10!
>
> ---
>
> The best would be to ship a tool which adds Python 3.10 support to
> existing projects without losing support with Python 3.6. Maybe
> something like https://github.com/asottile/pyupgrade could be used for
> that? pyupgrade looks more specific to the Python syntax, than the
> usage of the stdlib.
>
> I wrote such tool to add Python 3.10 support to C extensions without
> losing support with Python 2.7. It relies on a header file
> (pythoncapi_compat.h) which provides new C API functions on old Python
> versions.
> https://github.com/pythoncapi/pythoncapi_compat
>
> For example, it replaces "obj->ob_refcnt = refcnt;" with
> "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
> 3.8 and older (function added to Python 3.9)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CVLDV7VIVENV6HMQ4PRAN3VRFU26CMJI/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
Thanks for bringing attention to this, Victor, and to Ken Jin (GH:
Fidget-Spinner) for the PR. I've just completed reviewing and merging the
PR, so hopefully anyone affected will now have a more clear idea of how to
migrate their asyncio code to 3.10. Having the porting method explicitly
documented certainly helps to smooth the version transition process and
reduce headaches. :-)

Going forward, I'll try to make more of an active effort to ensure any
potentially incompatible changes I'm involved with include a clear method
of porting documented in their respective whatsnew. It can be easy to
forget at times that a seemingly minor fix which is intuitively clear to
the authors of a change may not be as clear to those not involved with it,
regardless of how difficult the fix actually is.

On Tue, Jan 19, 2021 at 12:03 PM Victor Stinner <vstinner@python.org> wrote:

> A PR was proposed to document the removal of the loop parameter:
> https://github.com/python/cpython/pull/24256
>
> Victor
>
> On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner <vstinner@python.org>
> wrote:
> >
> > Hi,
> >
> > We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
> > are facing many Python 3.10 incompatible changes. Most changes were
> > planned with a deprecation period, but, as usual, projects are not
> > tested with DeprecationWarning treated as errors, and so tons of
> > packages now fail to build.
> >
> > I'm not here to talk about DeprecationWarning, but how we communicate
> > on incompatible changes made on purpose. For example, What's New in
> > Python 3.8 announces: "In asyncio, the explicit passing of a loop
> > argument has been deprecated and will be removed in version 3.10". As
> > expected, the parameter was removed in Python 3.10 (bpo-42392), but I
> > cannot see anything in What's New in Python 3.10. The problem is that
> > I don't know how to fix projects broken by this change. I'm not
> > complaining about this specific change, but more generally.
> >
> > I strongly suggest to well document incompatible changes. The bare
> > minimum is to mention them in "Porting to Python 3.10" section:
> > https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
> >
> > A link to the bpo sometimes helps to understand how to port code. But
> > I would really appreciate if authors of incompatible changes would
> > explain how to add Python 3.10 support to existing projects, without
> > losing support for older Python versions. Not just "this function is
> > now removed, good luck!" :-)
> >
> > I didn't touch asyncio for at least 1 year, so I don't know what
> > happens if I remove a loop argument. Does an application remain
> > compatible with Python 3.6 without passing loop?
> >
> > I know that I made multiple incompatible changes myself and I'm sure
> > that the documentation should probably be enhanced, but I also expect
> > others to help on that! ;-)
> >
> > Please think about people who have to port 4000 Python projects to
> Python 3.10!
> >
> > ---
> >
> > The best would be to ship a tool which adds Python 3.10 support to
> > existing projects without losing support with Python 3.6. Maybe
> > something like https://github.com/asottile/pyupgrade could be used for
> > that? pyupgrade looks more specific to the Python syntax, than the
> > usage of the stdlib.
> >
> > I wrote such tool to add Python 3.10 support to C extensions without
> > losing support with Python 2.7. It relies on a header file
> > (pythoncapi_compat.h) which provides new C API functions on old Python
> > versions.
> > https://github.com/pythoncapi/pythoncapi_compat
> >
> > For example, it replaces "obj->ob_refcnt = refcnt;" with
> > "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
> > 3.8 and older (function added to Python 3.9)
> >
> > Victor
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CVLDV7VIVENV6HMQ4PRAN3VRFU26CMJI/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
I don't know if this is already covered in the discussion and in our
processes, but in addition to documenting the instructions in the release
in which things break, I think it would also be good to include such
instructions in any earlier release in which the thing is merely
deprecated. In those cases, the instructions would be the same, but they
would be on how to make the deprecation warning go away rather than to
unbreak things. That way, people have the information to make their code
forward-compatible earlier, as opposed to waiting for the breaking release.
Personally, I like to address DeprecationWarnings when I first see them, as
opposed to waiting until later.

Is there / would it make sense to have a section analogous to "Porting to
Python X" that covers "Make All DeprecationWarnings Go Away in X"? If we
had such a section, the "Porting to" section could be constructed by
copying the relevant bits from that section in the previous release.

--Chris


On Wed, Jan 20, 2021 at 4:47 PM Kyle Stanley <aeros167@gmail.com> wrote:

> Thanks for bringing attention to this, Victor, and to Ken Jin (GH:
> Fidget-Spinner) for the PR. I've just completed reviewing and merging the
> PR, so hopefully anyone affected will now have a more clear idea of how to
> migrate their asyncio code to 3.10. Having the porting method explicitly
> documented certainly helps to smooth the version transition process and
> reduce headaches. :-)
>
> Going forward, I'll try to make more of an active effort to ensure any
> potentially incompatible changes I'm involved with include a clear method
> of porting documented in their respective whatsnew. It can be easy to
> forget at times that a seemingly minor fix which is intuitively clear to
> the authors of a change may not be as clear to those not involved with it,
> regardless of how difficult the fix actually is.
>
> On Tue, Jan 19, 2021 at 12:03 PM Victor Stinner <vstinner@python.org>
> wrote:
>
>> A PR was proposed to document the removal of the loop parameter:
>> https://github.com/python/cpython/pull/24256
>>
>> Victor
>>
>> On Tue, Jan 19, 2021 at 1:34 PM Victor Stinner <vstinner@python.org>
>> wrote:
>> >
>> > Hi,
>> >
>> > We are working on upgrading Python from 3.9 to 3.10 in Fedora and we
>> > are facing many Python 3.10 incompatible changes. Most changes were
>> > planned with a deprecation period, but, as usual, projects are not
>> > tested with DeprecationWarning treated as errors, and so tons of
>> > packages now fail to build.
>> >
>> > I'm not here to talk about DeprecationWarning, but how we communicate
>> > on incompatible changes made on purpose. For example, What's New in
>> > Python 3.8 announces: "In asyncio, the explicit passing of a loop
>> > argument has been deprecated and will be removed in version 3.10". As
>> > expected, the parameter was removed in Python 3.10 (bpo-42392), but I
>> > cannot see anything in What's New in Python 3.10. The problem is that
>> > I don't know how to fix projects broken by this change. I'm not
>> > complaining about this specific change, but more generally.
>> >
>> > I strongly suggest to well document incompatible changes. The bare
>> > minimum is to mention them in "Porting to Python 3.10" section:
>> > https://docs.python.org/dev/whatsnew/3.10.html#porting-to-python-3-10
>> >
>> > A link to the bpo sometimes helps to understand how to port code. But
>> > I would really appreciate if authors of incompatible changes would
>> > explain how to add Python 3.10 support to existing projects, without
>> > losing support for older Python versions. Not just "this function is
>> > now removed, good luck!" :-)
>> >
>> > I didn't touch asyncio for at least 1 year, so I don't know what
>> > happens if I remove a loop argument. Does an application remain
>> > compatible with Python 3.6 without passing loop?
>> >
>> > I know that I made multiple incompatible changes myself and I'm sure
>> > that the documentation should probably be enhanced, but I also expect
>> > others to help on that! ;-)
>> >
>> > Please think about people who have to port 4000 Python projects to
>> Python 3.10!
>> >
>> > ---
>> >
>> > The best would be to ship a tool which adds Python 3.10 support to
>> > existing projects without losing support with Python 3.6. Maybe
>> > something like https://github.com/asottile/pyupgrade could be used for
>> > that? pyupgrade looks more specific to the Python syntax, than the
>> > usage of the stdlib.
>> >
>> > I wrote such tool to add Python 3.10 support to C extensions without
>> > losing support with Python 2.7. It relies on a header file
>> > (pythoncapi_compat.h) which provides new C API functions on old Python
>> > versions.
>> > https://github.com/pythoncapi/pythoncapi_compat
>> >
>> > For example, it replaces "obj->ob_refcnt = refcnt;" with
>> > "Py_SET_REFCNT(obj, refcnt);" and provides Py_SET_REFCNT() to Python
>> > 3.8 and older (function added to Python 3.9)
>> >
>> > Victor
>> > --
>> > Night gathers, and now my watch begins. It shall not end until my death.
>>
>>
>>
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/CVLDV7VIVENV6HMQ4PRAN3VRFU26CMJI/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RGSEPSSPDIVKWYKACONKRUFBEXQ5YKGC/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
Currently, we wait until the first user complains, sometimes after a
3.x.0 final release, before starting to document how to port existing
code. I agree that it should be done earlier.

I suggest that developers who want to introduce an incompatible change
think about how to port existing code, especially if it's possible to
write code compatible with the old and the new behavior. It should be
done while the incompatible change is designed.

I also agree that we document how to port code in the change which
introduce the deprecation, not when we introduce the incompatible
change (ex: remove a function).

I also suggest to communicate on future incompatible changes ;-)

Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/F4YKK4ERIDDYZ7RUVZ5FPJFA4TSA3AP7/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
I agree that when we land a feature that introduces incompatible change
like this, as part of the PR it should also include updating the
documentation on how to migrate.
I would think that the feature should not be merged unless the doc has been
updated too.

Perhaps we should include this explicitly in devguide, as one of the things
to consider when reviewing pull requests.
Do we have such a guide/doc yet, on how to review CPython pull requests?


On Thu, Jan 21, 2021 at 2:49 AM Victor Stinner <vstinner@python.org> wrote:

> Currently, we wait until the first user complains, sometimes after a
> 3.x.0 final release, before starting to document how to port existing
> code. I agree that it should be done earlier.
>
> I suggest that developers who want to introduce an incompatible change
> think about how to port existing code, especially if it's possible to
> write code compatible with the old and the new behavior. It should be
> done while the incompatible change is designed.
>
> I also agree that we document how to port code in the change which
> introduce the deprecation, not when we introduce the incompatible
> change (ex: remove a function).
>
> I also suggest to communicate on future incompatible changes ;-)
>
> Victor
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/F4YKK4ERIDDYZ7RUVZ5FPJFA4TSA3AP7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
On Jan 20, 2021 at 9:51 PM Chros Jerdonek <chris.jerdonek@gmail.com> wrote:

> Is there / would it make sense to have a section analogous to "Porting to
> Python X" that covers "Make All DeprecationWarnings Go Away in X"? If we
> had such a section, the "Porting to" section could be constructed by
> copying the relevant bits from that section in the previous release.


+1 from me to include this section in 3.10 and future whatsnew.

On Thu, Jan 21, 2021 at 2:17 PM Mariatta <mariatta@python.org> wrote:

> I agree that when we land a feature that introduces incompatible change
> like this, as part of the PR it should also include updating the
> documentation on how to migrate.
> I would think that the feature should not be merged unless the doc has
> been updated too.
>
> Perhaps we should include this explicitly in devguide, as one of the
> things to consider when reviewing pull requests.
> Do we have such a guide/doc yet, on how to review CPython pull requests?
>

This seems like it would be a good location to include that information:
https://devguide.python.org/pullrequest/#how-to-review-a-pull-request

Maybe also a brief mention in the checklist for
https://devguide.python.org/committing/#is-the-pr-ready-to-be-accepted.
Step 9 currently has: "Was “What’s New” updated (as appropriate)?", which
could have a sub-point to mention "Incompatible changes should include
migration information in their respective "What's New".".
Re: Please explain how to migrate when a function is removed, thanks ;-) [ In reply to ]
I opened an issue on Devguide: https://github.com/python/devguide/issues/655



On Thu, Jan 21, 2021 at 11:37 AM Kyle Stanley <aeros167@gmail.com> wrote:

> On Jan 20, 2021 at 9:51 PM Chros Jerdonek <chris.jerdonek@gmail.com>
> wrote:
>
>> Is there / would it make sense to have a section analogous to "Porting to
>> Python X" that covers "Make All DeprecationWarnings Go Away in X"? If we
>> had such a section, the "Porting to" section could be constructed by
>> copying the relevant bits from that section in the previous release.
>
>
> +1 from me to include this section in 3.10 and future whatsnew.
>
> On Thu, Jan 21, 2021 at 2:17 PM Mariatta <mariatta@python.org> wrote:
>
>> I agree that when we land a feature that introduces incompatible change
>> like this, as part of the PR it should also include updating the
>> documentation on how to migrate.
>> I would think that the feature should not be merged unless the doc has
>> been updated too.
>>
>> Perhaps we should include this explicitly in devguide, as one of the
>> things to consider when reviewing pull requests.
>> Do we have such a guide/doc yet, on how to review CPython pull requests?
>>
>
> This seems like it would be a good location to include that information:
> https://devguide.python.org/pullrequest/#how-to-review-a-pull-request
>
> Maybe also a brief mention in the checklist for
> https://devguide.python.org/committing/#is-the-pr-ready-to-be-accepted.
> Step 9 currently has: "Was “What’s New” updated (as appropriate)?", which
> could have a sub-point to mention "Incompatible changes should include
> migration information in their respective "What's New".".
>