Mailing List Archive

Regressions caused the recent work on ceval.c and frame objects
Hi,

The recent optimization work on ceval.c and frame objects introduced
regressions and the situation is stuck for 4 months (bpo-43760).

Right now, maybe the best would be to revert the 2 commits in the 3.10
branch, to get more time in Python 3.11 development cycle to solve
these issues.


(1) "The DISPATCH() macro is not as efficient as it could be (move
PyThreadState.use_tracing)"
https://bugs.python.org/issue43760

This change introduced an incompatible C API change. It's not
documented in What's New in Python 3.10 and there is no solution for
the 4 broken projects (including Cython).

I proposed a C API but so far, nobody implemented it.

Another option is to fix each project since the list is short. Right
now, only 4 projects are known to be broken. Fixing Cython is not
enough, you need to get a new release of broken projects (using
Cython) to regenerate the C code with the updated Cython.


(2) "Performance regression 3.10b1 and later on Windows: Py_DECREF()
not inlined in PGO build"
https://bugs.python.org/issue45116

Changes made in bpo-43760 caused a performance regression on the PGO
build of Windows.

It's a tricky issue about thresholds in compiler PGO optimization,
inlining or not static inline functions, number of statements per
function, etc.

It was proposed to workaround the specific case of the huge
_PyEval_EvalFrameDefault() function (3500 lines of C code) by
converting again some static inline functions to macros. They were
macros in Python 3.8 and were fine in Python 3.9. The performance
regression with the Windows PGO build was introduced by the recent
ceval.c work.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XLEMW7PWYGIK2RHHICO3CNITZ4ETO3OZ/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
Are you sure Cython is still broken? It looks like it was fixed back in May (https://github.com/cython/cython/issues/4153) and all the tests look to be passing on the 3.10-dev CI run for Cython. I think it only affected the profiling feature on Cython (which most people will have turned off) so probably won't cause widespread breakage.

Which is not to say that you shouldn't fix the issue, but I don't think it's a disaster from Cython's point of view.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B6OLXTTP5KQVNFLZD2WWV4YSYNSUZX6Q/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
On 19. 09. 21 21:34, dw-git@d-woods.co.uk wrote:
> Are you sure Cython is still broken? It looks like it was fixed back in May (https://github.com/cython/cython/issues/4153) and all the tests look to be passing on the 3.10-dev CI run for Cython. I think it only affected the profiling feature on Cython (which most people will have turned off) so probably won't cause widespread breakage.
>
> Which is not to say that you shouldn't fix the issue, but I don't think it's a disaster from Cython's point of view.

Cython is fixed. However all the projects that ship Cython pre-generated C
sources in sdists on PyPI that are affected and were not re-generated will not
compile on Python 3.10. (Which might already be the case for different changes
that affected Cython.)

--
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YHPXTAOHGYDKYF4WY42L2KSTDLR65Q2T/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
Are projects that ship pre-compiled wheels impacted? Eg twisted-iocpsupport
?

On Sun, 19 Sep 2021, 22:09 Miro Hron?ok, <mhroncok@redhat.com> wrote:

> On 19. 09. 21 21:34, dw-git@d-woods.co.uk wrote:
> > Are you sure Cython is still broken? It looks like it was fixed back in
> May (https://github.com/cython/cython/issues/4153) and all the tests look
> to be passing on the 3.10-dev CI run for Cython. I think it only affected
> the profiling feature on Cython (which most people will have turned off) so
> probably won't cause widespread breakage.
> >
> > Which is not to say that you shouldn't fix the issue, but I don't think
> it's a disaster from Cython's point of view.
>
> Cython is fixed. However all the projects that ship Cython pre-generated C
> sources in sdists on PyPI that are affected and were not re-generated will
> not
> compile on Python 3.10. (Which might already be the case for different
> changes
> that affected Cython.)
>
> --
> Miro Hron?ok
> --
> Phone: +420777974800
> IRC: mhroncok
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YHPXTAOHGYDKYF4WY42L2KSTDLR65Q2T/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
On 20. 09. 21 0:10, Thomas Grainger wrote:
> Are projects that ship pre-compiled wheels impacted? Eg twisted-iocpsupport ?

I guess that if they managed to compile with 3.10, they shouldn't be.

--
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TDGG4VFRZO5CDKTUH4AXYDLVXOKNLZZE/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
Will all packages that use Cython have to upgrade Cython to work with 3.10?

That can be a pretty heavy lift. A lot of us pin Cython.

-CHB


On Sun, Sep 19, 2021 at 4:32 PM Miro Hron?ok <mhroncok@redhat.com> wrote:

> On 20. 09. 21 0:10, Thomas Grainger wrote:
> > Are projects that ship pre-compiled wheels impacted? Eg
> twisted-iocpsupport ?
>
> I guess that if they managed to compile with 3.10, they shouldn't be.
>
> --
> Miro Hron?ok
> --
> Phone: +420777974800
> IRC: mhroncok
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TDGG4VFRZO5CDKTUH4AXYDLVXOKNLZZE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
Christopher Barker, PhD (Chris)

Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
> Will all packages that use Cython have to upgrade Cython to work with 3.10?

For this particular issue you'll only have to upgrade if you use "profiling" (I doubt that many packages routinely build with Cython profiling turned on). However, it's possible there are other 3.10 bugfixes in Cython - I'm not completely sure. I think the easiest way to know is to try to build a package for 3.10.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/52VFBHR7AHTXPLC434E4BPXNXVUU3SVF/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
On 20. 09. 21 7:40, Christopher Barker wrote:
> Will all packages that use Cython have to upgrade Cython to work with 3.10?

I am not sure if all, but many will do.

--
Miro Hron?ok
--
Phone: +420777974800
IRC: mhroncok

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MSKPT6P3JNVVDXLWXTVSB2B33DLNW6KA/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Regressions caused the recent work on ceval.c and frame objects [ In reply to ]
msgpack 1.0.2 sdist includes c++ code generated by Cython 0.29.21
released at 2020-07-09.
Python 3.10 can build it without any source modifications.
So I don't expect Python 3.10 requires re-generation for most Cython users.

Regards,

On Mon, Sep 20, 2021 at 4:11 PM <dw-git@d-woods.co.uk> wrote:
>
> > Will all packages that use Cython have to upgrade Cython to work with 3.10?
>
> For this particular issue you'll only have to upgrade if you use "profiling" (I doubt that many packages routinely build with Cython profiling turned on). However, it's possible there are other 3.10 bugfixes in Cython - I'm not completely sure. I think the easiest way to know is to try to build a package for 3.10.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/52VFBHR7AHTXPLC434E4BPXNXVUU3SVF/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Inada Naoki <songofacandy@gmail.com>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7MLTC3NK535YOQAKD6VQTT4QZIREG57N/
Code of Conduct: http://python.org/psf/codeofconduct/