Mailing List Archive

> - Cython doesn't work because of _PyGen_Send change [1]

I think this is fixed on the latest cython (0.29.23)


On Wed, 26 May 2021 at 01:05, Neil Schemenauer <nas-python@arctrix.com>
wrote:

> On 2021-05-04, ?ukasz Langa wrote:
> > We strongly encourage maintainers of third-party Python projects
> > to test with 3.10 during the beta phase and report issues found to
> > the Python bug tracker <https://bugs.python.org/> as soon as
> > possible.
>
> Testing with Python 3.10b1 is not easy, at least for me. Here is a
> list of initial problems I ran into, from memory:
>
> - Cython doesn't work because of _PyGen_Send change [1]
>
> - scipy cannot be installed because it has requires_python =
> ">=3.7,<3.10". If you manually install from source, it seems to
> work.
>
> - numpy cannot be installed because of _Py_HashDouble() change [2]
>
> - trio cannot be used because of TracebackException.__init__ changes [3]
>
> For the above problems, I would suggest the 3rd party package has
> the issue and it's not a problem with the Python release. However,
> I guess that few people are using Python without 3rd party packages.
> So, it seems unsurprising that beta and RC releases are not well
> tested. It has taken me quite a few hours to get a working version
> of Python 3.10 with all required dependancies such that I can run
> unit tests for some application code.
>
> Can we do any things to improve the situation? Perhaps using the
> pre-release functionality of PyPI would help. We would have to
> somehow encourage 3rd party packages to upload pre-releases that are
> compatible with our beta/RC releases.
>
>
> 1. https://github.com/cython/cython/issues/3876
> 2. https://github.com/numpy/numpy/issues/19033
> 3. https://github.com/python-trio/trio/issues/1899
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2Z23GRYT5VDHR4YJKS6YIQG7G46SC27T/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

Hi,

On Wed, May 26, 2021 at 2:05 AM Neil Schemenauer <nas-python@arctrix.com> wrote:
> - Cython doesn't work because of _PyGen_Send change [1]

My team fixed the Python 3.10 compatibility in the Cython 0.29.x
branch, but the latest 0.29.x release (0.29.23, April 14, 2021)
doesn't include these fixes yet. A new Cython release is needed. I'm
not sure how to help Cython to release a new version.

In Fedora, we have downstream patches until a new Cython is released:
https://src.fedoraproject.org/rpms/Cython/pull-request/28


> - numpy cannot be installed because of _Py_HashDouble() change [2]

"pip install numpy" also fails with an error about "incompatible tag".
It's an issue in packaging which is vendored in wheel which is used
(as a vendored copy?) by pip to build a local wheel package of numpy.
numpy doesn't ship wheel packages for Python 3.10 beta releases which
is a good idea, the ABI is not stable yet.


> Can we do any things to improve the situation? Perhaps using the
> pre-release functionality of PyPI would help. We would have to
> somehow encourage 3rd party packages to upload pre-releases that are
> compatible with our beta/RC releases.

In Fedora, we use a "COPR build": we partially rebuild the OS with
Python 3.10, and then we check for package build failures. We are
working on switching to Python 3.10 by default:

* https://fedoraproject.org/wiki/Changes/Python3.10
* https://bugzilla.redhat.com/show_bug.cgi?id=1890881

For example, my colleague Lumir submitted the trio bug report ;-)
https://github.com/python-trio/trio/issues/1899

We are doing our best to report issues to projects. Sometimes we help
to fix them, but we cannot fix all compatibility issues!

In the past, I tried to work on a small Python script to run the test
suite of some projects with the main branch of Python. I had practical
issues:

* There is no simple way to get install test dependencies of a Python
project, especially non-Python dependencies.
* When a dependency was not compatible with the next Python, I had to
maintain a patch until my fix is accepted upstream.
* Sometimes, I had to pull the code from Git rather than using a
tarball/ZIP (to not have to wait for a new release).

At the end, all these practical issues need solutions which look very
closely to a recipe to build a package for a Linux distribution:

* Recipe to pull the source.
* Recipe to pull build dependencies.
* Downstream patches (until a fix is merged upstream).
* Recipe to run tests: we try to always run tests when we build a
package in Fedora.

Advantages of using Fedora COPR:

* Reuse Fedora infrastructure (building packages need physical servers
to build packages and run tests)
* Reuse the public Fedora bug tracker to coordinate the work.
* Reuse Fedora "specfiles" (recipe to build a RPM package).
* Prepare Fedora to be updated to the next Python.

Sadly, Fedora COPR might be a little bit too Fedora specific: require
to understand specfiles, how to submit a PR on a specfile, use the
Fedora bug tracker, etc.

Fedora Rawhide was just updated to switch Python 3.10 by default!
/usr/bin/python3 is now Python 3.10 beta2. In COPR we could skip tests
to build dependencies and discover more compatibiliy issues. In
Rawhide, tests cannot be skipped and so some packages cannot be built
yet. For example, astropy has a few failing tests:
https://github.com/astropy/astropy/issues/11821

Again, the practical problem is that numpy cannot easily be installed
on Python 3.10 (pip compatibiltiy tag issue), and so astropy may wait
until Python 3.10 final is released to look into these tests failures.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6YRNA2F2G3RSXWG72WFEFOEAARIA57E5/
Code of Conduct: http://python.org/psf/codeofconduct/

For "pip install numpy", a new numpy release should fix the issue.

The issue was already fixed in packaging 20.8:

* https://github.com/pypa/packaging/commit/611982be44d7d91453a97082f58d8ea349f89d00
* https://github.com/pypa/packaging/pull/355

The fix is already included in wheel 0.36.2 (which includes a
vendored copy of packaging):

* https://github.com/pypa/wheel/commit/f6fd2472c7a4a836a3ebe12eab96a1ed8a31c061

And numpy pyproject.toml was updated to use wheel 0.36.2 which contains the fix:

* https://github.com/numpy/numpy/commit/9569f40917c80b9506f0969b1e89e2fc119d4ae3

The pyproject.toml fix is not part of 1.20.3:
https://github.com/numpy/numpy/blob/v1.20.3/pyproject.toml

--

I didn't even know that it was possible to require a specific wheel
version in pyproject.toml to build a binary wheel package in pip!

Current pyproject.toml:
https://github.com/numpy/numpy/blob/main/pyproject.toml

[build-system]
# Minimum requirements for the build system to execute.
requires = [.
"packaging==20.5; platform_machine=='arm64'", # macos M1
"setuptools<49.2.0",
"wheel==0.36.2",
"Cython>=0.29.21,<3.0", # Note: keep in sync with tools/cythonize.py
]

Victor

On Fri, Jun 11, 2021 at 4:30 PM Victor Stinner <vstinner@python.org> wrote:
>
> Hi,
>
> On Wed, May 26, 2021 at 2:05 AM Neil Schemenauer <nas-python@arctrix.com> wrote:
> > - Cython doesn't work because of _PyGen_Send change [1]
>
> My team fixed the Python 3.10 compatibility in the Cython 0.29.x
> branch, but the latest 0.29.x release (0.29.23, April 14, 2021)
> doesn't include these fixes yet. A new Cython release is needed. I'm
> not sure how to help Cython to release a new version.
>
> In Fedora, we have downstream patches until a new Cython is released:
> https://src.fedoraproject.org/rpms/Cython/pull-request/28
>
>
> > - numpy cannot be installed because of _Py_HashDouble() change [2]
>
> "pip install numpy" also fails with an error about "incompatible tag".
> It's an issue in packaging which is vendored in wheel which is used
> (as a vendored copy?) by pip to build a local wheel package of numpy.
> numpy doesn't ship wheel packages for Python 3.10 beta releases which
> is a good idea, the ABI is not stable yet.
>
>
> > Can we do any things to improve the situation? Perhaps using the
> > pre-release functionality of PyPI would help. We would have to
> > somehow encourage 3rd party packages to upload pre-releases that are
> > compatible with our beta/RC releases.
>
> In Fedora, we use a "COPR build": we partially rebuild the OS with
> Python 3.10, and then we check for package build failures. We are
> working on switching to Python 3.10 by default:
>
> * https://fedoraproject.org/wiki/Changes/Python3.10
> * https://bugzilla.redhat.com/show_bug.cgi?id=1890881
>
> For example, my colleague Lumir submitted the trio bug report ;-)
> https://github.com/python-trio/trio/issues/1899
>
> We are doing our best to report issues to projects. Sometimes we help
> to fix them, but we cannot fix all compatibility issues!
>
> In the past, I tried to work on a small Python script to run the test
> suite of some projects with the main branch of Python. I had practical
> issues:
>
> * There is no simple way to get install test dependencies of a Python
> project, especially non-Python dependencies.
> * When a dependency was not compatible with the next Python, I had to
> maintain a patch until my fix is accepted upstream.
> * Sometimes, I had to pull the code from Git rather than using a
> tarball/ZIP (to not have to wait for a new release).
>
> At the end, all these practical issues need solutions which look very
> closely to a recipe to build a package for a Linux distribution:
>
> * Recipe to pull the source.
> * Recipe to pull build dependencies.
> * Downstream patches (until a fix is merged upstream).
> * Recipe to run tests: we try to always run tests when we build a
> package in Fedora.
>
> Advantages of using Fedora COPR:
>
> * Reuse Fedora infrastructure (building packages need physical servers
> to build packages and run tests)
> * Reuse the public Fedora bug tracker to coordinate the work.
> * Reuse Fedora "specfiles" (recipe to build a RPM package).
> * Prepare Fedora to be updated to the next Python.
>
> Sadly, Fedora COPR might be a little bit too Fedora specific: require
> to understand specfiles, how to submit a PR on a specfile, use the
> Fedora bug tracker, etc.
>
> Fedora Rawhide was just updated to switch Python 3.10 by default!
> /usr/bin/python3 is now Python 3.10 beta2. In COPR we could skip tests
> to build dependencies and discover more compatibiliy issues. In
> Rawhide, tests cannot be skipped and so some packages cannot be built
> yet. For example, astropy has a few failing tests:
> https://github.com/astropy/astropy/issues/11821
>
> Again, the practical problem is that numpy cannot easily be installed
> on Python 3.10 (pip compatibiltiy tag issue), and so astropy may wait
> until Python 3.10 final is released to look into these tests failures.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.



--
Night gathers, and now my watch begins. It shall not end until my death.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MKVKZNQFWKKHTECAKDMRUT4KJGEC34SR/
Code of Conduct: http://python.org/psf/codeofconduct/