Thanks for your reply Matt.
Here's what's happening. This is a very old problem reported ages ago which has never been fixed. If I set the target arch to i686 (on an x86_64 build system) Python will cross-compile and bootstrap just fine. But if the host and build triple are the same, then the problem will occur. Python seems to be ASSuming that if the build and host triple match (in this case, both being 'x86_64-linux-gnu') therefore the host and build libraries are binary-compatible--which is NOT actually the case when one is cross-compiling to a different libc, for example. Actually it's kind of brain dead how this is implemented.
Some prior discussions/years-old bug reports:
https://bugs.python.org/issue39399
https://bugs.python.org/issue22724
https://bugs.gentoo.org/705970
In those links, various hacks are attempted with various degrees of success, but then Xavier de Gaye reworked the build system, apparently fixing the problem with this pull request in Dec. 2019:
https://github.com/python/cpython/pull/17420
Unfortunately he became annoyed in the comments, seemingly mostly due to the lack of interest from Python to actually do anything about this problem, and cancelled his pull request. His fix was never implemented, and Python cross-compiling remains broken to this day.
I downloaded his patches, made a minor fix or two, and merged them all together into the one patch attached to this email. When applied to both my build system and target Python, this problem seems to be resolved, for me at least. I'll know more later tonight as I get further into the distro build process.
It's surprising to hear that cross-compiling Python would be any kind of "unusual" thing, considering this is something that *always* has to be done any time one is bootstrapping anything on a new or different system. It surprises me further to see that Python actually requires the *minor* version number to be the same for bootstrapping, also. So not only is Python 3 essentially a different language from Python 2, each point release is different and incompatible also. Amazing.
I guess this shouldn't be surprising, after all of the other questionable design decisions this project has made over the years. I probably also shouldn't be surprised to see such an incredibly important bug going unfixed after all this time. It's Python--by far the #1 biggest annoyance and pain in the ass out of the 1,000+ packages on my distro, ranking just above CUPS and Texlive.
Dave
On Mon, 13 Jun 2022 16:12:26 -0500 (CDT)
Matthew Dixon Cowles <matt@mondoinfo.com> wrote:
> Dear Dave,
>
> > Hello, I'm trying to cross compile a new version of my Linux system
> > with some upgraded packages, including a new Glibc.
>
> I've never had to do that and I am beginning to suspect, from the
> lack of replies here better than this one, that nobody else here has
> had to either.
>
> > I've hit a major snag with Python 3.7 (also tried 3.9 with same
> > result) which makes it through the compile OK, but then bombs when
> > running ensurepip at the end.
>
> If it were me, I'd set --with-ensurepip to no, declare victory, and
> run ensurepip on the target machine.
>
> I haven't been able to find any very good evidence, but I wouldn't be
> surprised if the option to turn ensurepip off is there to help out
> with cross-compiling. For example, here it's being turned off for
> compiling to web assembly:
>
> https://t.ly/_ZE3
>
> (alternatively:
>
> https://github.com/python/cpython/commit/
> 9deb83468c56c7484645e6e3a6d0183cd6a0afd7
>
> )
>
> which looks to me a lot like what you're doing.
>
> What seems to be happening is that in Lib/esurepip/__init__.py is
> calling run_module() in Lib/runpy.py and that's calling
> _run_module_code() which is getting the new, but wrong libraries.
>
> I suspect that it's possible to hack that to do what you want, but as
> far as a moderately close look tells me, it's not written with
> cross-compiling in mind. If I had to do make it work, I'd break out
> into the debugger and fiddle around on a mostly trial-and-error basis.
>
> Even then, I'd have a hacked version of ensurepip/__init__.py and/or
> runpy.py, which I'd want to replace with the originals in the end.
>
> I'm pretty sure that running ensurepip on the target would be easier.
>
> If that's not useful to you, let us know what you think and I'll try
> to think some more.
>
> Regards,
> Matt
>
--
Dave Blanchard <dave@killthe.net>
Here's what's happening. This is a very old problem reported ages ago which has never been fixed. If I set the target arch to i686 (on an x86_64 build system) Python will cross-compile and bootstrap just fine. But if the host and build triple are the same, then the problem will occur. Python seems to be ASSuming that if the build and host triple match (in this case, both being 'x86_64-linux-gnu') therefore the host and build libraries are binary-compatible--which is NOT actually the case when one is cross-compiling to a different libc, for example. Actually it's kind of brain dead how this is implemented.
Some prior discussions/years-old bug reports:
https://bugs.python.org/issue39399
https://bugs.python.org/issue22724
https://bugs.gentoo.org/705970
In those links, various hacks are attempted with various degrees of success, but then Xavier de Gaye reworked the build system, apparently fixing the problem with this pull request in Dec. 2019:
https://github.com/python/cpython/pull/17420
Unfortunately he became annoyed in the comments, seemingly mostly due to the lack of interest from Python to actually do anything about this problem, and cancelled his pull request. His fix was never implemented, and Python cross-compiling remains broken to this day.
I downloaded his patches, made a minor fix or two, and merged them all together into the one patch attached to this email. When applied to both my build system and target Python, this problem seems to be resolved, for me at least. I'll know more later tonight as I get further into the distro build process.
It's surprising to hear that cross-compiling Python would be any kind of "unusual" thing, considering this is something that *always* has to be done any time one is bootstrapping anything on a new or different system. It surprises me further to see that Python actually requires the *minor* version number to be the same for bootstrapping, also. So not only is Python 3 essentially a different language from Python 2, each point release is different and incompatible also. Amazing.
I guess this shouldn't be surprising, after all of the other questionable design decisions this project has made over the years. I probably also shouldn't be surprised to see such an incredibly important bug going unfixed after all this time. It's Python--by far the #1 biggest annoyance and pain in the ass out of the 1,000+ packages on my distro, ranking just above CUPS and Texlive.
Dave
On Mon, 13 Jun 2022 16:12:26 -0500 (CDT)
Matthew Dixon Cowles <matt@mondoinfo.com> wrote:
> Dear Dave,
>
> > Hello, I'm trying to cross compile a new version of my Linux system
> > with some upgraded packages, including a new Glibc.
>
> I've never had to do that and I am beginning to suspect, from the
> lack of replies here better than this one, that nobody else here has
> had to either.
>
> > I've hit a major snag with Python 3.7 (also tried 3.9 with same
> > result) which makes it through the compile OK, but then bombs when
> > running ensurepip at the end.
>
> If it were me, I'd set --with-ensurepip to no, declare victory, and
> run ensurepip on the target machine.
>
> I haven't been able to find any very good evidence, but I wouldn't be
> surprised if the option to turn ensurepip off is there to help out
> with cross-compiling. For example, here it's being turned off for
> compiling to web assembly:
>
> https://t.ly/_ZE3
>
> (alternatively:
>
> https://github.com/python/cpython/commit/
> 9deb83468c56c7484645e6e3a6d0183cd6a0afd7
>
> )
>
> which looks to me a lot like what you're doing.
>
> What seems to be happening is that in Lib/esurepip/__init__.py is
> calling run_module() in Lib/runpy.py and that's calling
> _run_module_code() which is getting the new, but wrong libraries.
>
> I suspect that it's possible to hack that to do what you want, but as
> far as a moderately close look tells me, it's not written with
> cross-compiling in mind. If I had to do make it work, I'd break out
> into the debugger and fiddle around on a mostly trial-and-error basis.
>
> Even then, I'd have a hacked version of ensurepip/__init__.py and/or
> runpy.py, which I'd want to replace with the originals in the end.
>
> I'm pretty sure that running ensurepip on the target would be easier.
>
> If that's not useful to you, let us know what you think and I'll try
> to think some more.
>
> Regards,
> Matt
>
--
Dave Blanchard <dave@killthe.net>
Attached Files:
-
python-3.9.13-cross-compile-fix.patch (17.8 KB)