Mailing List Archive

[issue4954] native build of python win32 using msys under wine.
New submission from Luke Kenneth Casson Leighton <lkcl@lkcl.net>:

this patch uses work from #3871 to get a build of python for win32
by running msys under Wine, the windows emulator, on linux.
no proprietary operating system or proprietary software was used.

/bin/sh.exe is so xxxxing unbelievably slow on msys under wine
(each sh.exe takes two seconds to start up, loads 240 fonts
and 40 libraries) that running configure was completely intolerable.

consequently, given that PC/config.h is perfectly useable on nt,
this patch totally bypasses the majority of autoconf tests, and
thus cuts the configure time down from three hours to (only!)
fifteen minutes.

configure must be run as:
./configure --prefix/python25 --enable-win32build=yes \
--enable=shared=yes

there is an additional bug in msys /bin/sh which stops python.exe
from executing scripts that are handed to it on the command-line.
scripts that are fed as standard-input are absolutely fine
(even the same script). running the same script from python.exe
from wineconsole cmd prompt is _also_ fine.

as a result, the build process terminates at:
./python.exe -E setup.py build

to complete the build, "wineconsole cmd" must be run, cd $(BUILDDIR)
and then run the commands manually:
./python.exe -E setup.py build
./python.exe -E setup.py install
then, back in the borked /bin/sh.exe, make install can be run.

stunningly, this does actually work: python.exe is build,
libpython2.5.dll and libpython2.5.dll.a are built... amazing.

other necessary workarounds:

* gcc -shared .... --out-implib=libpython2.5.dll.a .... etc fails
spectacularly to create both the .a implib _and_ the .dll.
consequently, it was necessary to split these up and use dlltool
to create the dll, implib and .def file etc. etc.

* ar and ranlib on msys, versions 2.16, 2.17 _and_ 2.19 of binutils
are borked. 2.16 and 2.17 throw an exception; 2.19 segfaults.
consequently, building of $LIBRARY had to be terminated with
prejudice. --enable-shared=no will fail miserably.

regarding testing: it's all a bit... odd.

* ctypes tests cause segfaults! the S8H one manages a segfault
all on its own, but there are other tests that cause corruption
that only shows up later... euurgh.

* builtin test_open fails due to putting \r\n instead of \n in the file

* test_file.py gets a sharing violation

* test_str gets a great one!

test_str^M
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
fixme:msvcrt:MSVCRT__sopen : pmode 0x01b6 ignored
err:seh:setup_exception_record stack overflow 872 bytes in thread
001e eip 7bc3b1dc esp 00410fc8 stack 0x410000-0x411000-0x610000

yup.... test_floatformatting.

ahh, this is gooood :)


overall it's a damn good start - i'm dead impressed with
wine and msys in that they exist at all.

compiling and testing python under wine + msys should help
test and improve both those projects, and both this patch
and #3871 will help get python off of its dependence on
proprietary software.

----------
components: Build
files: f
messages: 79916
nosy: lkcl
severity: normal
status: open
title: native build of python win32 using msys under wine.
versions: Python 2.5
Added file: http://bugs.python.org/file12755/f

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Changes by Luke Kenneth Casson Leighton <lkcl@lkcl.net>:


----------
type: -> feature request

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Martin v. Löwis <martin@v.loewis.de> added the comment:

Are you seriously requesting inclusion of the file f into Python? It
contains many changes that are unsuitable for inclusion, such as the
removal of graminit.c and the removal of configure.

(besides, I personally think that the ability to compile Python with
msys under Wine doesn't belong into core Python, because it is a
minority platform)

----------
nosy: +loewis

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

Is Msys+Mingw32 (running on a regular Windows) an interesting
configuration to support?
The build tools are similar to the ones used by cygwin, except that the
C runtime is msvcrt.

----------
nosy: +amaury.forgeotdarc

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

martin, hi, thanks for responding.

* graminit and configure were removed because they are built
automatically. no project should ever include auto-generated files so i
assumed that it would be reasonable to remove them from the python_2.5.2
"original" git commit that i did, in order to produce the diff. on a
build, graminit got _replaced_ with one that had ^M after evvvvery
siiingle line.

so - if graminit and configure _are_ in the main
python source tree, it's a serious mistake that should be
rectified _immediately_.

... but i doubt that, and so graminit and configure "appearing"
to be "removed" can be ignored.

* regarding your comment that compiling python under msys under wine
is a "minority platform", i believe that that is also a serious
assumption.

the reason why it's a minority platform is because... THERE WASN'T
ANY CHOICE.

i.e. msys and wine simply have not been good enough - and certainly
not _demonstrated_ as being "good enough" - which this patch shows
that they now most definitely are - to _have_ any choice about
whether python should be compiled with purely free software tools.

instead of depending on some xxxxing proprietary piece of double-xxxx
operating system _and_ the development IDE it walked in on.

sorry about that - just to emphasise how distasteful i find it to
be _forced_ to use proprietary software, and i'm not the only
person.

basically, it should be pretty clear that now that python _can_
be compiled for win32 using an entirely free-software platform,
the proprietary build chain should be absolutely dropped like a
red-hot stone.

... but regarding "minority platform"? that's .... really quite
funny and ironic.

it was a _non-existent_ platform until about... yesterday :)

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

hi amaury, thanks for responding.

> Is Msys+Mingw32 (running on a regular Windows) an interesting
> configuration to support?

[wine+]msys+mingw32 is used to _build_ python - not depend on it.
[wine+]msys+mingw32 _replaces_ the proprietary build toolchain MSVC.

clarification:

* in the case of #3871, msys+mingw32 replaces MSVC.
* in the case of #4954, wine+msys+mingw32 replaces MSVC _and_ windows

but in either case, you end up with a complete build of python.exe,
libpython2.5.dll and modules that is perfectly well capable of
running under both wine _and_ native windows...

... WITHOUT requiring, in any way shape or form,
EITHER msys OR mingw32.

in other words, it's a big damn deal.

no more proprietary dependence.

... let me put it this way, martin: if i told richard stallman that
you said that "msys+mingw32 was a minority platform" he'd have
a fit!! :)


> The build tools are similar to the ones used by cygwin, except that the
> C runtime is msvcrt.

yeeees.... sort-of - but if you do s/#ifdef __CYGWIN__/#if defined
(__CYGWIN__) || defined(__MINGW32__) it all goes _horribly_ wrong :)

you actually have to do s/#ifdef _MSC_VER/#if defined(_MSC_VER) ||
defined(__MINGW32__) because _MSC_VER is used alll over the place
to detect and indicate a win32 build (as separate and distinct
from a cygwin build).

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

updated patch - also removes quotes removal quotes of graminit and
configure so that martin is happier :)

also included is an updated version of #4956 as it's an essential
integral part of compiling and using python.exe under msys that
it actually WORK! :)

Added file: http://bugs.python.org/file12764/x

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Changes by Luke Kenneth Casson Leighton <lkcl@lkcl.net>:


Removed file: http://bugs.python.org/file12755/f

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Martin v. Löwis <martin@v.loewis.de> added the comment:

> Is Msys+Mingw32 (running on a regular Windows) an interesting
> configuration to support?

It is certainly desirable to be able to build extension modules
with this configuration; AFAIU, distutils already supports that
case. Whether or not it is desirable to be build Python from
source in this configuration, I don't know - most people that
want to build with mingw seem to be using the Python binaries
available from python.org, or from ActiveState. I'm not sure
whether mingw is capable of building Python correctly, with
assembly manifests and all.

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Martin v. Löwis <martin@v.loewis.de> added the comment:

> * graminit and configure were removed because they are built
> automatically. no project should ever include auto-generated files so i
> assumed that it would be reasonable to remove them from the python_2.5.2
> "original" git commit that i did, in order to produce the diff.

Please trust that Python puts generated files into the repository for
good reasons. This is not going to change, regardless of how evil you
consider it.

> ... but regarding "minority platform"? that's .... really quite
> funny and ironic.
>
> it was a _non-existent_ platform until about... yesterday :)

Ok, so feel free to continue to work on it; good luck with
that project. If the project is still alive in a few years
from now, and enjoys a significant user base, we should reconsider
accepting patches.

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Martin v. Löwis <martin@v.loewis.de> added the comment:

> no more proprietary dependence.

So what CRT do you link with? Is it msvcrt? Which version?

If building with mingw becomes wide-spread, care must
be taken that it uses the same CRT as the official binaries.
Otherwise, it will be a desaster for Python.

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

> Please trust that Python puts generated files into the repository for
> good reasons.

i can respect that :) for no reason other than if somone says
"please trust", i do :)

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

> So what CRT do you link with? Is it msvcrt? Which version?

i was _just_ beginning to wonder about that, after i saw
rpetrov's comments about MSCVER stuff.

http://www.mingw.org/wiki/SpecsFileHOWTO

aww, _frick_. :)

well... it's _going_ to be msvcr80. why don't mingw ship with
a pre-prepared msvcr80 specfile??

*sigh*.

will get back to you on that one - it may make a difference on
the regression tests (250 passed, 12 failed, 8 skpped).

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

> It is certainly desirable to be able to build extension modules
> with this configuration;

yeah, and the nice thing is - it works, too! :)

> AFAIU, distutils already supports that case.

not without modification, it doesn't: #3871 adds "proper"
mingw32 compiler detection and link options, such as support
for dll.a which isn't auto-detected (but is on cygwin)

etc. etc.

> Whether or not it is desirable to be build Python from
> source in this configuration, I don't know - most people that
> want to build with mingw seem to be using the Python binaries
> available from python.org, or from ActiveState.

i tried that a few months ago.

1) there is no .tgz for python-win32 so i was forced to
install from a binary package

2) there is no .exe so i was forced to install from msi,
a proprietary installer - which as a free software
developer, i have issues with, but i tried it anyway.

3) msi installed, thanks to winetricks, but segfaulted
and borked. as i didn't really want to use it, i
wasn't that unhappy.

4) the msi turns out to be understood by the free software
cabextract package. but... the filenames are all borked
and mangled.

at that point, i decided i'd had enough: i wasn't going to
go through 100 files looking for the one that _might_ be
the python25.lib implib, plus other files that i'd need
to do a non-proprietary-dependent build of python software.

> I'm not sure
> whether mingw is capable of building Python correctly, with
> assembly manifests and all.

python setup.py build seems quite happy: rpetrov added .S
to the compile-extensions and happily compiled win32.S
in libffi with it.

the only slight issue there is that the assembly file
is incompatible with MSVC and so it borked (not at build
time, unfortunately - at runtime).

there's been a gcc bugreport raised, to get interoperability
back (urk).

GCC issue #36834.

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

yaay! here's the regression test log, including some
loovely wine segfaults :)

summary:

250 tests OK.

12 tests failed:
test_builtin test_cpickle test_file test_gzip test_locale
test_mailbox test_os test_pep277 test_socket test_unicode_file
test_xpickle test_zipfile
60 tests skipped:
....
....
....
8 skips unexpected on win32:
test_bz2 test_cProfile test_bsddb test_profile test_tcl
test_sundry test_sqlite test_winsound

not baaad :)

Added file: http://bugs.python.org/file12765/regressiontest.log

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Martin v. Löwis <martin@v.loewis.de> added the comment:

> 2) there is no .exe so i was forced to install from msi,
> a proprietary installer - which as a free software
> developer, i have issues with, but i tried it anyway.

Please understand that MSI is *not* a proprietary installer.
Instead, it is just a data format to be interpreted by the
installer. Whether that installer is proprietary or not
depends on whether you use the Microsoft one, or write one
yourself (or perhaps use a free interpreter for MSI that
somebody else might have written).

So to help Wine, it would be best to write a proper MSI
interpreter.

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

http://www.winehq.org/pipermail/wine-users/2006-May/021624.html

_drat_.

a rebuild of wine, adding a workaround to cope with lack of
support for msvcrt80 xml-based process files, is required,
commenting out a couple of functions from kernel32!

euuw :)

so - that's going to delay things somewhat. i'll test with
msvcr71, anyway, for the time-being, see if that is any
better than msvcrt (default).

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Martin v. Löwis <martin@v.loewis.de> added the comment:

I'm rejecting this patch for the moment, as it seems far from being
ready yet. If you consider it complete and applicable-as-is (i.e. if you
can't think of any possible further improvement to the patch), please
submit a new issue. If I had to review the patch "x", I would reject it
for many reasons, e.g. because it is not against the trunk, and because
it removes graminit.h.

----------
resolution: -> rejected
status: open -> closed

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

workarounds for a couple of wine bugs,
also includes e.g. #4977 64-bit assumptions on 32-bit systems.

Added file: http://bugs.python.org/file12780/f

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Changes by Luke Kenneth Casson Leighton <lkcl@lkcl.net>:


Removed file: http://bugs.python.org/file12764/x

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Changes by Martin v. Löwis <martin@v.loewis.de>:


----------
nosy: -loewis

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

updated. incorporating roumen's work as well.

Added file: http://bugs.python.org/file12826/f

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

manifests and rc files for msvcr80 build

Added file: http://bugs.python.org/file12830/x

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue4954] native build of python win32 using msys under wine. [ In reply to ]
Luke Kenneth Casson Leighton <lkcl@lkcl.net> added the comment:

martin, so sorry, i didn't see your comments - no dang hell no i'm
not done yet.

regarding graminit.h: graminit.h isn't being removed - i keep editing it
out of the patch.
i'm not _submitting_ it as part of the patch because what happens is that
Parser/pgen.exe generates CRLF and EVVVVERY single line of graminit.h
gets replaced.

so it's a bit irritating to keep having to even see it, so i removed
it from my local repository.

if i accidentally submit its removal please ignore it, entirely.

people _are_ tracking the progress of this development, and assisting
in the work. i'm using the bugtracker to communicate the progress.

closing the bug looks... like... i dunno - it's not very helpful,
though, i have to say.

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4954>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com